Checkpoint preservation

Abstract

An apparatus, the apparatus comprising: at least one processor to execute a software process; a memory system to store data for access to the at least one processor; and checkpoint setting circuitry to trigger saving a checkpoint of a context state associated with at least one software process executed by the at least one processor to the memory system. The checkpoint save is a background process executed by the checkpoint setting circuitry in a background where the at least one processor executes the software process.

Description

Background Art Technical Field

[0001] The present technology relates to the field of data processing. Technical Background

[0003] In order to improve fault tolerance in a data processing system including one or more processors, it may be useful to save a checkpoint of a context state associated with at least one software process executed by the processor. If a fault in a component of the data processing system results in corruption of the context state associated with the software process, a previously saved checkpoint may be restored. Summary of the invention

[0004] At least some examples of the present technology provide an apparatus comprising:

[0005] at least one processor, the at least one processor configured to execute a software process;

[0006] a memory system for storing data for access by at least one processor; and

[0007] A checkpointing circuit for triggering saving a checkpoint of a context state associated with at least one software process executed by at least one processor to a memory system; wherein:

[0008] The checkpoint saving is a background process performed by the checkpoint setting circuit in the background of at least one processor executing a software process.

[0009] At least some examples of the present technology provide a method comprising:

[0010] executing a software process using at least one processor;

[0011] storing the data in a memory system for access by at least one processor; and

[0012] triggering saving a checkpoint of a context state associated with at least one software process executed by at least one processor to a memory system; wherein:

[0013] The checkpoint saving is a background process performed by the checkpoint setting circuit in the background of at least one processor executing a software process.

[0014] At least some examples of the present technology provide a computer readable medium for storing a computer readable code for manufacturing an apparatus, the apparatus comprising:

[0015] at least one processor, the at least one processor configured to execute a software process;

[0016] a memory system for storing data for access by at least one processor; and

[0017] Checkpointing circuitry for triggering saving a checkpoint of a context state associated with at least one software process executed by at least one processor to a memory system; wherein:

[0018] The checkpoint saving is a background process performed by the checkpoint setting circuit in the background of at least one processor executing a software process.

[0019] Further aspects, features and advantages of the present technology will be apparent from the following description of examples read in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] For comparison, Figure 1 Stop-based checkpointing is shown, wherein software-managed saving of a checkpoint is performed by a checkpointing software routine that interrupts processing of other software processes;

[0021] Figure 2 An example of non-stop checkpointing is shown, where saving of checkpoints is performed in the background while other software processes are executing;

[0022] Figure 3 An example of a data processing system having checkpointing circuitry is shown, and the saving of a hierarchical set of checkpoints for a given software process is schematically illustrated;

[0023] Figure 4 A method including checkpoint saving is shown;

[0024] Figure 5 shows the steps performed in response to a context switch;

[0025] Figure 6 shows steps performed in response to the occurrence of a hardware error affecting context data of a given software process;

[0026] Figure 7 The use of a paired cache line to maintain a memory state difference indication indicating updates made to data stored in another cache line is shown;

[0027] Figure 8 shows the steps performed in response to a storage operation performed by a given software process;

[0028] Fig. 9 Another example of steps performed in response to a context switch is shown;

[0029] Fig.10shows steps performed in response to a checkpoint merge / propagation event; and

[0030] Fig.11 Data sharing between software processes is shown. DETAILED DESCRIPTION

[0031] An apparatus has: at least one processor for executing a software process; a memory system for storing data for access by the at least one processor; and a checkpoint setting circuit for triggering a checkpoint of a context state associated with at least one software process executed by the at least one processor to be saved to the memory system. The checkpoint saving is a background process performed by the checkpoint setting circuit in the background of the at least one processor executing the software process.

[0032] Checkpoint saving helps provide fault tolerance by enabling recovery in the presence of a fault that affects the context state of a given software process, so that the given software process is not affected by the error caused by the fault and continues to move forward based on the restored context state. By saving the checkpoint to a memory system, which is generally accessible by at least one processor in response to a load / store instruction executed in the software process for data access, checkpoints can be saved for many more software processes and time points, which is more practical than saving the checkpoint to a dedicated hardware storage device within the processor. Therefore, for example, the checkpoint scheme can be extended to different numbers of processors in the system, different numbers of threads running on the processor, and different time granularities for capturing checkpoints. However, a typical scheme for saving checkpoints to a memory system relies on stopping the execution of the software process of interest and running a checkpoint setting software routine, which is responsible for capturing the context state of the given software process and saving it to memory. This affects the performance available to the software process itself, especially in the case of desired fine-grained checkpoint settings, where checkpoints are captured at relatively short time intervals.

[0033] In the examples discussed below, checkpointing circuitry is provided that is coupled to a memory system to trigger saving a checkpoint of a context state associated with at least one software process to the memory system. The checkpoint saving is a background process performed by the checkpointing circuitry in the background of executing a software process executed by at least one processor. Thus, execution of the software process does not need to be interrupted to allow checkpointing, and the software process can continue to execute while the checkpointing circuitry manages the checkpoint saving in the background (e.g., for a previously executed software process). Thus, the performance cost of implementing checkpointing can be greatly reduced.

[0034] There may be a variety of ways in which the checkpointing circuitry may be implemented. In one example, the checkpointing circuitry includes hardware circuit logic that is configured to perform hardware-managed checkpoint saving. This means that no software control of the checkpoint saving process is required. The hardware may trigger the required save operation without directly involving software (although the hardware may still operate based on configuration information set by software).

[0035] Alternatively, the checkpointing circuitry may include an additional processor or microcontroller separate from at least one processor executing the software process (context) whose context state is being checkpointed. The additional processor or microcontroller may execute checkpointing firmware or software that manages checkpoint saving. When the checkpointing firmware or software is running on a processor or microcontroller separate from at least one processor running the software process being checkpointed, the checkpointing firmware / software may operate in the background of the software processes and therefore there is no need to stop the execution of the software processes to allow checkpoints of the context states of those software processes to be taken. In this case, memory accesses for saving / restoring checkpoints may be triggered by requests from the firmware or software executing on the additional processor or microcontroller.

[0036] The checkpointing circuitry may include memory access circuitry capable of initiating a memory access request to request a read, write, or transfer of data at a given location in a memory system without being prompted by a memory access request issued by software executing on at least one processor. For example, the checkpointing circuitry may include a separate structure or upgraded memory controller coupled to the memory system network traffic and having the ability to trigger the generation of checkpoints at various time intervals without requiring an explicit software trigger from a software process running on at least one processor, or the checkpointing circuitry may include an additional processor or microcontroller as described above. Although software may set configuration information that controls the manner in which the checkpointing circuitry functions, the capture of individual checkpoints does not require explicit software intervention from the software process that is performing the checkpointing.

[0037] The hardware circuit logic for the checkpointing circuit may be a single hardware unit, or may be distributed circuit logic including circuit logic implemented at multiple different locations in the system. For example, the distributed circuit logic may include some circuit logic local to a cache of a given processor, as well as some circuit logic within a memory system architecture and / or a memory controller coupled to a memory system bus. In general, by providing hardware circuit logic that is capable of initiating a checkpoint save without requiring the execution of explicit software instructions to direct data transfer operations for saving a checkpoint, this reduces the impact on the performance achieved by a software process executed by at least one processor.

[0038] The checkpointing circuitry may maintain tracking data indicating one or more locations of checkpoints storing context states for a given software process. Thus, in addition to being responsible for the capture of checkpoints, the checkpointing circuitry may also be responsible for maintaining a record of the memory addresses of the various checkpoints storing context data so that an appropriate checkpoint may be selected for recovery when necessary. Tracking of the locations where checkpoints are stored may be performed in hardware without requiring explicit software-triggered updates of the tracking data structure, or may be performed by firmware / software executed on another processor or microcontroller as described above. The tracking data structure may be stored in a dedicated hardware storage circuit (e.g., a set of registers maintained by the checkpointing circuitry), or may be a memory-based data structure stored in a memory system accessed using a selected set of addresses within an address space mapped for use by the checkpointing circuitry. For example, the checkpointing circuitry may include a register that may be programmed with an address for accessing a checkpoint tracking data structure. The use of a memory-based tracking data structure may help enable the checkpointing circuitry to scale to different numbers of processors and different numbers of software processes executing on the processors.

[0039] The checkpoint setting circuit can restore a given checkpoint in response to a hardware error signal indicating the occurrence of a hardware error. For example, a hardware error can be a RAS (reliability, availability and security) error, which indicates an event that has caused a specific hardware component failure, so that the context state may have been damaged. For example, a hardware error can be a single event upset (bit flip) caused by cosmic rays, ionizing particles or other background radiation hitting a trigger, latch or other data storage element, which may cause one or more bits of the stored state to flip the value from 0 to 1 or from 1 to 0, thereby damaging the stored data. Bit flips may also occur due to electrical interference, crosstalk or other reasons. Other types of hardware failures include electronic components degrading over time due to electromigration or other effects. Hardware errors can be detected using techniques such as error detection codes or error correction codes, or by providing redundant components that can operate synchronously. Such hardware errors can be events that cause hardware to behave incorrectly (contrary to the correct behavior designed by the designer). This type of hardware error can be distinguished from other types of errors, such as speculation errors when hardware that makes speculative predictions about program execution outcomes incorrectly predicts the behavior of future events, and therefore the incorrectly executed instructions must be flushed and execution resumed from the program flow point before the speculation error occurred (in this type of speculation error, the hardware has behaved correctly according to the expected behavior specified by the system designer, but the error occurs because the program behavior changes compared to the previous behavior used to form the prediction).

[0040] The checkpointing circuitry may trigger the saving of a hierarchical set of checkpoints, the hierarchical set of checkpoints comprising two or more checkpoints at different points in time, wherein the checkpointing circuitry controls the checkpoints to be stored in at least two different levels of a memory system hierarchy. For example, the levels of the memory system hierarchy may include one or more cache levels, on-chip memory system storage (e.g., dynamic random access memory (DRAM)), and / or off-chip memory system storage. By storing multiple checkpoints corresponding to different points in time of software execution, and storing those checkpoints in at least two different levels of the memory system hierarchy, fault tolerance is improved because if a hardware error affects a given level of the memory system hierarchy, it is more likely that an error-free checkpoint can be recovered based on a checkpoint stored in another level of the memory system hierarchy.

[0041] The checkpointing circuitry may maintain the set of hierarchical checkpoints to store checkpoints corresponding to earlier points in time at a level of the memory system hierarchy that is accessed more slowly than a level of the memory system hierarchy for storing checkpoints corresponding to more recent points in time. Thus, recently captured checkpoints may be stored in faster access levels of the memory system hierarchy (such as in a cache closer to the processor), while earlier captured checkpoints may be stored in slower access levels of the memory system hierarchy (such as random access memory or off-chip storage). This approach has several advantages. First, if an error occurs (given the location where the error occurred, if possible), the most recent checkpoint is typically the checkpoint that is most desirable to restore, because this reduces the amount of software execution that must be replayed after restoration of the checkpoint, thereby improving performance, and therefore making the most recent checkpoint the fastest to access may help improve performance. However, maintaining checkpoints for earlier timings may also be useful in situations where the most recently computed state is corrupted due to an error.

[0042] Maintaining the set of hierarchical checkpoints such that the most recent checkpoints are in the faster access levels of the memory hierarchy and older checkpoints are gradually moved down to the slower access levels of the memory system hierarchy may also be simpler to manage from the perspective of the checkpointing circuitry. For example, in response to a checkpoint propagation triggering event, the checkpointing circuitry may propagate at least one checkpoint to the next level of the memory system hierarchy (e.g., by triggering a cache writeback event, or exporting data from an on-chip memory to an off-chip memory, etc.). For example, the checkpoint propagation triggering event may be the passage of a specific period of time, the passage of a number of processing cycles, the passage of a specific number of checkpoint capture events, the passage of a specific number of context switches between software processes, or the amount of remaining space in a higher level of the memory hierarchy available for checkpointing becoming less than a specific threshold. By periodically moving older checkpoints down to the next level of the memory hierarchy to make room for more recently captured checkpoints, this enables the capture of a fine-grained set of hierarchical checkpoints corresponding to a range of different points in time during program execution, which may be spread across the memory system to improve resistance to failures affecting different portions of the memory system.

[0043] In response to a checkpoint merge trigger event, the checkpoint setting circuit can trigger a checkpoint merge operation to merge two or more checkpoints corresponding to different time points of a given software process to form a merged checkpoint for the given software process. For example, a set of checkpoints for multiple different time points can be reduced to a checkpoint corresponding to a single time point in those time points (e.g., the most recent time in the set of time points). This approach can be particularly useful if the checkpoints initially captured are based on difference indications indicating differences compared to previously stored states (as further described below), in which case the checkpoint merge operation can merge the differences indicated for a set of given time points to form a merged checkpoint indicating an absolute value of data to be restored when necessary in response to the detection of a hardware error. The absolute value can be restored faster than a set of difference indications. The checkpoint merge trigger event can be any of the event types indicated for the above checkpoint propagation trigger event.

[0044] In some examples, the checkpoint setting circuitry may write the merged checkpoint to a level of the memory system hierarchy that is accessed more slowly than a level of the memory system hierarchy that is used to store two or more checkpoints that are combined to form the merged checkpoint. For example, the merged checkpoint may be written to a next level of the memory system hierarchy after the level that stores the multiple checkpoints. Thus, in some cases, the checkpoint propagation triggering event and the checkpoint merging triggering event may be the same event, such that the occurrence of the event triggers merging the multiple checkpoints into a single combined checkpoint and writing the combined checkpoint to the next level of the memory system hierarchy, thereby facilitating moving previously captured checkpoints down through the memory system, as discussed above.

[0045] The checkpointing circuitry may capture checkpoints for a plurality of different software processes, each checkpoint being associated with a process identifier indicating a corresponding software process. The checkpointing circuitry may maintain a tracking structure that tracks the location (memory address) at which the checkpoints are stored for each software process so that the process may recover from a hardware error affecting a system component storing a context state associated with the process.

[0046] In other examples, checkpointing may be performed on the entire system rather than on individual processes, and in this case, checkpoints need not be associated with individual processes. Instead, the context states of multiple processes may be covered by a set of shared checkpoints.

[0047] Furthermore, it is not necessary to checkpoint the context state of every software process executing on the device. Some processes may not be considered necessary for checkpointing, for example because their data is not critical to the reliability, availability, or security of the system. Thus, in some examples, checkpointing may involve only a subset of the software processes executing on the device.

[0048] A checkpoint for a given software process may include both register state and data stored to memory by the given software process. Thus, a checkpoint is not limited to register state, but may also include context data stored to memory. This may be useful because the execution context of a given software process may be larger than what can be accommodated within the architectural register set of the processor. Where a checkpoint includes data stored to memory for a given software process, the data may include data written to memory by a storage operation submitted on the architecture of the given software process (i.e., data written to memory by a storage instruction that has been parsed to be correctly executed on the architecture).

[0049] When a checkpoint captured for a given software process includes coherently stored data shared with another software process, restoring the checkpoint for the given software process may also make the coherently stored data available to the other software process. Thus, checkpointing of the state of one process may also benefit other processes that share access to the data.

[0050] Checkpoints can be captured at any time, including at relatively arbitrary points in time that are not necessarily synchronized with context switch timing.

[0051] However, in one example, in response to a context switch at a given processor from processing a first software process to processing a second software process, the checkpointing circuitry may trigger the saving of a new checkpoint for the first context. Triggering the saving of a new checkpoint after a context switch may be useful because, upon a context switch, software executing on a given processor may in any case write register state to memory (to allow the processor's registers to be overwritten by the state of the incoming second software process), and thus the checkpointing circuitry may detect this register state writeback and use it to capture a new checkpoint. Moreover, context switches are typically expected to occur with relatively regular timing, which may make context switch events suitable for generating checkpoints.

[0052] In an example where a checkpoint is captured in response to a context switch, the checkpoint setting circuit may use a context switch signal exposed to hardware to identify the timing of the context switch. Typically, the context switch may be transparent to the hardware circuit logic coupled to the memory system, so that the memory system may not know which software processes are executed at the processor or the timing when the processor switches from one software process to another. However, in order to enable the checkpoint setting circuit within the memory system architecture to capture a checkpoint in response to a context switch using a background process managed in hardware, a mechanism for exposing the context switch signal to hardware may be provided so that the checkpoint setting circuit can know the timing of the context switch. The signal may be implemented in different ways. In one example, by sending a context switch signal exposed to hardware to the processor to trigger the processor to perform a context switch, the checkpoint setting circuit itself may instruct the processor core to perform a context switch at a specific time interval. In this case, the checkpoint setting circuit may be directly responsible for triggering a context switch between software processes executed at a given processor. Alternatively, a separate hardware unit responsible for controlling the timing of the context switch may send a context switch signal exposed to hardware to the processor to trigger the context switch, and send a context switch signal exposed to hardware to the checkpoint setting circuit to trigger the capture of the checkpoint. Another approach is that the processor itself may be responsible for determining the timing of context switches, and when a context switch occurs, the processor may transmit a hardware exposed context switch signal to the checkpointing circuitry to notify the checkpointing circuitry that a context switch has occurred and therefore a checkpoint should be acquired.

[0053] In some examples, a context switch may be reported to the checkpointing circuitry using a context switch signal sent by the processor or a separate context switch triggering hardware unit that performs the context switch, but the checkpointing circuitry may also have the ability to trigger a context switch by sending a hardware-exposed context switch signal to the processor core. One approach is to use deadline-driven context switch timing, where the checkpointing circuitry allows the processor or context switch triggering hardware unit to determine the context switch timing (e.g., at specific time intervals or triggered by software), but if the checkpointing circuitry determines that there has been no context switch within a specific period of time, the checkpointing circuitry itself may send a hardware-exposed context switch signal to the processor to trigger an additional context switch (which may be a "virtual" context switch, where the software process executed after the context switch is the same as the software process executed before the context switch), so that the checkpointing circuitry can ensure that checkpoints are taken at sufficiently regular time intervals. Virtual context switches may force register states to be saved to a cache, which may then be checkpointed by the checkpointing circuitry.

[0054] In the example of capturing a checkpoint in response to a context switch to enable the checkpointing circuitry to capture the register state of the outgoing context (the first software process), one approach may be to provide the checkpointing circuitry with direct access to the registers of the processor. However, in practice, this may be difficult to achieve while meeting the circuit timing for the processing pipeline of the processor, and additional design effort would be required to redesign the processor circuit layout to accommodate the additional wiring in order to allow the checkpointing circuitry access.

[0055] Thus, in some examples, in response to a context switch, it may be useful to cause at least one processor to save the register state of a first software process to a memory system location corresponding to a set of mapping addresses mapped for storing the register state of the first software process according to an architecturally defined context save format. The checkpointing circuitry may then obtain the saved register state from the memory system location corresponding to the set of mapping addresses and use the saved register state to form a new checkpoint. Thus, by defining an architecturally defined context save format, there is an understandable protocol between the processor and the checkpointing circuitry so that the checkpointing circuitry can understand which memory addresses will be written with the stored register state of the first software process, and a checkpoint can be captured in such a way that the mapping between portions of the checkpoint and specific architectural registers is known, so that upon recovery, the corresponding registers can be restored along with the context state that those registers were storing at the point in time represented by the checkpoint. This approach avoids the need for the checkpointing circuitry to directly access processor registers, thereby avoiding the above-mentioned disadvantages. The specific context save format defined by the architecture may vary from architecture to architecture, but in general, by having an agreed-upon context save format, this simplifies the capture of checkpoints because there is no need to include explicit identification data identifying which register corresponds to each piece of saved data.

[0056] In response to the context switch, in response to a signal sent by the given processor confirming that the register state save of the first software process (the outgoing process executed before the context switch) is complete, the checkpoint setting circuit can be triggered to capture the register state of the outgoing context in a new checkpoint. For example, once the register save is complete, the given processor can send a confirmation message to the checkpoint setting circuit. In response to the signal from the given processor, the checkpoint setting circuit can then save the contents of the mapped memory system location (allocated for saving register state according to the context save format defined by the architecture) to the new checkpoint.

[0057] Saving register state (obtained by the checkpointing circuitry from memory system locations corresponding to the set of mapping addresses) to form a new checkpoint occurs as a background process performed in the background of ongoing execution of a second software process at the given processor following a context switch to avoid the need to delay commencement of execution of the second software process while the checkpointing is performed.

[0058] In order to capture memory-based context data associated with a given software process at the time of a context switch, one approach may be to scan an address range in response to the context switch to identify data that may have been modified by the outgoing first software process. However, such a scan may be inefficient and may delay servicing of cache accesses of the incoming second software process.

[0059] One method for improving performance may be that a new checkpoint captured in response to a context switch may include a memory state differential snapshot indicating updates made by the first software process to data stored in memory during a period in which the first software process was active prior to the context switch. The memory state differential snapshot may be based on differential indications maintained on an ongoing basis during execution of the first software process, such that each time the software process writes data to a memory system location corresponding to an address for which a checkpoint is to be maintained, the corresponding differential indication is updated so that subsequently upon a context switch, the latest value of the differential indication may be used to determine data that changed while the first software process was executing. Tracking differences in memory state resulting from execution of the first software process in a window in which a context switch is run may help reduce the number of different addresses for which checkpoint data needs to be captured, as this may avoid the need to checkpoint data from addresses for which memory data did not change during that time window.

[0060] Thus, the checkpointing circuitry may maintain one or more memory state difference indications using a background process operating in the background of executing the first software process. The background process updates a given memory state difference indication to track updates made to corresponding data by the first software process. In response to a context switch, the checkpointing circuitry may capture information of current values ​​of the one or more memory state difference indications indicating that the first software process has made at least one update to the corresponding data as a memory state difference snapshot for a new checkpoint.

[0061] In some examples, the memory status difference indication may be tracked in dedicated storage separate from the cache to which the first software processes write data.

[0062] However, to avoid the need for modification of the hardware storage logic of the memory system itself, one approach may be to allocate a particular cache line in the memory address space as a paired cache line for tracking a memory state difference indication associated with another cache line. Thus, the checkpoint setting circuit may maintain a given memory state difference indication in a paired cache line associated with a given cache line of the cache to indicate an update to the data of the given cache line. By using a paired cache line (an additional cache line designated for storing a difference indication of a given cache line) to track the difference between the value of the associated cache line at a previous checkpoint and the latest value of the associated cache line, this avoids the need for dedicated hardware storage logic for tracking differences and makes it relatively efficient for the checkpoint setting circuit to capture any updates to the context state of a given software process when a checkpoint needs to be obtained. The address of the paired cache line may be an address specified by software (e.g., by setting configuration data used by the checkpoint setting circuit to determine which cache line is the paired cache line).

[0063] The term "cache line" refers to a block of data (or equivalently, a block of memory addresses in a memory address space) having a size corresponding to a single cache entry. The cache line does not necessarily need to be actually stored in the cache, as the corresponding data in the random access memory may also be referred to as a cache line, as it has a size corresponding to a single cache entry. Thus, the differential tracking method may also be used when data has been written back from the cache to the memory.

[0064] The checkpointing circuitry may select which cache line is the paired cache line based on a paired cache line indicator associated with a given cache line, and in response to a context switch, the checkpointing circuitry may update the paired cache line indicator to switch which cache line is the paired cache line associated with the given cache line. Thus, independent difference indications may be maintained in different paired cache lines corresponding to the same associated data cache line to allow multiple different difference indications to be maintained for different software processes that may have written to the same cache line and therefore need to obtain independent checkpoints of the data in the data cache line at different points in time. For example, a data cache line may be associated with a pairing cache line indicator that indicates which of a plurality of pairing cache lines is the pairing cache line to be used for currently tracking differences caused by updates to the associated data cache line, and in response to a context switch, the pairing cache line indicator may be updated to point to a next pairing cache line for tracking difference indications for an incoming software process to be executed after the context switch, such that any updates to the associated data cache line by the incoming software process will not affect the difference tracking indications maintained for an outgoing software process executing before the context switch. This therefore enables the checkpointing circuitry to save a checkpoint for an outgoing software process based on the difference tracking indication in the pairing cache line that is not at risk of being overwritten by an update made by an incoming software process because the incoming software process will cause an update to the difference indication in a different pairing cache line due to the switching of the pairing cache line triggered by the context switch.

[0065] In the case where a new checkpoint is captured as a memory state difference snapshot that indicates current values ​​of one or more memory state difference indications for which at least one update has been made by a first software process since a previous checkpoint was taken, performing a merge of multiple checkpoints into a single merged checkpoint may be particularly useful, as previously discussed. In this case, the merge may include replacing multiple difference-based checkpoints (each difference-based checkpoint is indicated as a difference relative to a previous checkpoint) with a single merged checkpoint that provides an absolute value for each checkpointed cache line (and any captured register state). To perform the merge, the checkpointing circuitry may reference an earlier captured merged checkpoint for the same software process to find the absolute data value of the first snapshot in the group of snapshots being merged relative to which it indicates a difference indication.

[0066] For a given checkpoint, it is not necessary to cover the entire address space accessible to a given software process. In some cases, the checkpointing circuitry can be configured by software to define which areas of the address space store data that require checkpointing, and can omit the saving of data updated in areas of the address space that do not require checkpointing.

[0067] For comparison, Figure 1 A typical stop-based approach for checkpointing the context state of a software process is shown. Figure 1 As shown on the left hand side of FIG, multiple software processes P1, P2, P3, P4 share the processing resources of a given processor on a time-sharing basis (context switches are performed from time to time to switch between one process and another). Figure 1 At time τ shown, process P3 is active, while other processes P1, P2, and P4 are inactive. After a certain checkpoint interval CI (i.e., at time τ+CI), processing of the active software process P3 is stopped, so that the checkpoint setting software code can be executed instead. Therefore, at time τ+CI, none of the software processes P1 to P4 are making any forward progress. The checkpoint setting software code scans the memory address space to identify pages modified by the previously active software process P3. For example, this can be accomplished using a copy-on-write technique, where when the software process P3 begins execution within a given time window after a previous checkpoint capture or context switch, all pages accessible to the software process P3 may initially be marked as read-only in the page table, even if it is ultimately intended to allow the software process P3 to write to those pages. When software process P3 first attempts to write to a given page, the read-only permissions may cause a fault to be triggered, and the fault may cause an exception handler to be executed to modify the access permissions to be read / written, and optionally record an indication of the given page in a tracking structure used to track pages accessed by software process P3. Therefore, when the checkpointing software code executes, the checkpointing software code may check the page table structure to check which pages accessible to process P3 have been marked as read / write, or may check the tracking structure maintained by the software exception handler based on a copy-on-write technique (using a tracking structure separate from the page table may allow the checkpointing software code to be parsed more efficiently). Subsequently, data from the memory pages that are found to have been modified by process P3 may be saved to a checkpoint data structure stored in memory to act as a backup copy that may be restored if a hardware failure occurs that damages any context state associated with process P3.

[0068] However, a problem with this approach is that processing performance is degraded because all software processes P1 to P4 are interrupted to allow the checkpoint setting software to capture the checkpoint.

[0069] Figure 2 Shown is a non-stop checkpointing that enables the context state of a previously active software process (including both register state and data written to memory) to be checkpointed in the background while other software processes are executed. This is achieved by providing a checkpointing circuit in hardware that performs a hardware-triggered save of the checkpoint while other software processes can continue to execute on the processor core.

[0070] Therefore, if Figure 2 As shown, after a context switch from processing software process P3 to software process P4 at time τ+1, in parallel with the ongoing execution of the newly active software process P4, the checkpoint setting circuit saves a checkpoint of the context state associated with the outgoing process P3. Similarly, at time τ+2, when a context switch from software process P4 to software process P1 is executed, the checkpoint setting circuit saves a checkpoint of the context state associated with the outgoing process P4, while the newly active software process P1 continues to execute. Similar saves of checkpoints of the context states of processes P1 and P2 are shown to occur at times τ+3 and τ+4. Therefore, context snapshots are taken in units of specific time quanta corresponding to the time window for context switching. As a background process running in parallel with the processor executing instructions from the new software process, the checkpoint setting circuit saves each captured context snapshot to the memory system.

[0071] Figure 3 An example of a data processing system 2 is shown that includes a plurality of processors 6 accessing a memory system that includes a plurality of hierarchical levels of memory storage. In this example, the data processing system 2 is a server-type system that includes one or more nodes 4, each node including a plurality of processors 6. For example, each node 4 may be on a separate integrated circuit. Although Figure 3 Two nodes 4 are shown, but it should be understood that other examples may have only a single node 4 or more than two nodes 4. In this example, each processor 6 is a CPU (central processing unit), but the system 2 may also include other types of processors, such as a GPU (graphics processing unit) or an NPU (neural processing unit, which is a processor with hardware targeted at accelerating neural network processing or other machine learning algorithms). Similarly, although Figure 3 Each node 4 is shown to have two processors 6 , but other examples may provide a different number of processors 6 for each node 4 .

[0072] Figure 3A subset of the components of each processor 6 is shown, which may include a register file structure 10 for storing register states (e.g., operands of instructions executed by the processor, results of executed instructions, a program counter value indicating the instruction address representing the current program flow point reached during program execution, and control register states), a plurality of buffer or queue structures 12 for queuing load / store requests and associated data awaiting transfer to or from a memory system, and at least one on-chip cache 14. Although in Figure 3 Although not explicitly shown in the figure, it should be understood that each processor may also have many other components, including a processing pipeline for decoding and executing instructions, a memory management unit for managing address translation and memory access control, a prediction unit such as a branch predictor or prefetcher, etc. Figure 3 The registers 10, buffers / queues 12 and on-chip cache 14 are only explicitly shown for the processor 6 in node 0, but the processor 6 in node 1 may have similar components. Not all processors 6 necessarily have the same design. There may be heterogeneous processors with different microarchitecture designs.

[0073] The memory system hierarchy includes multiple levels of memory storage in order of increasing distance from the execution logic of the processor 6, and therefore in order of decreasing access speed to data stored at each level, including:

[0074] On-chip cache 14 dedicated to an individual processor 6, such as a level 1 cache, and possibly at least one further level of cache (e.g., a level 2 or level 3 cache), which represent the fastest access level of the memory system hierarchy;

[0075] A last level cache (LLC) shared between processors 6 in the same node 4

[0076] 16, which has a slower access speed than the on-chip cache 14;

[0077] On-chip memory storage 18 (e.g., on-chip DRAM (dynamic random access memory), on-chip SRAM (static random access memory) or any other form of on-chip memory) within a particular node 4 that is accessed more slowly than the last level cache 16; and

[0078] • Off-chip memory 20 (e.g., off-chip DRAM or off-chip non-volatile memory (NVM), such as flash memory, phase change memory, ferroelectric RAM, magnetoresistive RAM, etc.) shared between nodes 4, which is the slowest access level of the memory system hierarchy.

[0079] It should be understood that the number of levels in the memory system hierarchy may be different than Figure 3 The number of levels shown.

[0080] Figure 3 Also shown is a RAS (reliability, availability, and security) management engine (RME) 22, which acts as a checkpointing circuit to coordinate hardware-triggered non-stop saving of checkpoints of context states of software processes executing on processors 6 of system 2. Although RME 22 is shown as a single entity external to processing node 4 for ease of understanding, in practice, RME may be implemented using distributed circuit logic implemented on-chip at multiple locations in each processor node 4 (e.g., including on-chip cache 14, memory controller, or logic local to the memory system architecture).

[0081] Another specific implementation of RME may be a processor core 6 that runs an independent thread that performs the function of RME in the background of other function threads executed on other cores. Thus, in some cases, one of the cores 6 may be dedicated to the purpose of acting as RME. Alternatively, the core 6 that executes the RME thread may not be fixed, so the RME thread may migrate between cores. In either case, the RME thread operates in the background of other function threads executed on other cores 6 to allow forward progress without having to stop to perform checkpointing.

[0082] RME 22 controls the capture of a set of hierarchical checkpoints of register states and data in memory, the set of hierarchical checkpoints representing the execution context for a given software process. A set of checkpoints may be maintained for multiple software processes, the set of checkpoints being identified based on the process identifier of the software process. For a given software process, the set of hierarchical checkpoints includes checkpoints captured at different times, and generally speaking, checkpoints for earlier times are gradually propagated to slower access levels of the memory system, so that checkpoints for more recent times are stored at higher levels of the memory hierarchy closer to processor 6.

[0083] For example, Figure 3 As shown, for a given software process, multiple snapshots (corresponding to different points in time) of register state CS1 and memory data are initially captured in the on-chip cache 14 .

[0084] For example, when context switching, software usually saves the register state of the outgoing software process to the memory to save the register state when the incoming software process rewrites the register. In order to enable RME 22 to detect this write-back of the register state and to capture it in the checkpoint in a software-independent format that allows the register state to be restored to the register file structure 10 later in a manner that restores each individual data item to the correct register, an architected format can be defined to be used to save the register state context in the memory. For example, a set of architectural registers defined for a given software process can be mapped to a corresponding set of mapped physical addresses in the memory (for example, registers r1-r8 can be mapped to a set of specific physical addresses, such as PA 0x101-108). This means that the external privileged agent (RME 22) can refresh and copy the register content from the set of mapped physical addresses to the area of ​​the memory address space that is allocated for storing checkpoints, without the need to understand the specific software-specific way in which the software component saves the register state to the memory in the absence of an architecturally defined format. To enable RME 22 to detect when saving of register states to memory for an outgoing software process has been completed so that a checkpoint for the outgoing software process can now be captured, processor 6 may send a register save complete signal to RME 22 to confirm when register save is completed after a context switch.

[0085] To enable the capture of memory-based context data for a given software process, a difference-based background process may be used, wherein, while the given software process is executing, each time the software process writes data to a given cache line, a corresponding update may also be made to an associated paired cache line that tracks the difference between the data of the given cache line at the beginning of a time window and the latest value of the data of the given cache line. Thus, when a new snapshot of memory-based context data needs to be captured for a given software process, the relevant paired cache line associated with the data cache line used by the software process may be saved to a checkpoint structure that is stored in a memory (e.g., within an on-chip cache storage device).

[0086] From time to time (e.g., at intervals of a predetermined number of context switches), RME 22 may perform actions to merge multiple separately captured checkpoints into a smaller number of checkpoints at a coarser temporal resolution granularity (e.g., multiple checkpoints for multiple different points in time may be compressed into a single checkpoint corresponding to a single point in time). For example, multiple independent difference indications captured at different times may be compressed into a single difference indication or absolute value of data stored for a particular cache line. Similarly, the most recent snapshot of register state may be retained for a given compressed checkpoint, and other snapshots of register state indicated in the original set of checkpoints may be discarded. Thus, over time, the granularity of maintaining checkpoints for different points in time may become less fine.

[0087] Furthermore, RME 22 may propagate checkpoints captured at a given level of the memory system hierarchy downward to slower accessed lower levels of the memory system hierarchy. This may be done while merging multiple checkpoints into a smaller number of checkpoints. For example, when a checkpoint merge event occurs (e.g., after a certain number of context switches), a compressed checkpoint derived from combining multiple checkpoints stored at a given level of the memory system hierarchy may be written for storage in a next level of the memory system hierarchy. For example, if Figure 3 As shown, multiple checkpoints CS1 for the same software process stored in the on-chip cache 14 can be combined to form a single checkpoint CS2 stored in the LLC 16; multiple checkpoints CS2 for the same software process stored in the LLC 16 can be combined to form a single checkpoint CS3 stored in the on-chip memory 18; and multiple checkpoints CS3 for the same software process stored in the on-chip memory 18 can be combined to form a single checkpoint CS4 stored in the off-chip memory.

[0088] Thus, over time, RME 22 controls the checkpoints for a given software process to gradually move down through the memory system hierarchy, so that there is a wide range of options available for restoring the context state of a given software in the event of a hardware error (e.g., a RAS error). RAS errors (e.g., single or multi-bit errors caused by particle impacts) can be detected by any known error detection technology, such as using error detection / correction codes or synchronous redundant components. RME 22 can maintain a checkpoint tracking data structure (e.g., stored in a memory system) that tracks the address of a specific checkpoint corresponding to a specific moment saved for a given software process. For example, RME 22 can associate each checkpoint with a process identifier that identifies the corresponding software process and a timestamp value that identifies the corresponding moment. Therefore, when a hardware error is detected at a given level of the memory system hierarchy, RME 22 can determine which checkpoint is the most suitable checkpoint for recovery based on the location where the error was detected.

[0089] Figure 4 1 is a flow chart illustrating a method of capturing a checkpoint. At step 100, a software process is executed using at least one processor 6. At step 102, data is stored in a memory system 14, 16, 18, 20 for access by at least one processor 6. At step 104, the checkpointing circuitry 22 triggers saving of a context state associated with at least one software process executed by at least one processor to the memory system 14, 16, 18, 20. The saving is a background process performed by the checkpointing circuitry 22 in the background of executing the software process executed on the at least one processor 6. Figure 3 In the example of , the save is managed by RME 22 hardware, but other examples may execute checkpointing software on one of the processors 6 or on a separate microcontroller to provide software managed checkpointing in the background of executing software processes running on the other processors 6.

[0090] Figure 51 is a flow chart illustrating a method of responding to a context switch. At step 120, a context switch occurs on a given processor 6 to switch between processing of an outgoing software process and processing of an incoming software process. The occurrence of the context switch is signaled by a hardware-exposed context switch signal sent to or from the checkpointing circuitry 22. The context switch signal may be a signal sent from the processor 6 to the checkpointing circuitry 22 to notify the checkpointing circuitry 22 that a context switch has occurred, or may be a signal sent from the checkpointing circuitry 22 to the processor 6 to indicate that the processor 6 should switch between contexts. Some implementations may support a combination of these methods, wherein, generally speaking, the processor 6 (or software executing on the processor) may be responsible for determining the timing of context switches, but the checkpointing circuitry 22 may monitor the time since a previous context switch, and if that time exceeds a certain limit, the checkpointing circuitry 22 may trigger a context switch for the purpose of enabling the capture of a checkpoint. For example, the context switch triggered by the checkpointing circuit 22 may be a "virtual" context switch, for which the incoming software process is effectively the same as the outgoing software process, such that the saving of register states to memory is performed solely for the purpose of enabling the capture of a checkpoint, even though the outgoing software process may continue after the context switch. Alternatively, context switches may be forced to occur at specific regular time intervals, and in such cases, it may not be necessary to trigger a virtual context switch. Moreover, the timing of the context may be controlled by an external agent separate from both the processor 6 and the checkpointing circuit 22. In an embodiment where software-managed checkpointing is performed in the background of process execution, the timing of the context switch may be notified to the software by writing to a specific memory address designated for indicating an indication that a context switch has occurred.

[0091] Regardless of how the context switch is triggered, at step 122, the checkpointing circuitry 22 determines that a context switch has occurred and, in response, captures a new checkpoint of the latest register state of the outgoing software process and the data stored to memory by the outgoing software process in the period since the last context switch. As previously discussed, this may be done by copying the saved register state from a set of mapping addresses to which the processor 6 saves the outgoing process register state in accordance with the architected context save format, and by capturing a set of latest difference indications maintained in the paired cache lines of a corresponding set of data cache lines (see discussion below). Figure 7 ).

[0092] Figure 61 is a flow chart illustrating restoration of a given checkpoint. At step 130, the checkpoint setting circuitry 22 detects that a hardware error has occurred that affects context data of a given software process. In response to the hardware error, at step 132, the checkpoint setting circuitry 22 selects a particular checkpoint associated with the given software process based on the location of the hardware error and checkpoint tracking data maintained by the checkpoint setting circuitry 22 to track the location of checkpoints saved for different timestamp values ​​of the given software process. At step 134, the checkpoint setting circuitry restores the selected checkpoint by: restoring a memory-based state of the given software process based on the captured memory data of the selected checkpoint, and restoring a register state of the given software process. The register state can be restored by the checkpoint setting circuitry 22 restoring the checkpoint setting value of the register state to the set of mapping addresses in memory allocated to represent the register context state of the given software process. When execution of the given software process is later resumed, the given software process can execute a load instruction to read back its register state from the set of mapped addresses according to an architecturally defined register state save format (with an understandable mapping between the relative location of the data value within the set of mapped addresses and the specific architectural register whose data will be restored based on the data value). This approach avoids the need for the checkpointing circuitry 22 to directly access processor registers, which enables the checkpointing technique to work with existing processor designs without modifying the circuit logic of the processor 6.

[0093] Figure 7 An example of using paired cache lines to track the difference between memory-based data at the beginning of an execution time window for a given software process and the memory-based data at the end of the time window when a checkpoint is taken is shown. Here, the term "cache line" is used to refer to a block of data of a given size that corresponds to one cache entry of a cache. A cache line may also be understood as a unit of data that forms the basic unit of data transfer between different levels of a memory system hierarchy. Thus, a data cache line does not necessarily need to be currently stored in the cache and may also be stored in on-chip memory 18 or off-chip memory 20. Regardless of the current location of the data, for a given data cache line 150, the associated cache line 152 (in Figure 7150) is designated as the paired cache line for tracking the difference indication for the corresponding data cache line 150. Thus, each time a given software process writes to a data cache line 150, the checkpointing circuitry 22 triggers a write to the corresponding paired cache line 152 (which may exist in the cache 14, 16, or may need to be brought into the cache if not already in the cache) so that the difference indication in the paired cache line 152 tracks the difference between the latest value of the data cache line 150 and the value that the data cache line 150 had at the beginning of the current processing window since the previous checkpoint was taken.

[0094] When a checkpoint is taken on a context switch, in order to prevent an incoming software process from overwriting a differential indication captured for an outgoing software process in a paired cache line before a checkpoint can be taken, while avoiding the need to delay execution of the incoming software process, the paired cache line identifier 154 may be updated to switch which other cache line is the paired cache line for a given data line 150, and subsequently, differential updates in response to writes by the incoming software process to the data cache line 150 may occur in another paired cache line 156, as indicated by the updated paired cache line identifier 154. For example, the paired cache line identifier 154 may be represented by the number of upper bits of the data cache line 150. This therefore enables the checkpointing circuitry 22 to save the differential indication stored in the previous paired cache line 152 in the background while continuing to process the incoming software process, and if the incoming software process also writes to the cache line 150, it may update the new paired cache line 156. Such use of paired cache lines to track differences can make context saving more efficient because if any of the difference indications 152 for a set of data cache lines 150 to be checkpointed indicates that there are no differences since a previous checkpoint, then the state of those cache lines does not have to be saved to a new checkpoint, but rather the checkpoint can indicate that those cache lines 150 have not changed since the previous checkpoint.

[0095] Thus, in one example, a context snapshot is taken after the end of each time quantum (a time window for context switching). At the expiration of each quantum, the checkpointing function is managed by the RME 22. An example process for capturing the memory state at a given checkpoint is:

[0096] The RME 22 saves / processes the "diff" (difference indication) of each cache line 150 to be checkpointed at each checkpoint timing.

[0097] A cache line 150 may have a "pair" of cache lines 152 assigned to maintain a "difference" (e.g., using the N upper bits as a "pair" indicator address 154, i.e., the address hash used to determine which cache set to access for the paired line 152 should not conflict with the hash of the parent cache line 150, so that both the data cache line 150 and the difference cache line 152 are more likely to co-reside in the cache). The pair indicator 154 may be incremented on each checkpoint, and all other "difference" bits may be reset to 0 on each checkpoint after saving the previous value.

[0098] • Track memory writes during a process runtime quantum using a "bad" indication, and construct new checkpoints from the "possibly modified" "bad" cache lines 152 that have been updated during that quantum.

[0099] • On "heartbeat" timing (eg, every N time quanta), the "differences" for a given software process can be flushed to the next level of the memory system hierarchy and / or consolidated into fewer checkpoints.

[0100] Attach to each context snapshot a timestamp value representing the time quantum represented by that snapshot.

[0101] Figure 8 2 is a flow chart illustrating maintaining a difference indication in a paired cache line. At step 200, a given software process performs a store operation. In response, at step 202, data in a given cache line 150 having an address specified by the store operation is updated based on the store data specified by the store operation. In response to detecting the store operation, at step 204, the checkpoint setting circuit 22 also maintains a memory state difference indication in a paired cache line 152 associated with the data cache line 150 to track updates made to the given cache line (e.g., by setting the paired cache line to the difference between the value of the data cache line 150 at the previous checkpoint capture timing and the updated value written to the cache line by the store operation at step 202).

[0102] Fig. 9 is a flowchart showing the steps performed in response to a context switch in an embodiment using a paired cache line method to track memory state updates for a given software process. At step 210, a context switch occurs, which can be performed by the checkpoint setting circuit using the above description for Figure 5The processor 6 executing the outgoing software process writes the register state of the outgoing software process to a set of addresses in an architecture-defined format in response to the context switch, or the processor 6 may be triggered by a software instruction, such as an instruction of a supervisory code that manages the context switch or an instruction of the incoming software process to be executed after the context switch. At step 214, the checkpoint setting circuit 22 captures a new checkpoint, which includes a snapshot of the register state obtained from the memory system location corresponding to the mapped address (once the processor 6 confirms that the save of the register state to the set of mapped addresses has been completed), and a snapshot of the updated memory data of the outgoing software process, which is determined based on the current value of one or more memory state difference indications stored in the paired cache lines 152 corresponding to the set of data cache lines 150 to be checkpointed. The new checkpoint is saved in association with a process identifier identifying the outgoing software process and a timestamp indicating the point in time represented by the checkpointed context state. Checkpointing circuitry 22 updates its checkpoint tracking structure to track the location in the memory system where the checkpoint was saved.

[0103] Fig.10 is a flow chart illustrating the steps of merging checkpoints and propagating the merged checkpoints to the next level of the memory system hierarchy. At step 220, a checkpoint merge / propagate event occurs (e.g., this may occur after a certain number N of context switches have occurred since a previous checkpoint merge / propagate event (each context switch triggering Fig. 9 In response to the checkpoint merge / propagation event, at step 222, the checkpoint setting circuit merges multiple checkpoints for a given software process corresponding to different points in time into a merged checkpoint. For example, multiple difference indications captured at different points in time can be merged into absolute data values ​​of corresponding memory data (by applying the indicated difference indications to the earlier absolute values ​​of the memory data of the previous merged checkpoint, or to a default value (e.g., zero) in the absence of a previous merged checkpoint). All snapshots except the most recent snapshot of the register state can be eliminated from the merged checkpoint. At step 224, the merged checkpoint is propagated to the next level of the memory system hierarchy (e.g., by triggering a cache write-back operation or other operation to force the state to be written back from a faster access level to a slower access level of the memory system hierarchy).

[0104] Although Fig.10Examples are shown in which both merging of checkpoints and propagation to the next level of the memory system hierarchy occur in response to the same event, but other examples may trigger propagation to the next level of the memory system hierarchy separately from merging multiple checkpoints into a compressed form.

[0105] Therefore, in summary, a method for cost-effective fault tolerance in server-class systems or other data processing systems is provided. In such systems, single-bit faults and multi-bit faults may occur in components of any node 4 of the system 2, and therefore data anywhere in the node 4 may become unrecoverable, causing the execution of the thread to fail and become unrecoverable. The processor core 6 may also fail, resulting in inaccessible memory. In order to provide fault tolerance for such hardware errors, a hardware-supported checkpointing mechanism is provided, which saves the execution state of the executed process (and associated memory data). The method allows fine granularity and multi-level checkpointing of the application execution state. The periodic snapshots of the executed threads are incrementally updated and promoted to the slower access level of the memory to recover from faults in the components 6, 16, 18 of the node and / or the entire node 4. RME 22 is connected to the memory system network traffic and is provided as an independent structure, an upgraded memory controller or as a distributed circuit logic, and has the ability to trigger the generation of snapshots with synchronized time quanta and the ability to monitor data traffic and generate data "differences" (differences) to store with the checkpoint state. This enables recovery using a saved snapshot of the context state that is consistent with the storage data accessed by the context. In the case where shared data is shared between processes according to a consistency protocol implemented by the memory system, the consistently stored snapshots of the associated processes are restored together, so a domino effect is possible here. Advantages include: non-stop checkpointing enables cost-free forward error-free execution; a fine-grained hierarchical checkpointing approach allows error containment; the technique implements distributed checkpointing and memory tracking through background processes; and the technique does not require modification of the processor core 6, because checkpointing can be handled by a hardware component 22 added within the memory system. In order to support the checkpointing technique, it may be useful for the checkpointing circuit 22 to have the ability to monitor context switches / migrations in the system from the hardware layer (the hardware dispatches the context switch, or the operating system executing on the processor 6 notifies the hardware of the switch / migration). Moreover, it may be useful to map the registers 10 to a memory backend structure having an architecturally understood format.

[0106] It should be appreciated that other implementations may use a separate tracking structure to track the addresses of cache lines modified by software during a given window between context switches, so the use of paired cache lines is not necessary.

[0107] As described above, when a checkpoint captured for a given software process includes coherently stored data shared with another software process, restoring the checkpoint for the given software process may also make the coherently stored data available to the other software process. Fig.11 Two processes P1 and P2 are shown, each accessing exclusive data E1 and E2, respectively, and also accessing shared data SH (i.e., P1 accesses E1 and SH, but not E2, and P2 accesses E2 and SH, but not E1). When checkpointing, the creation of memory snapshots, data differences, and checkpoint states for each process includes all memory accessed by each process. In addition to the corresponding exclusive data, shared data "SH" is also considered during the checkpoint state generation of each process. Each checkpoint is recorded with its corresponding time quantum. The checkpoints stored for each of the processes P1 and P2 are appropriately marked to indicate their access to shared data. When an unrecoverable error occurs on either process (P1 or P2) and recovery is triggered, recovery can be triggered for both processes, synchronizing to the checkpoint state at the same time quantum. In order to recreate the memory state using the stored differences (intermediate checkpoints stored in closer memory), these differences can be replayed in the reverse order of their corresponding quantum timestamps. In the case of recreating shared data, memory consistency can be ensured by replaying the memory segments of each process (including the shared data accessed by each process) while taking into account the corresponding timestamps of each copy of the shared data.

[0108] The concepts described herein may be embodied in computer readable code for manufacturing devices embodying the described concepts. For example, the computer readable code may be used at one or more stages of the semiconductor design and manufacturing process, including the electronic design automation (EDA) stage, to manufacture integrated circuits including devices embodying the concepts. The above computer readable code may additionally or alternatively enable the definition, modeling, simulation, verification and / or testing of devices embodying the concepts described herein.

[0109] For example, computer readable code for making an apparatus embodying the concepts described herein may be embodied in code defining a hardware description language (HDL) representation of the concepts. For example, the code may define a register transfer level (RTL) abstraction for defining one or more logic circuits of an apparatus embodying the concepts. The code may define an HDL representation of one or more logic circuits embodying the apparatus in Verilog, SystemVerilog, Chisel, or VHDL (Very High Speed ​​Integrated Circuit Hardware Description Language), as well as intermediate representations such as FIRRTL. The computer readable code may provide definitions embodying concepts using system-level modeling languages ​​such as SystemC and SystemVerilog, or other behavioral representations of concepts that may be interpreted by a computer to enable simulation, functional and / or formal verification, and testing of the concepts.

[0110] Additionally or alternatively, the computer readable code may define a low-level description of an integrated circuit component embodying the concepts described herein, such as one or more netlists or integrated circuit layout definitions, including representations such as GDSII. One or more netlists or other computer readable representations of integrated circuit components may be generated by applying one or more logic synthesis processes to the RTL representation to generate definitions for manufacturing devices embodying the present invention. Alternatively or additionally, one or more logic synthesis processes may generate from the computer readable code a bitstream to be loaded into a field programmable gate array (FPGA) to configure the FPGA to embody the described concepts. The FPGA may be deployed for the purpose of verifying and testing the concepts prior to manufacturing the integrated circuit, or the FPGA may be deployed directly in a product.

[0111] The computer readable code may include a mixture of code representations for making a device, such as a mixture including one or more of an RTL representation, a netlist representation, or another computer readable definition used in a semiconductor design and manufacturing process to make a device embodying the invention. Alternatively or additionally, concepts may be defined in a combination of a computer readable definition used in a semiconductor design and manufacturing process to make a device and computer readable code defining instructions to be executed by the defined device once manufactured.

[0112] Such computer readable code may be provided in any known transient computer readable medium (such as wired or wireless code transmission over a network) or non-transitory computer readable medium such as a semiconductor, a magnetic disk, or an optical disk. An integrated circuit manufactured using the computer readable code may include components such as one or more of the following: a central processing unit, a graphics processing unit, a neural processing unit, a digital signal processor, or other components that embody the concepts alone or collectively.

[0113] In this application, the words "configured to..." are used to mean that elements of a device have a configuration that enables the performance of a defined operation. In this context, "configuration" means the arrangement or manner of interconnection of hardware or software. For example, the device may have dedicated hardware that provides the defined operation, or a processor or other processing device may be programmed to perform the function. "Configured to" does not mean that the device elements need to be changed in any way in order to provide the defined operation.

[0114] Although exemplary embodiments of the present invention have been described in detail herein with reference to the accompanying drawings, it should be understood that the invention is not limited to those precise embodiments and that various changes and modifications may be effected therein by those skilled in the art without departing from the scope of the invention as defined in the appended claims.

Claims

1. A device, comprising: at least one processor, the at least one processor configured to execute a software process; a memory system for storing data for access by the at least one processor; and checkpointing circuitry for triggering saving a checkpoint of a context state associated with at least one software process executed by the at least one processor to the memory system; wherein: The checkpointing is a background process performed by the checkpointing circuitry in the background of the at least one processor executing the software process.

2. The apparatus of claim 1 , wherein the checkpointing circuitry comprises memory access circuitry capable of initiating a memory access request requesting a read, write, or transfer of data at a given location in the memory system without being prompted by a memory access request issued by software executing on the at least one processor.

3. An apparatus according to any one of claims 1 and 2, wherein the checkpointing circuitry is configured to maintain tracking data indicating one or more locations of checkpoints storing context states for a given software process.

4. An apparatus as claimed in any preceding claim, wherein the checkpointing circuitry is configured to restore a given checkpoint in response to a hardware error signal indicating the occurrence of a hardware error.

5. An apparatus according to any preceding claim, wherein the checkpointing circuitry is configured to trigger the saving of a hierarchical set of checkpoints, the hierarchical set of checkpoints comprising a plurality of checkpoints at a plurality of different points in time, wherein the checkpointing circuitry is configured to control the plurality of checkpoints to be stored to at least two different levels of a memory system hierarchy.

6. The apparatus of claim 5 , wherein the checkpointing circuitry is configured to maintain the set of hierarchical checkpoints to store checkpoints corresponding to earlier points in time at a level of the memory system hierarchy that is accessed more slowly than a level of the memory system hierarchy used to store checkpoints corresponding to more recent points in time.

7. An apparatus according to any preceding claim, wherein in response to a checkpoint merge triggering event, the checkpoint setting circuit is configured to trigger a checkpoint merge operation to merge multiple checkpoints corresponding to different time points of a given software process to form a merged checkpoint for the given software process.

8. The apparatus of claim 7, wherein the checkpointing circuit is configured to write the merged checkpoint to a level of the memory system hierarchy that is accessed more slowly than a level of the memory system hierarchy used to store the plurality of checkpoints.

9. An apparatus as claimed in any preceding claim, wherein the checkpointing circuitry is configured to capture checkpoints for a plurality of different software processes, each checkpoint being associated with a process identifier indicative of a corresponding software process.

10. An apparatus as claimed in any preceding claim, wherein a checkpoint for a given software process comprises both register state and data stored to memory by the given software process.

11. An apparatus according to any preceding claim, wherein: When a checkpoint captured for a given software process includes coherently stored data shared with another software process, restoring the checkpoint for the given software process also makes the coherently stored data available to the other software process.

12. An apparatus according to any preceding claim, wherein: In response to a context switch at a given processor from processing a first software process to processing a second software process, the checkpointing circuitry is configured to trigger saving of a new checkpoint for the first context.

13. The apparatus of claim 12, wherein the checkpoint setting circuit is configured to use a context switch signal exposed to hardware to determine a timing at which the new checkpoint is to be saved.

14. The device according to any one of claims 12 and 13, wherein: In response to the context switch, the at least one processor is configured to save the register state of the first software process to a memory system location corresponding to a set of mapping addresses mapped for storing the register state of the first software process according to an architecture-defined context save format, and The new checkpoint includes saved register states obtained by the checkpointing circuitry from the memory system locations corresponding to the set of mapped addresses.

15. The apparatus of any one of claims 12 to 14, wherein the new checkpoint comprises a memory state differential snapshot indicating updates made by the first software process to data stored in memory during a period in which the first software process was active before the context switch.

16. The apparatus of claim 15, wherein the checkpointing circuitry is configured to maintain one or more memory state difference indications using a background process operating in the background of execution of the first software process, the background process updating a given memory state difference indication to track updates made to corresponding data by the first software process; In response to the context switch, the checkpointing circuitry is configured to capture information of current values ​​of one or more memory state difference indications indicating that the first software process has made at least one update to the corresponding data as the memory state difference snapshot of the new checkpoint.

17. The apparatus of claim 16, wherein the checkpointing circuitry is configured to maintain a given memory state difference indication in a paired cache line associated with a given cache line of a cache to indicate an update to data of the given cache line.

18. The apparatus of claim 17, wherein the checkpointing circuitry is configured to select which cache line is the companion cache line based on a companion cache line indicator associated with the given cache line; and In response to a context switch, the checkpointing circuitry is configured to update the pairing cache line indicator to switch which cache line is the pairing cache line associated with the given cache line.

19. A method comprising: executing a software process using at least one processor; storing data in a memory system for access by the at least one processor; as well as triggering saving a checkpoint of a context state associated with at least one software process executed by the at least one processor to the memory system; wherein: The checkpointing is a background process performed by a checkpointing circuit in the background of the at least one processor executing the software process.

20. A computer readable medium storing computer readable code for manufacturing an apparatus comprising: at least one processor, the at least one processor configured to execute a software process; a memory system for storing data for access by the at least one processor; and checkpointing circuitry for triggering saving a checkpoint of a context state associated with at least one software process executed by the at least one processor to the memory system; wherein: The checkpointing is a background process performed by the checkpointing circuitry in the background of the at least one processor executing the software process.

Images