Method and apparatus for providing persistence to remote non-volatile memory

Abstract

A processing device and method for controlling remote durable writes are provided. The method includes receiving a program instruction to issue a durable write to a remote memory. The method includes logging an entry in a local domain when the durable write instruction is received and providing a first indication that the durable write will be persisted to the remote memory. The method includes performing the durable write to the remote memory and providing a second indication that the durable write to the remote memory is complete. The method includes providing the first and second indicators when it is determined not to perform the durable write according to global ordering, and providing the second indicator without providing the first indicator when it is determined to perform the durable write to the remote memory according to global ordering.

Description

Technical Field

[0001] (Cross - reference to Related Applications) This application claims the benefit of U.S. Patent Application No. 17 / 031,518, filed on September 24, 2020, entitled "METHOD AND APPARATUS FOR PROVIDING PERSISTENCE TO REMOTE NON - VOLATILE MEMORY", which is hereby incorporated by reference in its entirety.

Background Art

[0002] Persistent memory is becoming an important area of the memory system architecture. Persistent memory efficiently stores data structures so that the data structures can continue to be accessed (e.g., via memory instructions or memory APIs) even after the process that generated or last modified the content of the memory location has ended. For example, persistent memory retains its content over power cycles so that an application can resume from where it stopped when power was lost, rather than having to restart the application when power is lost.

[0003] Persistent memory can be directly accessed by a processor without stopping the execution of block I / O like conventional storage devices. By connecting persistent memory to a memory bus, a processor (e.g., a CPU) can directly access data without the overhead of a driver or PCIe. As an example, since the memory is accessed within a 64 - byte cache line, the CPU reads only what needs to be read instead of rounding each access up to the block size as in the case of conventional storage devices.

[0004] A more detailed understanding can be obtained from the following description given by way of example in conjunction with the accompanying drawings.

Brief Description of the Drawings

[0005] [Figure 1] This is a block diagram of an exemplary device that can implement one or more features of the present disclosure. [Figure 2] This block diagram shows exemplary components for implementing one or more features of the present disclosure. [Figure 3] This flowchart illustrates an exemplary method for controlling persistent writes to remote persistent memory based on the features of this disclosure. [Figure 4] This figure shows the performance benefits of an exemplary method that utilizes early persistence fences and full persistence fences for persistence ordering operations, as described in the features of this disclosure. [Modes for carrying out the invention]

[0006] The ability of a system (e.g., a processor and memory architecture) to reliably persist data to memory typically requires primitives and instructions to guarantee data persistence. Conventional techniques achieve reliable hardware persistence through a combination of cache line flushing and memory fences. For example, some conventional techniques achieve persistence by writing the data to be persisted to a region of the physical address space that maps to non-volatile memory (NVM), evicting the data from the cache using one or more cache line write back (CLWB) instructions, and issuing a memory fence by using a store fence (SFENCE) instruction to generate a memory fence.

[0007] Memory fences are used to prevent subsequent write operations (also referred to herein as store operations) from completing until the memory fence is complete. Typically, NVM write bandwidth is significantly smaller than DRAM bandwidth, but NVM write latency is significantly larger than DRAM. As a result, memory fences, which must wait for preceding NVM memory write operations to complete, can take a long time to complete. Therefore, the overhead added by memory fences is not insignificant.

[0008] This overhead is exacerbated when writing to (also referred to as storing) a remote NVM (e.g., fabric-attached memory (FAM) or a shared pool of NVM attached to a remote node on a multi-node machine). This is because writing to a remote NVM adds the potentially large network-trajectory round-trip latency and the bandwidth constraints of such a network, which can be heavily loaded, to the critical path to store-fence completion. Store-fence completion can then cause prolonged stalls and performance degradation for parts of the application or program (e.g., threads) that issue persistent writes.

[0009] Some conventional techniques modify applications to avoid persistence to a remote NVM. However, these techniques are inefficient because they restrict system design flexibility and reduce the FAM value. In addition, each affected application must be modified to enforce persistence, resulting in redundant work. Therefore, conventional techniques fail to reduce the significant performance overhead associated with fine-grained persistent writes to a remote NVM (e.g., writes within threads or work items).

[0010] Features of this disclosure include apparatus and methods for controlling persistent writes of an executable program for multiple NVM pools having different performance characteristics. For example, features disclosed herein avoid overall performance overhead (e.g., round-trip latency across potentially large networks) in fine-grain NVM use cases by utilizing local (local to the processor issuing the persistent write) persistent memory to provide the processor issuing the persistent write with early confirmation that persistent writes to remote NVMs are ultimately persisted in the remote persistent memory intended for them. Fine-grain NVM use cases include examples where program dependent instructions (i.e., instructions that depend on persistent writes) can continue to execute efficiently in response to confirmation of the completion of persistent writes or confirmation of the ordering between persistent writes issued by the same sequence of instructions (e.g., threads or work items), the same core, or the same processor.

[0011] In the examples described herein, different performance characteristics include local NVMs having higher performance (e.g., higher bandwidth and lower latency) than remote NVMs. However, the features of this disclosure are implemented for devices with a variety of different performance characteristics, such as multiple locally attached NVM pools with different performance characteristics, multiple fabric-attached NVM pools with different characteristics (e.g., due to the distance from the processor running the operation to each on the network), and multiple remote nodes having NVMs with different performance characteristics from other nodes (e.g., due to the distance from the processor running the operation to each on the network).

[0012] In some cases, a program determines whether to use global ordering for persistent write operations across multiple instruction sequences (e.g., threads or work items) and multiple cores or processors for a portion of the program (e.g., one or more remote persistent writes). If it is determined that global ordering should be used, local persistent memory is not used for early confirmation of one or more remote persistent writes. Otherwise, local persistent memory is used for early confirmation of remote persistent writes. This determination is made, for example, depending on specific instructions or other annotations associated with one or more persistent writes. For example, if the issuing thread or work item contains instructions for the complete completion of a persistent write (e.g., for synchronization with other threads), local persistent memory is not used for early confirmation. Therefore, if instructions for the complete completion of a persistent write are not included, performance for fine-grained use cases is improved (e.g., latency is reduced), and if the issuing thread or work item contains instructions for the complete completion of a persistent write, global ordering is also made easier.

[0013] A feature of this disclosure is the provision of two separate metrics for persistent writes. For example, when a persistent write to remote memory is issued (e.g., by a processor core of a processor executing a thread or work item), an entry is logged via the processor to a local persistence domain (e.g., local persistent memory such as a local NVM, or the memory controller of the local NVM if the memory controller is in the persistence domain). Once the entry is logged, a first metric (e.g., an early persistence completion (EPC) metric is provided (e.g., by the processor to the processor core) indicating that the persistent write will eventually be persisted to the intended remote persistent memory (e.g., a remote NVM). When the persistent write to the remote persistent memory is complete, full persistence completion (FPC) is provided (e.g., by the processor to the processor core).

[0014] A method is provided for controlling remote persistent writes. This method includes receiving an instruction to issue a persistent write to remote memory and logging an entry to the local domain when a persistent write instruction is received. The method also includes providing a first indicator that a persistent write will be persisted to remote memory and performing the persistent write to remote memory.

[0015] A method is provided for controlling remote persistent writes. This method includes receiving an instruction to issue a persistent write to remote memory. If it is determined that a persistent write to remote memory will not be performed according to global ordering, and if it is determined that a persistent write will not be performed according to global ordering, the method includes logging an entry to the local domain when a persistent write instruction is received, providing the remote memory that a persistent write to a first index will be persisted in the remote memory, and performing the persistent write to the remote memory.

[0016] A processing device having local memory, remote memory, and a processor is provided. The processor is configured to receive instructions to issue persistent writes to the remote memory and to log entries to the local domain when persistent write instructions are received. The processor is also configured to provide a first indicator that a persistent write will be persisted in the remote memory and to perform the persistent write to the remote memory.

[0017] Figure 1 is a block diagram of an exemplary device 100 in which one or more features of the present disclosure may be implemented. Device 100 may include, for example, a computer, a game device, a handheld device, a set-top box, a television, a mobile phone, or a tablet computer. Device 100 includes a processor 102, memory 104, storage device 106, one or more input devices 108, and one or more output devices 110. Device 100 may also optionally include an input driver 112 and an output driver 114. It should be understood that device 100 may include additional components not shown in Figure 1.

[0018] In various alternatives, the processor 102 includes a central processing unit (CPU), a graphics processing unit (GPU), a CPU and GPU located on the same die, or one or more processor cores, each of which may be a CPU or a GPU. In various alternatives, the memory 104 may be located on the same die as the processor 102 or separately from the processor 102. The memory 104 includes volatile or non-volatile memory (e.g., random access memory (RAM), dynamic RAM (DRAM), cache).

[0019] The storage device 106 includes fixed or removable storage devices (e.g., hard disk drives, solid-state drives, optical discs, flash drives). The input device 108 includes, but is not limited to, a keyboard, keypad, touchscreen, touchpad, detector, microphone, accelerometer, gyroscope, biometric scanner, or network connectivity (e.g., a wireless local area network card for transmitting and / or receiving wireless IEEE 802 signals). The output device 110 includes, but is not limited to, a display, speaker, printer, haptic feedback device, one or more lights, antennas, or network connectivity (e.g., a wireless local area network card for transmitting and / or receiving wireless IEEE 802 signals).

[0020] The input driver 112 communicates with the processor 102 and the input device 108, enabling the processor 102 to receive input from the input device 108. The output driver 114 communicates with the processor 102 and the output device 110, enabling the processor 102 to send output to the output device 110. Note that the input driver 112 and the output driver 114 are optional components, and that device 100 operates similarly when the input driver 112 and the output driver 114 are not present. The output driver 114 includes an accelerated processing device (APD) 116 coupled to the display device 118. The APD is configured to accept computation and graphics rendering commands from the processor 102, process those computation and graphics rendering commands, and provide pixel output to the display device 118 for display. As will be described in more detail below, the APD116 includes one or more parallel processing units configured to perform computations according to the single-instruction-multiple-data (SIMD) paradigm. Thus, although various functions are described herein as being performed by or in conjunction with the APD116, in various alternative examples, functions described as being performed by the APD116 are not driven by the host processor (e.g., processor 102) but are additionally or alternatively performed by other computing devices having similar capabilities configured to provide graphic output to the display device 118. For example, any processing system that performs processing tasks according to the SIMD paradigm is intended to be configured to perform the functions described herein. Alternatively, a computing system that does not perform processing tasks according to the SIMD paradigm is intended to perform the functions described herein.

[0021] FIG. 2 is a block diagram showing exemplary components of a processing device for implementing one or more features of the present disclosure. Some of the components shown in FIG. 2 are components of a processor, such as an accelerated processor (e.g., GPU). The features of the present disclosure can be executed by one or more processors of a first type, such as a CPU, and / or one or more processors of a second type, such as a GPU.

[0022] As shown in FIG. 2, the components of the processing device include a processor 200, a memory (e.g., main memory) 202, a local NVM 204, and a remote NVM 206.

[0023] The processor 200 is an accelerated processor such as a CPU or GPU, for example. The processor 200 includes a plurality of processor cores 208. Each processor core 208 can include a dedicated cache 210. Alternatively, the cache 210 can be shared by a plurality of processor cores 208.

[0024] In addition, the processor 200 includes a remote persistent controller 212 that communicates with each processor core 202. The remote persistent controller 206 receives an instruction to issue a persistent write (e.g., from a processor core 208), logs an entry in a local persistent domain (e.g., local NVM 204) when a persistent write instruction is received, and provides a first indicator (e.g., an early persistent completion (EPC) indicator) to the processor core 208 that a persistent write is intended to be persisted in a remote persistent memory (e.g., remote NVM 206), executes the write to the remote persistent memory, and provides a second indicator (e.g., a full persistent completion (FPC) indicator) to the processor core 208 indicating the completion of the persistent write to the remote persistent memory. The remote persistent controller 206 is implemented, for example, by hardware, firmware, software (e.g., executed on one or more processor cores 208), or a combination thereof.

[0025] The local NVM can execute a part of a program more efficiently than, for example, the remote NVM (e.g., with higher bandwidth, lower latency, both higher bandwidth and lower latency, or other performance characteristics).

[0026] FIG. 3 is a flowchart showing an exemplary method 300 for controlling persistent writes to remote memory.

[0027] As shown in block 302 of FIG. 3, method 300 includes receiving an instruction to issue a persistent write to remote memory. For example, an instruction to issue a persistent write (e.g., write data) to remote memory (e.g., to an address of remote NVM 206) is provided by a processor core (e.g., processor core 208) executing a program and received by processor 200 (e.g., received by remote persistent controller 212 of processor 200).

[0028] As shown in block 304 of FIG. 3, method 300 includes logging an entry to the local persistent domain when a persistent write instruction is received. For example, when an instruction to issue a persistent write to remote memory is received, the processor (e.g., remote persistent controller 212 of processor 200) logs an entry to the local persistent domain (e.g., local NVM 204 or a memory controller of local NVM 204 (not shown)).

[0029] Unlike application-level logs (e.g., application-level logs for disaster recovery), logs are not visible to running applications / programs. Logs are used exclusively by system software, firmware, or hardware to temporarily stage persistent writes to remote persistent memory into local memory, such as a local NVM, which is faster (e.g., has higher bandwidth and / or lower latency) than the remote NVM. A portion of the local NVM204 is reserved for log entries (e.g., before or during runtime) via hardware, firmware (e.g., BIOS), or software.

[0030] As shown in block 306 of Figure 3, method 300 includes providing a first indicator (first instruction) to a processor core 208 that provided an instruction to issue a persistent write, indicating that the persistent write will be persisted to remote persistent memory. For example, after an entry is logged to a local persistence domain, the processor (e.g., the remote persistence controller 212 of processor 200) provides the processor core 208 with an Early Persistence Complete (EPC) indicator indicating that the persistent write will be persisted to remote persistent memory (e.g., remote NVM 206).

[0031] After an EPC is generated, pending remote NVM writes are trickled out from the local reserved NVM area to the intended remote NVM location in the background. This trickling is performed in the order in which the pending remote NVM writes are placed in their corresponding logs.

[0032] As shown in blocks 308 and 310 of Figure 3, method 300 includes reading log entries from a local persistence domain and performing persistent writes to remote persistent memory. For example, a processor (e.g., a remote persistence controller 212 of processor 200) tricks the persistent write by reading log entries from a local persistence domain (e.g., local NMV 204) and writing the data to an intended address in remote persistent memory (e.g., local NMV 206).

[0033] As shown in blocks 312 and 314 of Figure 3, method 300 includes deleting a log entry from a log in local persistent memory and providing a second indicator (second instruction) that the persistent write to remote persistent memory is complete. For example, a processor (e.g., a remote persistence controller 212 of processor 200) deletes a log entry from a log in a local persistent domain (e.g., local NVM 204) and provides a second indicator (e.g., a fully persistent completion (FPC) indicator) to the processor core 208 that the persistent write to remote persistent memory (e.g., remote NVM 206) is complete.

[0034] In the example shown in Figure 3, the first metric (e.g., the EPC metric) is generated before the second metric (e.g., the FPC metric). In other alternative examples, the first and second metrics are generated simultaneously.

[0035] In some cases, persistent writes to remote memory are performed without providing a first indicator (e.g., early confirmation). For example, a determination is made as to whether to use global ordering to execute a part of the program (e.g., one or more persistent writes). If it is determined that global ordering should be used, local persistent memory is not used for early confirmation of one or more remote persistent writes, and the first indicator is not provided. If it is not determined that global ordering should be used (e.g., for parts of the program where a later issued write (store) operation is permitted to complete only when it is confirmed that a previous persistent write has finally completed, or when the intended ordering between persistent operations issued by the same thread, core, or processor has been confirmed), local persistent memory is used for early confirmation of one or more remote persistent writes, and both the first and second indicators are provided.

[0036] This determination is made, for example, in response to a specific instruction (e.g., a programmed instruction or annotation associated with one or more persistent writes), such as when an instruction issuing one or more persistent writes includes a request for confirmation of complete persistent completion without a request for confirmation of early persistent completion.

[0037] An example of this determination, which is performed in response to a specific instruction, is shown in Figure 3 by a dashed line. As shown, in determination block 316, a determination is made as to whether the instruction issuing the persistent write includes an instruction request for the complete completion of the persistent write. If determination block 316 determines that the instruction for complete completion is included, the method proceeds to block 310, and local persistent memory is not used for early confirmation. If determination block 316 determines that the instruction for complete completion is not included, the method proceeds to block 304, and local persistent memory is used for early confirmation. Therefore, if the instruction for complete completion of the persistent write is not included, performance for fine-grained use cases is improved (e.g., latency is reduced), and global ordering is also facilitated if the issuing thread or work item includes an instruction for complete completion of the persistent write.

[0038] The example described above includes determining whether global ordering should be used after receiving an instruction to issue a persistent write, whereas the embodiment includes determining whether the determination is made before receiving an instruction to issue a persistent write.

[0039] Events such as thread migration from one core to another, or the completion of a program's process (e.g., an execution thread), are held in the local persistent domain log, and any remote persistent writes belonging to the corresponding thread, core, and / or process are delayed until they reach their final destination.

[0040] In the example described above in Figure 3, the issuing core (e.g., 208) considers the persistent write complete when it receives either a first indicator (e.g., an EPC instruction) or a second indicator (e.g., an FPC instruction). Therefore, an early persistent memory fence (EPFENCE) is provided that considers the persistent write complete when either the first or second indicator is received. In other words, the program does not need to wait for an FPC in a fine-grained NVM example where the program's instructions can continue executing efficiently in response to confirmation of the completion of the persistent write or confirmation of the ordering between persistent writes issued by the same sequence of instructions (e.g., a thread or work item), the same core, or the same processor.

[0041] Therefore, a thread issuing an EPFENCE primitive can continue executing dependent instructions if an EPC corresponding to an unprocessed persistent write is received. To ensure that subsequent reads to the remote NVM receive the latest data, reads to the remote NVM check any pending writes to the same remote address in a local reserved NVM area for staging remote writes. This check can be accelerated via a tracking structure stored in volatile memory (e.g., SRAM, DRAM) because this tracking structure does not need to be persistent.

[0042] In some examples, persistent completion is used to synchronize with other threads that may be running on other processors, such as when persistent writes across multiple threads are explicitly ordered. In these examples, a full persistence fence (FPFENCE) is used, which causes the program to wait for a second indicator (e.g., an FPC indicator) before performing other actions to confirm the persistent completion of the remote NVM write to the final destination. FPFENCE facilitates the ordering of remote persistent writes trickring from multiple processors, which would otherwise not be ordered correctly because the logs in the local NVM are independent. FPFENCE also avoids the overhead cost of other threads checking pending remote writes in the local NVM of each processor running on other processors.

[0043] In some cases, once EPFENCE is complete, bookkeeping (software-level actions that update application-level persistent metadata, such as application-level logs) is performed before EPFENCE is complete, which effectively overlaps the application-level bookkeeping task with the remote NVM write latency.

[0044] Figure 4 shows the performance benefits of an exemplary method utilizing EPFENCE and FPFENCE for persistent ordering operations, as featured in the present disclosure. Figure 4 illustrates these performance advantages compared to using FPFENCE without EPFENCE for each persistent ordering operation. The left side of Figure 4 shows a process in which FPFENCE is used for each persistent ordering operation. The right side of Figure 4 shows a process in which EPFENCE is used for the first and second persistent ordering operations, and a single FPFENCE is used for the third persistent ordering operation.

[0045] As shown in the process on the left side of Figure 4, three separate ordering operations (indicated by vertical arrows) used to facilitate global ordering are executed sequentially (e.g., over time or clock cycles) at three ordering points P1, P3, and P6 of a persistent write (i.e., a persistent write transaction). Latency costs associated with the ordering operations to facilitate global ordering occur at each of the three points P1, P3, and P6. That is, before the next persistent write is permitted at point P7, a first latency cost associated with the global ordering operation occurs after the recovery log from point P1 to fence completion is completed, a second latency cost associated with the global ordering operation occurs after the transaction commit from P3 to fence completion is completed, and a third latency cost associated with the global ordering operation occurs after the log update commit from P6 to fence completion is completed. Therefore, if the process shown on the left side of Figure 4 is executed, the next transaction can start at point P7.

[0046] As shown on the right side of Figure 4, fine-grained persistent ordering (e.g., ordering within thread operations) is performed at two intermediate points P1 (i.e., after the recovery log is completed) and P2 (after the transaction commit is completed), and a global ordering operation is performed once at the point (P4) after the completion of the final stage (i.e., writing the commit log).

[0047] In other words, after the recovery log from point P1 to the first EPC is completed, a first latency cost associated with the fine-grained ordering operation occurs. Since the latency cost associated with the fine-grained ordering operation is smaller than the latency cost associated with the global ordering operation (shown on the left side of Figure 4), the commit transaction on the right side of Figure 4 starts and finishes earlier (at P2) than the commit transaction on the left side of Figure 4 finishes (at P3). After the commit transaction from point P2 to the second EPC is completed, a second latency cost associated with the fine-grained ordering operation occurs. Again, since the latency cost associated with the fine-grained ordering operation is smaller than the latency cost associated with the global ordering operation (shown on the left side of Figure 4), the write commit log on the right side of Figure 4 finishes considerably earlier (at P4) than the write commit log on the left side of Figure 4 finishes (at P6).

[0048] After the write commit log on the right side of Figure 4 is completed, the next transaction begins after the latency cost associated with the global ordering operation has occurred at point P5 via the FPC from point P4 to P5.

[0049] Because the program does not wait as long for early confirmation at the first two points, the process shown on the right in Figure 4 incurs lower latency costs for each of the first two ordering operations than the process on the left in Figure 4. Therefore, the right side of Figure 4 shows the performance benefits and overall latency reduction (i.e., the difference between points P7 and P5).

[0050] As described above, a local NVM area is reserved where log entries for writes to remote memory are stored. In an example where the reserved space is full, subsequent requests to issue persistent writes (e.g., by the processor) may be delayed until enough space is freed by trickling currently stored writes to remote memory. Alternatively, some examples allow the local reserved area to be resized (e.g., made larger when more space is needed, or smaller when space is not being fully utilized).

[0051] Memory consumption can be mitigated, for example, by periodically moving reserved areas of the local NVM to other parts of the local NVM, thereby distributing frequent writes across the entire memory over a period of time. Additionally or alternatively, the local NVM is a more expensive solution such as DRAM with a backup power supply capable of holding data for a desired length of time (e.g., until a backup power supply for the system can be deployed, or until the data stored therein can be extracted and persisted by other means). Additionally or alternatively, remote persistent writes are buffered in queues within the persistence domain, i.e., supplied with enough battery power to drain the contents into non-volatile memory in the event of a power outage. These queues must then store additional metadata in order to complete the remote writes and issue appropriate events (e.g., EPCs).

[0052] The features of this disclosure are useful in events such as power failures or other catastrophic events that cause a processing device to shut down while holding remote writes in a locally reserved NVM area. Upon subsequent power-up, the pending persistent writes are completed to their final destinations before user software execution is permitted.

[0053] It should be understood that many modifications are possible based on the disclosures herein. Although features and elements are described above in specific combinations, each feature or element can be used alone without other features and elements, or in various combinations with or without other features and elements.

[0054] Various functional units shown in the figures and / or described herein (including, but not limited to, the processor 102, input driver 112, input device 108, output driver 114, output device 110, accelerated processing device 116, processor core 208, and remote persistent controller 212) may be implemented as a general-purpose computer, processor, or processor core, or as a program, software, or firmware stored on a non-temporary computer-readable medium or another medium executable by a general-purpose computer, processor, or processor core. The methods provided can be implemented in a general-purpose computer, processor, or processor core. Suitable processors include, by example, general-purpose processors, dedicated processors, conventional processors, digital signal processors (DSPs), multiple microprocessors, one or more microprocessors associated with a DSP core, controllers, microcontrollers, application-specific integrated circuits (ASICs), field-programmable gate array (FPGA) circuits, any other type of integrated circuit (IC), and / or state machines. Such a processor can be manufactured by configuring a manufacturing process using the results of processed hardware description language (HDL) instructions and other intermediate data (instructions that can be stored in a computer-readable medium), including a netlist. The results of such processing may be a mask work, which is used in a subsequent semiconductor manufacturing process to manufacture a processor that implements the features of the present disclosure.

[0055] The methods or flowcharts provided herein can be implemented in computer programs, software, or firmware embedded in a non-temporary computer-readable storage medium for execution by a general-purpose computer or processor. Examples of non-temporary computer-readable storage media include read-only memory (ROM), random access memory (RAM), registers, cache memory, semiconductor memory devices, magnetic media such as internal hard disks and removable disks, magneto-optical media, and optical media such as CD-ROM disks and digital versatile disks (DVDs).

Claims

1. A method for controlling remote persistent writes, The processor receives an instruction to issue a persistent write to remote memory, If the aforementioned persistent write is not performed according to global ordering, When a command to issue the aforementioned persistent write is received, the entry is logged to the local domain, To provide a first indicator that the persistent write is persisted to the remote memory when the entry is logged to the local domain, and a second indicator that the persistent write to the remote memory is completed. Performing the aforementioned persistent write to the remote memory, When the persistent write is performed according to global ordering, the persistent write to the remote memory is performed, and the second index is provided without providing the first index, method.

2. Reading the entry from the local domain, Further including deleting the entry from the local domain, The method according to claim 1.

3. The instruction to issue the persistent write is received from the processor core, and the first and second indicators are provided to the processor core. The method according to claim 1.

4. If it is determined that the persistent write to the remote memory should be performed according to global ordering, the persistent write to the remote memory is performed without logging the entry to the local domain, further comprising: The method according to claim 1.

5. Determining whether the instruction issuing the persistent write requests confirmation that the persistent write has been completed, without requesting confirmation that the persistent write is persisted in the remote memory, If it is determined that the instruction issuing the persistent write requests confirmation that the persistent write has been completed without requesting confirmation that it will be persisted in the remote memory, the persistent write to the remote memory is executed without logging the entry to the local domain and without providing the first indicator, and the second indicator is provided. The method according to claim 1.

6. To provide an early persistent memory fence in which, given either the first indicator or the second indicator, it is determined that the persistent write has been completed, Given either the first indicator or the second indicator, the following actions are performed: The method according to claim 1.

7. The present invention provides a fully persistent memory fence in which, given the second indicator, it is determined that the persistent write operation has been completed. Given the second indicator, the following actions are performed: The method according to claim 1.

8. The local domain includes a first portion of non-volatile memory, the remote memory includes a second portion of non-volatile memory, and the first portion of non-volatile memory is closer to the processor than the second portion of non-volatile memory. The method according to claim 1.

9. A method for controlling remote persistent writes, Receiving an instruction to issue a persistent write to remote memory, If it is determined that the persistent write will not be performed according to the global ordering, When a command to issue the aforementioned persistent write is received, the entry is logged to the local domain, To provide a first indicator that the persistent write is persisted in the remote memory, This includes performing the persistent write to the remote memory, method.

10. The further includes providing a second indicator that the persistent write to the remote memory has been completed. The method of claim 9.

11. Determining whether to perform the aforementioned persistent write to remote memory according to global ordering, If it is determined that the persistent write to the remote memory will not be performed according to the global ordering, the logged entry will be deleted from the local domain after the persistent write to the remote memory is completed. If it is determined that the persistent write to the remote memory should be performed according to global ordering, the persistent write to the remote memory should be performed without logging the entry to the local domain and without providing the first indicator, and the second indicator should be provided. The method of claim 10.

12. Determining whether to perform the persistent write to remote memory according to global ordering includes determining whether the instruction issuing the persistent write includes an instruction to confirm the complete completion of the persistent write, without including an instruction to confirm that the persistent write is persisted to the remote memory, If it is determined that the instruction includes an instruction for confirming the complete completion of the persistent write and an instruction for confirming that the persistent write is persisted in the remote memory, When an instruction to issue the aforementioned persistent write is received, the entry is logged to the local domain, To provide the first indicator that the persistent write is persisted in the remote memory, Performing the aforementioned persistent write to the remote memory, To provide the second indicator that the persistent write to the remote memory has been completed, If it is determined that the instruction includes an instruction to confirm the complete completion of the persistent write, but does not include an instruction to confirm that the persistent write is persisted to the remote memory, the persistent write to the remote memory is executed without logging the entry to the local domain and without providing the first indicator, and the second indicator is provided. The method according to claim 11.

13. If it is determined that the aforementioned persistent write to remote memory will not be performed according to the global ordering, To provide an early persistent memory fence in which, given either the first indicator or the second indicator, it is determined that the persistent write has been completed, Given either the first indicator or the second indicator, the following actions are performed: The method according to claim 11.

14. If it is determined that the persistent write to the remote memory should be performed according to the global ordering, The present invention provides a fully persistent memory fence in which, given the second indicator, it is determined that the persistent write operation has been completed. Given the second indicator, the following actions are performed: The method according to claim 11.

15. Local memory and Remote memory and Equipped with a processor, The aforementioned processor, Receiving an instruction to issue a persistent write to the aforementioned remote memory, If the aforementioned persistent write is not performed according to global ordering, When an instruction to issue the aforementioned persistent write is received, the entry is logged to the local memory, To provide a first indicator that the persistent write is persisted to the remote memory when the entry is logged to the local memory, and a second indicator that the persistent write to the remote memory is completed. Performing the aforementioned persistent write to the remote memory, When the persistent write is performed according to global ordering, the persistent write to the remote memory is performed, and the second index is provided without providing the first index. It is configured to do, Processing device.

16. The local memory includes a first portion of non-volatile memory, the remote memory includes a second portion of non-volatile memory, and the first portion of non-volatile memory is closer to the processor than the second portion of non-volatile memory. The processing device of claim 15.

17. The aforementioned processor includes multiple processor cores, The instruction to issue the persistent write is received from any of the multiple processor cores, The aforementioned processor, Providing the first indicator to any of the processor cores, To provide one of the processor cores with a second indicator that the persistent write to the remote memory has been completed, It is configured to do, The processing device of claim 15.

18. The aforementioned processor, The system is configured to perform the persistent write to the remote memory without logging the entry to the local memory when it is determined that the persistent write to the remote memory should be performed according to the global ordering. The processing device according to claim 17.

19. The aforementioned processor, To provide an early persistent memory fence in which, given either the first indicator or the second indicator, it is determined that the persistent write has been completed, When either the first indicator or the second indicator is given, the subsequent instruction is executed. It is configured to do, The processing device according to claim 17.

Images