Remote memory operations for computing systems with shared memory

Abstract

Embodiments may include systems and methods for performing remote memory operations in a shared memory address space. The device includes a first network controller coupled to a first processor core. The first network controller processes a remote memory operation request generated by a first memory coherence agent based on a first memory operation for an application running on the first processor core. The remote memory operation request is associated with a remote memory address local to a second processor core coupled to the first processor core. The first network controller forwards the remote memory operation request to a second network controller coupled to the second processor core. The second processor core and the second network controller perform a second memory operation to extend the first memory operation as a remote memory operation. Other embodiments may be described and / or claimed.

Description

Technology Field

[0001] The embodiments of this specification generally relate to the technical field of computing systems, and in particular to computing systems having shared memory. Background Technology

[0002] The background description provided in this specification is intended to generally present the context of the disclosed content. Unless otherwise indicated in this specification, the content described in this section is not prior art with respect to the claims of this application, nor is it recognized as prior art by inclusion in this section.

[0003] To improve computing performance, multiple computing devices or nodes may be combined to form a memory-pooled or shared memory system, such as a remote direct memory access (RDMA) system. A partitioned global address space (PGAS) model is a parallel programming model that can be implemented by multiple computing devices sharing memory, for example, a global memory address space, and aims to improve programmer productivity and system performance. However, for some computing systems with shared memory address spaces, the performance in currently implementing some remote memory operations may not be satisfactory. Background art for this application is U.S. Patent Publication No. 2015 / 0081977, published on March 19, 2015. Brief explanation of the drawing

[0004] The embodiments will be easily understood from the following detailed description in conjunction with the accompanying drawings. To facilitate this description, the same reference numbers designate the same structural elements. The embodiments are illustrated in the drawings of the accompanying drawings as examples, not limitations. FIG. 1 illustrates an exemplary computing system having a memory address space shared by multiple computing devices according to various embodiments. FIGS. 2a-2b illustrates exemplary processes of remote memory operations performed by a processor core or network controller of a computing device included in a computing system having a shared memory address space according to various embodiments. FIG. 3 illustrates an exemplary sequence diagram of remote memory operations performed by multiple computing devices in a computing system having a shared memory address space according to various embodiments. FIG. 4 illustrates an exemplary sequence diagram of remote memory operations performed by multiple computing devices in a computing system having a shared memory address space according to various embodiments. FIG. 5 illustrates an exemplary sequence diagram of remote memory operations performed by multiple computing devices in a computing system having a shared memory address space according to various embodiments. FIG. 6 illustrates a drawing of a hardware component of a computing platform suitable for implementing the present disclosure according to various embodiments. FIG. 7 illustrates a storage medium having instructions for carrying out the methods described with reference to FIG. 1 to 6 according to various embodiments. Specific details for implementing the invention

[0005] Devices, methods, and storage media associated with a computing system having a memory address space shared by multiple computing devices are described herein. In some cases, a device for a computing system having a memory address space includes a first processor core and a first network controller coupled to a network switch. The first network controller processes a remote memory operation request. A remote memory operation request is generated by a first memory coherency agent (CHA) based on a first memory operation for an application running on the first processor core. The remote memory operation request is associated with a remote memory address local to a second processor core coupled to the first processor core via a network switch. The first network controller also forwards the remote memory operation request to a second network controller coupled to the second processor core via a network switch. The second processor core and the second network controller perform a second memory operation at a remote memory address local to the second processor core. The second memory operation corresponds to a remote memory operation request generated based on the first memory operation.

[0006] In some cases, a method for a first processor core in a computing system having a memory address space comprises receiving a memory operation instruction for operating on a memory address in a shared memory address space, wherein the memory operation instruction is for an application running on the first processor core. The method further comprises determining that the memory address is a remote memory address local to a second processor core. Subsequently, the method comprises generating a remote memory operation request corresponding to a memory operation instruction for operating on a remote memory address, and transmitting the remote memory operation request to a network controller so that it is forwarded to the second processor core.

[0007] In some cases, a computing device in a computing system having a memory address space comprises a first processor core and a first network controller coupled to the first processor core. The first processor core is arranged to generate a remote memory operation request based on a first memory operation for an application running on the first processor core. The remote memory operation request is associated with a remote memory address local to a second processor core coupled to the first processor core via a network switch. The first network controller processes the remote memory operation request generated by the first processor core and forwards the remote memory operation request to a second network controller coupled to the second processor core via a network switch. The second processor core and the second network controller perform a second memory operation that extends the first memory operation through the network switch, the first network controller, and the second network controller.

[0008] To increase computing power, multiple computing devices or nodes are combined through communication technologies, such as network switches or computer buses, to form memory-pooled or shared memory systems, such as Remote Direct Memory Access (RDMA) systems. Many memory operations performed on a single computing device can be extended to remote memory operations on another computing device in a computing system that has a shared memory address space. For example, memory monitoring or wait operations from the x86 instruction set architecture (ISA), such as MONITOR / MWAIT instructions, are valuable for coordinating multithreaded applications. Specifically, MONITOR / MWAIT instructions monitor memory addresses and wait until the contents of those addresses are modified by instructions executed on different cores or threads. The counterpart remote memory operations of MONITOR / MWAIT will be useful in memory-pooled or shared memory systems.

[0009] Generally, some memory operations, such as memory monitoring or waiting, can be extended to remote memory operations by emulating these memory operations in software. Software can do this over existing network layers, for example, based on the Transmission Control Protocol (TCP). An application running on a computing node and intending to monitor a remote memory address on a remote node will instantiate a daemon on the remote node. Subsequently, the daemon will either invoke the MONITOR / MWAIT instruction pair on the remote node or periodically poll the remote memory address until it is modified. Once a modification of the remote memory address is detected, the daemon will notify the application. However, this daemon-based software approach incurs significant overhead in terms of performance and resources to monitor a single remote memory address. Furthermore, the implementation of these remote memory operations is unreliable due to the existence of many types of errors occurring in processes, such as scheduling errors, network delays and timeouts, once-and-only-once notification semantics, or software bugs in operating systems and applications.

[0010] The embodiments of this specification extend some memory operations performed on one computing device, e.g., MONITOR / MWAIT instructions, to remote memory operations on another computing device of a computing system having a shared memory address space. The computing system includes multiple computing devices interconnected via a network and network controllers. Memory pools or shared memory address spaces are hosted on disaggregated memory drawers, accelerators, or any other device capable of hosting memory or addressable storage. Instead of using software emulation of local memory operations remotely, the embodiments of this specification utilize a network controller and a processor core to work together. In detail, the network controller processes requests for remote memory operations from the processor core of a computing node, configures local monitoring logic, and transmits a callback to the requester when a trigger is generated by the aforementioned monitoring logic for a specific configured memory operation. Although the embodiments of this specification are described based on current architectural platforms, they may be applied to any architecture having a memory or storage controller that manages a set of multiple memory or storage devices. The memory operations MONITOR and MWAIT are used as examples. Other memory operations are similarly extended to remote memory operations. Additionally, the computing system has a shared memory address space, which is shared between any two computing devices of the system or by all computing devices of the system.

[0011] The remote memory operations presented in the embodiments of this specification allow one-sided writes from event, notification, data, or packet producers to trigger the activation of dependent tasks on other nodes through memory-based monitoring. The embodiments of this specification allow many-to-many, many-to-one, and one-to-many synchronization by individual party monitoring and incoming notifications without explicit peer-to-peer or multicasting group notifications. Remote memory waits allow multiple tasks to proceed forwards from barriers even if they are waiting on different nodes. Extended remote memory operations are applicable to computing devices sharing memory on the same node, in a passive memory pool, or in a distributed shared memory environment. Accordingly, the embodiments of this specification provide uniform, flexible, and high-performance ad-hoc data and task coordination and event synchronization between tasks or activities without requiring computing devices to join, manage, and leave multicasting groups. As a result, the embodiments in this specification integrate symmetric multiprocessing (SMP) scale-up programming with distributed shared memory (DSM) scale-out programming. Additionally, partitioned global address space (PGAS) parallel programming models will benefit in terms of performance. For example, implementing distributed locking becomes efficient. In some cases, it is possible to eliminate the need for distributed lock-based exclusion for many uses.Since large pools of shared memory-based access achieve memory elasticity and facilitate direct and seamless shared memory-based multi-node scaling, this capability becomes highly valuable in eliminating programming discontinuities between scale-up and scale-out computations. The techniques presented in the embodiments of this specification can facilitate the deployment of tiered cloud-native solutions where memory is tiered across node boundaries. This type of capability is becoming increasingly important as scale-out and decoupled memory pool solutions, where software architectures with highly complex dependencies (such as Amazon® Lambda, distributed edge cloud, and the Internet of Things (IoT)) become more relevant.

[0012] The following detailed description refers to the accompanying drawings. The same reference numbers are used in different drawings to identify identical or similar elements. In the following description, specific details such as specific structures, architectures, interfaces, technologies, etc., are presented for the purpose of providing a thorough understanding of various aspects of the embodiments, for the sake of non-limiting description. However, it will be apparent to those skilled in the art who benefit from this disclosure that various aspects of the embodiments may be practiced in other examples that deviate from these specific details. In certain cases, descriptions of well-known devices, circuits, and methods are omitted so as not to obscure the description of the embodiments with unnecessary details.

[0013] Operations of various methods may be described in sequence as a number of individual actions or operations in a manner most helpful to understanding the subject matter of the claim. However, the order of description should not be interpreted as implying that these operations are necessarily order-dependent. In particular, these operations are not performed in the order presented. The described operations may be performed in a different order than the described embodiments. Various additional operations may be performed and / or the described operations may be omitted, divided, or combined in additional embodiments.

[0014] For the purposes of this disclosure, the phrases “A / B”, “A or B”, and “A and / or B” mean (A), (B), or (A and B). For the purposes of this disclosure, the phrases “A, B, or C” and “A, B, and / or C” mean (A), (B), (C), (A and B), (A and C), (B and C), or (A, B and C).

[0015] This description may use phrases such as "in an embodiment" or "in embodiments," which may respectively refer to one or more of the same or different embodiments. Furthermore, terms used in relation to the embodiments of this disclosure, such as "comprising," "including," and "having," are synonyms.

[0016] As discussed herein, the term “module” may be used to refer to one or more physical or logical components or elements of a system. In some embodiments, a module may be a separate circuit, but in other embodiments, a module may include a plurality of circuits.

[0017] Where a disclosure refers to an element or a first element or equivalents thereof, such disclosure includes one or more such elements and does not require or exclude two or more such elements. Furthermore, ordinal indicators for identified elements (e.g., first, second, or third) are used to distinguish between elements and do not indicate or imply a required or limited number of such elements, nor do they indicate a specific position or order of such elements unless otherwise specifically stated.

[0018] The terms “coupled with” and “coupled to” may be used herein. “Coupled” may mean one or more of the following: “Coupled” may mean that two or more elements are in direct physical or electrical contact. However, “coupled” may also mean that two or more elements are in indirect contact with each other but still interact or cooperate with each other, and that one or more other elements are coupled or connected between the elements referred to as being coupled with each other. By example, but not limiting, “coupled” may mean that two or more elements or devices are coupled by electrical connections on a printed circuit board, for example, a motherboard. By example, but not limiting, “coupled” may mean that two or more elements / devices cooperate and / or interact through one or more network links, such as wired and / or wireless networks. By example, but not limiting, a computing device may include two or more computing devices by being “coupled” on a motherboard or by one or more network links.

[0019] As used herein, the term “circuit” refers to, is part of, or includes hardware components such as electronic circuits, logic circuits, processors (shared, dedicated, or group), and / or memory (shared, dedicated, or group), ASIC (Application Specific Integrated Circuit), FPD (field-programmable device), FPGA (field-programmable gate array), PLD (programmable logic device), CPLD (complex PLD), HCPLD (high-capacity PLD), structured ASIC, or programmable SoC (programmable System on Chip), DSP (digital signal processor), etc., configured to provide the described function. In some embodiments, the circuit may execute one or more software or firmware programs to provide at least some of the described function.

[0020] As used herein, the term “computer device” may describe any physical hardware device capable of performing a series of arithmetic or logical operations sequentially and automatically, configured to record / store data on a machine-readable medium and to transmit and receive data from one or more other devices in a communication network. A computer device may be considered synonymous with a computer, a computing platform, a computing device, etc., and may be referred to by these terms from time to time. The term “computer system” may include any type of interconnected electronic device, computer device, or component thereof. Additionally, the terms “computer system” and / or “system” may refer to various components of a computer that are coupled to communicate with one another. Additionally, the terms “computer system” and / or “system” may refer to multiple computer devices and / or multiple computing systems that are coupled to communicate with one another and configured to share computing and / or networking resources. Examples of "computer devices," "computer systems," etc. may include cellular phones or smartphones, feature phones, tablet personal computers, wearable computing devices, autonomous sensors, laptop computers, desktop personal computers, video game consoles, digital media players, handheld messaging devices, personal data assistants, e-book readers, augmented reality devices, server computer devices (e.g., standalone, rack-mounted, blade, etc.), cloud computing services / systems, or network elements.

[0021] FIG. 1 illustrates an exemplary computing system (100) having a memory address space (110) shared by multiple computing devices, e.g., computing device (101) and computing device (103), according to various embodiments. In the embodiments, computing device (101) and computing device (103) are coupled by a network switch (107). A computing device may also be referred to as a node or computing node. A computing device (101) includes a network device (111), a processor core (113), and a memory device (115). A computing device (103) includes a network device (131), a processor core (133), and a memory device (135). A network switch (107) is coupled to the network device (111) and the network device (131). Additionally, the computing system (100) includes other computing devices, e.g., computing device (105), coupled to the network switch (107), computing device (101), or computing device (103).

[0022] In the embodiments, for the computing device (101), the network device (111) is coupled to the processor core (113). The processor core (113) includes various storage devices, such as a cache (125) and a memory device (115). A memory coherence agent (CHA) (127) operates on the processor core (113) to manage memory coherence of the cache (125) and the memory device (115). The processor core (113) further includes other components, such as monitor logic (122). An operating system operates on the processor core (113) to support various applications, such as application (124) and application (126). Additionally, the network device (111) includes a network controller (123) and a local storage (121). The network device (111) may be a host fabric interface (HFI) or a network interconnect card (NIC).

[0023] In the embodiments, for the computing device (103), the network device (131) is coupled to the processor core (133). The processor core (133) includes various storage devices, such as a cache (145) and a memory device (135). A CHA (147) operates on the processor core (133) to manage memory coherence of the cache (145) and the memory device (135). The processor core (133) further includes other components, such as monitor logic (142). An operating system operates on the core (133) to support various applications, such as application (144) or application (146). Additionally, the network device (131) includes a network controller (143) and a local storage (141). The network device (131) may be an HFI or a NIC.

[0024] In the embodiments, the memory device (115) is a local memory device for the processor core (113), while the memory device (135) is a local memory device for the processor core (133). Additionally, the memory device (115) and the memory device (135) are shared between the processor core (113) and the processor core (133) to form a shared memory address space (110). Thus, the memory device (135) is accessible as a remote memory address to the processor core (113), and the memory device (115) is accessible as a remote memory address to the processor core (133).

[0025] In some embodiments, the computing device (103) has the same or similar functions as the computing device (101). Functions performed by the network device (131) and the processor core (133) are also performed by the network device (111) and the processor core (113). Similarly, functions performed by the network device (111) and the processor core (113) may be performed by the network device (131) and the processor core (133) respectively.

[0026] In the embodiments, the processor core (113) or CHA (127) operating on the processor core (113) performs various operations. FIGS. 2a-2b illustrate exemplary processes (210 and 220) of remote memory operations performed by a network controller or processor core of a computing device included in a computing system having a shared memory address space according to various embodiments. In the embodiments, process (210) is performed by the processor core of the computing device (101) and the computing device (103), and process (220) is performed by the network controller of the computing device (101) and the computing device (103).

[0027] As illustrated by the process (210) of FIG. 2a, in the interaction (211), the processor core (113) or CHA (127) receives a memory operation instruction to operate on a memory address, wherein the memory operation instruction is for an application running on the processor core (113), e.g., application (124). The memory operation may be a memory monitor operation, a memory wait operation, a merged memory monitor and wait operation, or some other memory operations. The memory address is a local address (128) corresponding to a storage location in the memory device (115). Additionally and alternatively, the memory address is a remote memory address (148) of the memory device (135) that is local to the processor core (133).

[0028] In interaction (213), the processor core (113) or CHA (127) determines that the memory address is a remote memory address, for example, a remote memory address (148) local to the processor core (133). In interaction (215), the processor core (113) or CHA (127) generates a remote memory operation request corresponding to a memory operation instruction to operate on the remote memory address (148). The remote memory operation request is a request for a remote memory operation, for example, a remote memory monitor operation, a remote memory wait operation, or a merged remote memory monitor and wait operation. In interaction (217), the processor core (113) or CHA (127) transmits the remote memory operation request to the network controller (123) so that it is forwarded to the processor core (133). In some embodiments, the remote memory operation request includes the remote memory address (148). Additionally, the processor core (113) or CHA (127) further receives the status of a remote memory operation request from the network controller (123), wherein the status is based on a remote memory operation executed on the processor core (133), and the remote memory operation corresponds to the remote memory operation request. Furthermore, the processor core (113) or CHA (127) reports the result of a memory operation command for the application (124) based on the status of the remote memory operation request.

[0029] Furthermore, the computing device (101) receives a remote memory operation request from another computing device, for example, the computing device (105). The network controller (123) receives the remote memory operation request and configures the processor core (113) to perform operations related to the received remote memory operation request. In detail, the processor core (113) receives a memory operation or a memory operation request from the network controller (123), wherein the memory operation corresponds to the remote memory operation request received by the network controller (123) and is associated with a local memory address of the memory device (115). The processor core (113) performs a memory operation on the local memory address and notifies the network controller (123) of the status of the memory operation.

[0030] In embodiments, the network controller (123) performs various operations. As illustrated by the process (220) of FIG. 2b, in interaction (221), the network controller (123) processes a remote memory operation request generated by the processor core (113) or the CHA (127), wherein the remote memory operation request is associated with a remote memory address, for example, a remote memory address (148) local to the processor core (133). The remote memory operation request is generated by the CHA (127) based on a first memory operation for an application (124) running on the processor core (113). In some embodiments, the remote memory operation request is a request for a remote memory monitor operation, a remote memory wait operation, or a merged remote memory monitor and wait operation.

[0031] In the embodiments, in interaction (223), the network controller (123) also forwards a remote memory operation request to a network controller (143) coupled to the processor core (133) via the network switch (107). The processor core (133) and the network controller (143) perform a second memory operation to extend the first memory operation through the network switch (107), the network controller (123), and the network controller (143). Furthermore, in interaction (225), the network controller (123) receives a notification regarding the status of the remote memory operation request from the network controller (143). Additionally, in interaction (227), the network controller (123) updates the CHA (127) regarding the status of the remote memory operation request received from the network controller (143).

[0032] In the embodiments, except for the teachings of the integrated disclosure, the computing device (101) or the computing device (103) may be any computing system, e.g., a laptop computer, an ultra-laptop computer, a tablet, a touchpad, a portable computer, a handheld computer, a wearable device, a palmtop computer, a personal digital assistant (PDA), an e-reader, a cellular phone, a combination cellular phone / PDA, a mobile smart device (e.g., a smartphone, a smart tablet, etc.), a mobile internet device (MID), a mobile messaging device, a mobile data communication device, a mobile media playback device, a camera, a mobile game console, etc. In the embodiments, the computing device (101) or the computing device (103) may also be a non-mobile device, e.g., a personal computer (PC), a television, a smart television, a data communication device, a media playback device, a gaming console, a gateway, an IoT device, etc., but is not limited thereto. A computing device (101) or a computing device (103) includes processors such as a processor core (113) or a processor core (133), controllers, and other components that run software and / or control hardware to run local programs or to consume services provided by external service providers over a network. A computing device (101) or a computing device (103) may also, or instead, include a web interface running in a browser from which the electronic device can access these web-based services. A computing device (101) or a computing device (103) may also include storage devices that store logic and data associated with programs and services used by the computing device (101) or the computing device (103).

[0033] In embodiments, the processor core (113) or processor core (133) may be a central processing unit (CPU) located on a printed circuit board (PCB). In some embodiments, the processor core (113) or processor core (133) may be a programmable device capable of executing a program. In embodiments, the processor core (113) or processor core (133) may be a microcontroller, a 16-bit processor, a 32-bit processor, a 64-bit processor, a single-core processor, a multi-core processor, a digital signal processor, an embedded processor, or any other processor. In embodiments, an operating system may be operated on the processor core (113) or processor core (133), and may include system drivers for various protocols, a first protocol, or a second protocol. The operating system may include any system software that manages hardware or software resources for the computing device (101) or computing device (103) and may provide services to applications. The operating system may be, among others, Windows®, Android OS, iOS, Linux, a real-time operating system (RTOS), or an automotive infotainment operating system. For example, the operating system may be a real-time operating system such as VxWorks, PikeOS, eCos, QNX, MontaVista Linux, RTLinux, Windows CE, or another operating system.

[0034] In the embodiments, the computing device (101), computing device (103), processor core (113), or processor core (133) may be implemented in various ways, for example by the computing platform (600) shown in FIG. 6, and may execute instructions stored in a computer-readable medium, for example, the computer-readable medium (702) shown in FIG. 7, or may be pre-configured with logic (for example, with appropriate bit streams, logic blocks, etc.).

[0035] FIGS. 3 through 5 illustrate exemplary sequence diagrams of remote memory operations performed by multiple computing devices in a computing system having a shared memory address space according to various embodiments. FIG. 3 illustrates a sequence diagram of operations for the memory operation MONITOR, FIG. 4 illustrates a sequence diagram of operations for the memory operation MWAIT, while FIG. 5 illustrates a sequence diagram of operations for the merged memory operation MERGE / MWAIT. The sequence diagrams illustrated in FIGS. 3 through 5 are examples of the process (210) or process (220) illustrated in FIGS. 2a-2b. In some embodiments, MONITOR / MWAIT instructions are implemented by various registers for storing control variables, e.g., general-purpose registers, e.g., the EAX, EBX, and ECX registers in Intel® processors, which are stored as control variables TRIGGER.

[0036] FIG. 3 illustrates an exemplary sequence diagram (300) for a remote memory operation MONITOR performed by multiple computing devices, e.g., computing device (301) and computing device (303), in a computing system having a shared memory address space according to various embodiments. The computing device (301) and computing device (303) are connected together by a network switch (307). In the embodiments, the computing device (301), computing device (303), and network switch (307) are examples of the computing device (101), computing device (103), and network switch (107). The computing device (301) includes an application (354), a CHA (357) operating on the processor core of the computing device (301), and a network controller (361), whereas other components of the computing device (301), e.g., the processor core operating the CHA (357), are not illustrated. The computing device (303) includes a CHA (387) and a network controller (371) operating on the processor core of the computing device (303), while other components of the computing device (303) are not shown. Operations performed by the CHA (357) or the CHA (387) are performed by the respective processor cores supporting the CHA.

[0037] In the embodiments, at interaction (311), a memory operation, e.g., MONITOR, is generated by the application (354) along with the memory address @a and transmitted to the CHA (357). At interaction (313), the CHA (357) receives a memory operation instruction, e.g., MONITOR, for operating on the memory address @a in a shared memory address space. The CHA (357) determines that the memory address @a is a remote memory address local to the processor core in the computing device (303). At interaction (315), the CHA (357) generates a remote memory operation request, e.g., RMONITOR, corresponding to the memory operation instruction MONITOR for operating on the remote memory address @a, and transmits the remote memory operation request, e.g., RMONITOR, to the network controller (361) so that it is forwarded to the processor core in the computing device (303). The remote memory operation request RMONITOR includes the memory address @a. In the interaction (317), through the network switch (307), the network controller (361) forwards a remote memory operation request, e.g., RMONITOR, to the network controller (371) at the computing device (303).

[0038] In the embodiments, in interaction (321), the network controller (371) receives a remote memory operation request, e.g., RMONITOR, associated with a remote memory address @a local to the processor core of the computing device (303). In interaction (323), the network controller (371) determines whether there are sufficient resources to fulfill the remote memory operation request, e.g., RMONITOR. The network controller (371) stores multiple remote memory operation requests by being coupled to a local storage, e.g., a local storage (141) for the network device (131), and the network controller (371) determines whether there are sufficient resources to fulfill the remote memory operation request based on the processor core and the available resources for the stored multiple remote memory operation requests. The information stored in the local storage includes the status of the remote memory operation request, the identification of the network controller from which the remote memory operation request is received, and the local memory address associated with the remote memory operation request. For example, Table 1 below illustrates an example of a 4-entry table for managing four outstanding remote MONITOR or other remote memory instructions. As shown in Table 1, only two entries are in use (entries with IDs 0 and 1). The first entry refers to a monitoring request from the network controller (361) for address @a. The network controller (371) has registered a request from node N2 to monitor another address (@b). After execution, the other entries are freed (indicated by N in Used). Table 1 illustrates only an example. For different computing devices (301 and 303), there may be tables of different sizes for storing remote memory operation requests.

[0039]

[0040] In the embodiments, in the interaction (326), after the network controller (371) determines that there are sufficient resources to fulfill a remote memory operation request, e.g., RMONITOR, at address @a, the network controller (371) converts the remote memory address @a to a local memory address and converts the remote memory operation request, e.g., RMONITOR, to a local memory operation, e.g., MONITOR, so that it is operated on the local memory address by the CHA (387). Thus, the network controller (361) mimics the execution of the MONITOR command on the CHA (387) while still allowing the CHA (387) to serve different remote monitoring requests simultaneously.

[0041] Additionally and alternatively, when there are insufficient resources to fulfill an associated remote memory operation request, the network controller (371) notifies the network controller (361) that the remote memory operation request cannot be fulfilled, which is not illustrated. There may be many different ways of notifying the network controller (361) that the remote memory operation request cannot be fulfilled, such as by returning a NOP operation or returning an ERROR message, for example.

[0042] In the embodiments, at interaction (327), the CHA (387) or the processor core operating the CHA (387) configures local memory operation logic based on local memory operation and local memory address and performs MONITOR operation. At interaction (328), the network controller (371) receives a trigger message generated by the local memory operation logic related to the local memory operation and local memory address to indicate the status of the local memory operation and local memory address. At interaction (329), the network controller (371) notifies the network controller (361) regarding the status of the remote memory operation request through the network switch (307).

[0043] Additionally, in interaction (312), the network controller (361) receives a notification from the network controller (371) regarding the status of a remote memory operation request. In interaction (314), the network controller (361) updates the CHA (357) regarding the status of the remote memory operation request received from the network controller (371). In interaction (316), the application (354) receives the status of the remote memory operation request from the network controller (361), where the status is based on a remote memory operation executed on the processor core of the computing device (303).

[0044] FIG. 4 illustrates an exemplary sequence diagram (400) for a remote memory operation MWAIT performed by multiple computing devices, e.g., computing device (401) and computing device (403), in a computing system having a shared memory address space according to various embodiments. The computing device (401) and computing device (403) are coupled together by a network switch (407). In the embodiments, the computing device (401), computing device (403), and network switch (407) are examples of computing device (101), computing device (103), and network switch (107). The computing device (401) includes an application (454), a CHA (457) operating on the processor core of the computing device (401), and a network controller (461), whereas other components of the computing device (401), e.g., the processor core operating the CHA (457), are not illustrated. The computing device (403) includes a CHA (487) and a network controller (471) operating on the processor core of the computing device (403), while other components of the computing device (403) are not shown. Operations performed by the CHA (457) or the CHA (487) are performed by the respective processor cores supporting the CHA.

[0045] In the embodiments, at interaction (411), a memory operation, e.g., MWAIT, is generated by the application (454) along with memory address @a and transmitted to the CHA (457). At interaction (413), the CHA (457) receives a memory operation instruction, e.g., MWAIT, to operate on memory address @a in a shared memory address space. The CHA (457) determines that memory address @a is a remote memory address local to the processor core in the computing device (403). Additionally, at interaction (431), the CHA (457) further determines whether a related remote memory operation, e.g., RMOINTOR, has been executed. If the associated remote memory operation has been executed, in interaction (415), the CHA (357) generates a remote memory operation request, e.g., RMWAIT, corresponding to a memory operation instruction MWAIT for operating on a remote memory address @a, and transmits the remote memory operation request, e.g., RMWAIT, to the network controller (461) so that it is forwarded to the processor core in the computing device (403). The remote memory operation request RMWAIT may include the memory address @a. In interaction (417), through the network switch (407), the network controller (461) forwards the remote memory operation request, e.g., RMWAIT, to the network controller (471) in the computing device (403).

[0046] In the embodiments, in interaction (421), the network controller (471) receives a remote memory operation request, e.g., RMWAIT, associated with a remote memory address @a that is local to the processor core of the computing device (403). In interaction (426), the network controller (471) converts the remote memory address @a to a local memory address and converts the remote memory operation request, e.g., RMWAIT, to a local memory operation, e.g., MWAIT, so that it is operated on the local memory address by the CHA (387). In interaction (427), the CHA (487) or the processor core operating the CHA (487) configures local memory operation logic based on the local memory operation and the local memory address and performs an MWAIT operation to determine whether any external modification to address @a has been performed.

[0047] In the embodiments, at interaction (428), the network controller (471) receives a trigger message generated by local memory operation logic related to the local memory operation and local memory address to indicate the status of the local memory operation and local memory address. At interaction (429), the network controller (471) notifies the network controller (461) about the status of the remote memory operation request through the network switch (407).

[0048] Furthermore, in interaction (412), the network controller (461) receives a notification from the network controller (471) regarding the status of a remote memory operation request. In interaction (414), the network controller (461) updates the CHA (457) regarding the status of the remote memory operation request received from the network controller (471). In interaction (416), the application (454) receives the status of the remote memory operation request from the network controller (461), wherein the status is based on a remote memory operation executed on the processor core of the computing device (403).

[0049] FIG. 5 illustrates an exemplary sequence diagram (500) for remote memory operations MWAIT and MONITOR performed by multiple computing devices, e.g., computing device (501) and computing device (503), in a computing system having a shared memory address space according to various embodiments. The computing device (501) and computing device (503) are coupled together by a network switch (507). In the embodiments, the computing device (501), computing device (503), and network switch (507) are examples of the computing device (101), computing device (103), and network switch (107). The computing device (501) includes an application (554), a CHA (557) operating on the processor core of the computing device (501), and a network controller (561), whereas other components of the computing device (501), e.g., the processor core for operating the CHA (557), are not illustrated. The computing device (503) includes a CHA (587) and a network controller (571) operating on the processor core of the computing device (503), while other components of the computing device (503) are not illustrated. The operation performed by the CHA (557) or the CHA (587) is performed by the respective processor cores supporting the CHA. Unlike the sequence diagram (300) and the sequence diagram (400), the sequence diagram (500) illustrated in FIG. 5 illustrates a remote memory operation request that merges the semantics of MERGE / MWAIT instructions to save network bandwidth by transmitting a single request over the network for two remote memory operations.

[0050] In the embodiments, at interaction (511), a memory operation, e.g., MONITOR, is generated by the application (554) along with the memory address @a and transmitted to the CHA (557). At interaction (531), the CHA (557) waits for a related memory operation, e.g., MWAIT, to be generated. At interaction (532), a related memory operation, e.g., MWAIT, is generated by the application (554) and transmitted to the CHA (557), and the CHA (557) has received both MONITOR and MWAIT by this point. At interaction (513), the CHA (557) receives memory operation instructions, e.g., MONITOR and MWAIT, to operate on the memory address @a in the shared memory address space. The CHA (457) determines that the memory address @a is a remote memory address local to the processor core in the computing device (503). In interaction (515), CHA (557) generates a remote memory operation request, e.g., RMONITOR, corresponding to merged memory operations, MONITOR and MWAIT, to operate on a remote memory address @a, and transmits the remote memory operation request, e.g., RMONITOR, to a network controller (561) so that it is forwarded to a processor core in the computing device (503). In interaction (517), through the network switch (507), the network controller (561) forwards the remote memory operation request, e.g., RMONITOR, to a network controller (571) in the computing device (503).

[0051] In the embodiments, in interaction (521), the network controller (571) receives a remote memory operation request, e.g., RMONITOR, associated with a remote memory address @a that is local to the processor core of the computing device (503). In interaction (533), the network controller (571) determines whether there are sufficient resources to fulfill the remote memory operation request, e.g., RMONITOR. Once the network controller (571) determines that there are sufficient resources to fulfill the remote memory operation request, in interactions (534) and (536), the network controller (571) converts the remote memory address @a to a local memory address and converts the remote memory operation request, e.g., RMONITOR, into multiple local memory operations, e.g., MONITOR and MWAIT, so that they are operated on the local memory address by the CHA (587). As illustrated in FIG. 5, in interaction (535), a MONITOR operation is performed first, and once the MONITOR operation is performed, the CHA (587) notifies the network controller (571). In some other embodiments, an MWAIT operation is performed first before the MONITOR operation. In interaction (527), the CHA (587) or the processor core operating the CHA (587) configures local memory operation logic based on local memory operations and local memory addresses, and performs an MWAIT operation to determine whether any external modification to address @a has been performed.

[0052] In the embodiments, in interaction (528), the network controller (571) receives a trigger message generated by local memory operation logic related to the local memory operation and local memory address to indicate the status of the local memory operation and local memory address. In interaction (529), the network controller (571) notifies the network controller (561) about the status of the remote memory operation request through the network switch (507).

[0053] Additionally, in interaction (512), the network controller (561) receives a notification from the network controller (571) regarding the status of a remote memory operation request. In interaction (514), the network controller (561) updates the CHA (557) regarding the status of the remote memory operation request received from the network controller (571). In interaction (516), the application (554) receives the status of the remote memory operation request from the network controller (561), wherein the status is based on a remote memory operation executed on the processor core of the computing device (503).

[0054] The memory operations MONITOR, MWAIT, and merged MONITOR / MWAIT illustrated in FIGS. 3 through 5 are merely examples. There may be other memory operations that can be extended to remote memory operations. For example, a memory operation capable of reflecting changes at a monitored address may be performed similarly. In some detail, a first application on a first device may place a remote MONITOR via a remote memory address wherever this scope can be implemented. The first application includes the local storage address of the first device along with this remote request to transmit the updated contents at the remote memory address when the remote MWAIT is triggered by a write to the remote memory address. The first application on the first device may not have any idea which application writes to the monitored remote memory address. A network controller at the node to which this remote monitored memory address is mapped generates a transport when triggered by a write to the monitored remote memory address and transmits page data thereto, whichever transport to the first device is. When the network controller in the first device completes the local reflection of a remote memory address, the first application receives notification results that update the contents at the allocated local storage address to reflect the contents updated at the remote memory address. Additionally, multiple applications may operate on the first device to request the same remote memory address to be monitored and updated. The network controller of the first device may automatically update multiple applications operating on the first device.

[0055] FIG. 6 illustrates an exemplary computer device (600) that may be suitable as a device for carrying out selected embodiments of the present disclosure. As illustrated, the computer device (600) has one or more processor cores, each having one or more processors (602) and is associated with one or more hardware accelerators (603) (which may be an ASIC, an FPGA, or a collection of one or more ICs). In alternative embodiments, the hardware accelerator(s) (603) may be part of the processor (602) or may be integrated together on an SOC. Additionally, the computer device (600) may include a memory (604) which may be any one of a plurality of known random access memories, and a mass storage (606) which may be any one of a plurality of permanent storage media. Additionally, the computer device (600) may include input / output devices (608). Furthermore, the computer device (600) may include communication interfaces (614) and a network device (631). The communication interface (614) or network device (631) may be any one of a number of known communication interfaces. The elements may be combined with each other via a system bus (612) that may represent one or more buses. In the case of multiple buses, they may be bridged by one or more bus bridges (not shown).

[0056] In embodiments, one or more processors (602), communication interfaces (614), network device (631), memory (604), or mass storage (606) may each be similar to the network device (131), processor core (133), and memory device (135) as illustrated in FIG. 1. For example, the processors (602) may include a cache (635), a CHA (637), and monitor logic (634). The network device (631) includes a local storage (633) and a network controller (632). In some embodiments, the communication interfaces (614) may be part of the network device (631). The cache (635), CHA (637), monitor logic (634), local storage (633), and network controller (632) may be examples of corresponding parts of the computing device (101) or computing device (103).

[0057] Each of these elements may perform its conventional functions known in the art, or may perform them as described in this disclosure. In various embodiments, the computational logic (622) may implement one or more applications such as an operating system or the process (210) or process (220) shown in FIGS. 2a-2b, or operations shown in the sequence diagram (300), sequence diagram (400), or sequence diagram (500) shown in FIGS. 3 through 5. The computational logic (622) may be implemented by assembler instructions supported by the processor(s) (602), or by high-level languages ​​such as, for example, C, which can be compiled into such instructions.

[0058] The number, capability, and / or capacity of these elements (601-637) may vary depending on the use of the computer device (600). In other cases, the configurations of the elements (601-637) are known and therefore will not be described further.

[0059] As will be known to those skilled in the art, the present disclosure may be implemented as methods or computer program products. Accordingly, in addition to being implemented in hardware as described above, the present disclosure may also include specific software components (including firmware, resident software, microcode, etc.) or embodiments combining software and hardware aspects that may all generally be referred to as "circuits," "modules," or "systems."

[0060] Furthermore, the present disclosure may include a computer program product embodied in any tangible or non-transient medium having computer-readable program code embodied in the medium. FIG. 7 illustrates an exemplary computer-readable non-transient storage medium suitable for use in storing instructions that cause the device to perform selected embodiments of the present disclosure in response to the execution of instructions by the device. As illustrated, the non-transient computer-readable storage medium (702) may include a plurality of programming instructions (704). The programming instructions (704) may be configured to enable the device, e.g., device (600), to perform various operations of various applications, e.g., the operations shown in the process (210) or process (220) illustrated in FIG. 2a-2b, or the operations shown in the sequence diagram (300), sequence diagram (400), or sequence diagram (500) illustrated in FIG. 3 through 5, in response to the execution of programming instructions in a controller or processor.

[0061] In alternative embodiments, programming instructions (704) may instead be placed on multiple computer-readable non-transient storage media (702). In alternative embodiments, programming instructions (704) may be placed on computer-readable transient storage media (702), such as signals. Any combination of one or more computer-readable or computer-readable media(s) may be used. A computer-readable or computer-readable medium may be, for example, an electric, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or radio wave medium, but is not limited thereto. More specific examples of computer-readable media (a list not exhaustive) may include: electrical connections having one or more wires, portable computer diskettes, hard disks, RAM (random access memory), ROM (read-only memory), EPROM (erasable programmable read-only memory) or flash memory, optical fibers, portable CD-ROMs (compact disc read-only memory), optical storage devices, transmitting media such as those supporting the Internet or intranet, or magnetic storage devices. It should be noted that computer-usable or computer-readable media may even be paper or other suitable media on which the program is printed, because a program can be stored in computer memory after being electronically captured, for example, through optical scanning of paper or other media, and then, if necessary, compiled, interpreted, or otherwise processed in a suitable manner. In the context of this document, computer-usable or computer-readable media may be any medium capable of containing, storing, communicating, propagating, or transmitting a program to be used by or in connection with an instruction execution system, device, or device.A computer-usable medium may include a propagated data signal in which computer-usable program code is implemented, either in the baseband or as part of a carrier wave. The computer-usable program code may be transmitted using any suitable medium, including but not limited to wireless, wired, fiber optic cables, RF, etc.

[0062] Computer program code for performing the operations of the present disclosure may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" programming language or similar programming languages. The program code may be executed entirely on a user's computer, partially on a user's computer, as a standalone software package, partially on a user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer through any type of network, including a LAN (local area network) or a WAN (wide area network), or such connection may be made to an external computer (e.g., over the Internet using an ISP (Internet Service Provider)).

[0063] The present disclosure has been described with reference to flowchart examples and / or block diagrams of methods, apparatuses (systems), and computer program products according to embodiments of the present disclosure. It will be understood that each block of the flowchart examples and / or block diagrams, and combinations of blocks in the flowchart examples and / or block diagrams, may be implemented by computer program instructions. Such computer program instructions may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing unit producing a machine, so that instructions executed through the processor of the computer or other programmable data processing unit create means for implementing a function / act specified in a block or in a block of a flowchart and / or block diagram.

[0064] These computer program instructions may also be stored in a computer-readable medium capable of guiding a computer or other programmable data processing device to function in a specific manner, so that the instructions stored in the computer-readable medium produce a manufactured article comprising instruction means that implements a function / act specified in a block or blocks of a flowchart and / or block diagram.

[0065] Computer program instructions can also be loaded onto a computer or other programmable data processing device, causing a series of operational steps to be performed on the computer or other programmable device to produce a computer implementation process, thereby enabling instructions executed on the computer or other programmable device to provide processes for implementing functions / acts specified in blocks or blocks of a flowchart and / or block diagram.

[0066] The flowcharts and block diagrams in the drawings illustrate the architecture, function, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowcharts or block diagrams may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function(s). It should also be noted that in some alternative implementations, functions represented by blocks may occur in an order different from the order shown in the drawings. For example, two blocks shown consecutively may, in fact, be executed substantially simultaneously or, at times, in reverse order, depending on the associated functions. It should also be noted that each block in the block diagrams and / or flowchart examples, and combinations of blocks in the block diagrams and / or flowchart examples, may be implemented by special-purpose hardware-based systems that perform the specified functions or acts, or by combinations of special-purpose hardware and computer instructions. As used in this specification, "computer-implemented method" may refer to any method executed by one or more processors, a computer system having one or more processors, a mobile device such as a smartphone (which may include one or more processors), a tablet, a laptop computer, a set-top box, a game console, etc.

[0067] The embodiments may be implemented as a computer process, a computing system, or as a manufactured article such as a computer program product on a computer-readable medium. The computer program product may be a computer storage medium that is readable by a computer system and encodes computer program instructions for executing a computer process.

[0068] The corresponding structures, materials, acts, and equivalents of all means or steps plus functional elements in the following claims are intended to include any structures, materials, or acts for performing a function in combination with other specifically claimed elements. The description of this disclosure is provided for illustrative and illustrative purposes, but is not intended to be incomplete or limited only to the disclosed form. Many modifications and changes will be apparent to those skilled in the art without departing from the scope and spirit of this disclosure. The embodiments are selected and described to best illustrate the principles and practical applications of this disclosure and to enable other people skilled in the art to understand this disclosure in relation to various embodiments with various modifications suitable for the specific use considered.

[0069] Accordingly, various exemplary embodiments of the present disclosure have been described, including but not limited to the following:

[0070] Example 1 may include a device, the device includes a first network controller coupled to a first processor core, and the first network controller: processes a remote memory operation request - the remote memory operation request is generated by a first memory coherence agent (CHA) based on a first memory operation for an application running on the first processor core, and the remote memory operation request is associated with a remote memory address local to a second processor core coupled to the first processor core via a network switch -; and forwards the remote memory operation request to a second network controller coupled to the second processor core via a network switch, wherein the second processor core and the second network controller perform a second memory operation at a remote memory address local to the second processor core, and the second memory operation corresponding to the remote memory operation request is generated based on the first memory operation.

[0071] Example 2 may include the device of Example 1 and / or some other examples of the present specification, wherein a remote memory operation request includes a remote memory address in a shared memory address space.

[0072] Example 3 may include the device of Example 1 and / or some other examples of the present specification, wherein the remote memory operation request is a request for a remote memory monitor operation, a remote memory standby operation, or a merged remote memory monitor and standby operation.

[0073] Example 4 may include the device of Example 1 and / or some other examples of this specification, wherein the device is a host fabric interface (HFI) or a network interconnect card (NIC).

[0074] Example 5 may include the device of Example 1 and / or some other examples of the present specification, wherein the first network controller additionally: receives a notification from the second network controller regarding the status of a remote memory operation request; and updates the first memory CHA regarding the status of the remote memory operation request received from the second network controller.

[0075] Example 6 may include the device of Example 1 and / or some other examples of the present specification, wherein the remote memory operation request is the first remote memory operation request and the remote memory address is the first remote memory address, and the first network controller additionally: receives a second remote memory operation request associated with a second remote memory address local to the first processor core from a third network controller coupled to the first network controller through a network switch; and determines whether there are sufficient resources to fulfill the second remote memory operation request.

[0076] Example 7 may include the device of Example 6 and / or some other examples of the present specification, wherein the first network controller additionally: converts the second remote memory address to a local memory address for the first processor core.

[0077] Example 8 may include the device of Example 7 and / or some other examples of the present specification, wherein the first network controller additionally: if there are sufficient resources to fulfill the second remote memory operation request, converts the second remote memory operation request into a local memory operation to be performed on a local memory address by the first memory CHA.

[0078] Example 9 may include the device of Example 8 and / or some other examples of the present specification, wherein the first network controller further: configures local memory operation logic based on local memory operation and local memory address; receives a trigger message generated by local memory operation logic associated with local memory operation and local memory address to indicate the status of a second remote memory operation request; and notifies a third network controller regarding the status of the second remote memory operation request.

[0079] Example 10 may include the device of Example 9 and / or some other examples of the present specification, wherein a trigger message is generated by local memory operation logic when there is another local memory operation to be performed on a local memory address.

[0080] Example 11 may include the device of Example 10 and / or some other examples of the present specification, wherein another local memory operation is performed on a local memory address by another application running on the first processor core or by another remote memory operation request.

[0081] Example 12 may include the device of Example 6 and / or some other examples of the present specification, wherein the device further includes a local storage for storing multiple remote memory operation requests, and the first network controller determines whether there are sufficient resources to fulfill a second remote memory operation request based on the first processor core and available resources for the stored multiple remote memory operation requests, wherein the information stored in the local storage includes the status of the remote memory operation request, the identification of the network controller from which the remote memory operation request is received, and a local memory address associated with the remote memory operation request.

[0082] Example 13 may include the device of Example 12 and / or some other examples of the present specification, wherein the first network controller further: responds to the third network controller that, based on information stored in a local storage for multiple remote memory operation requests, there are not enough resources to fulfill a second remote memory operation request associated with a second remote memory address local to the first processor core.

[0083] Example 14 may include a method for a first processor core, the method comprising: receiving a memory operation instruction for operating on a memory address in a shared memory address space - the memory operation instruction is for an application operating on the first processor core -; determining that the memory address is a remote memory address local to the second processor core; generating a remote memory operation request corresponding to the memory operation instruction for operating on the remote memory address; and transmitting the remote memory operation request to a network controller to forward it to the second processor core.

[0084] Example 15 may include the method of Example 14 and / or some other examples of the present specification, and the method further includes: determining that a memory address is a local memory address for a first processor core; and executing a memory operation instruction by the first processor core.

[0085] Example 16 may include the method of Example 14 and / or some other examples of the present specification, wherein the remote memory operation request is a request for a remote memory monitor operation, a remote memory standby operation, or a merged remote memory monitor and standby operation.

[0086] Example 17 may include the method of Example 14 and / or some other examples of the present specification, wherein instructions further cause a first processor core to: receive the status of a remote memory operation request from a network controller in response to execution by the first processor core - the status is based on a remote memory operation executed on a second processor core, and the remote memory operation corresponds to a remote memory operation request -; and report the result of a memory operation instruction for an application based on the status of the remote memory operation request.

[0087] Example 18 may include the method of Example 14 and / or some other examples of the present specification, wherein the memory operation instruction is the first memory operation instruction, the remote memory operation request is the first remote memory operation request, the remote memory address is the first remote memory address, and the instructions further cause the first processor core to: receive a second memory operation from a network controller in response to execution by the first processor core - the second memory operation corresponds to a second remote memory operation request received by the network controller, and the second memory operation is associated with a second local memory address -; perform a second memory operation on the second local memory address; and notify the network controller of the status of the second memory operation.

[0088] Example 19 may include the method of Example 18 and / or some other examples of the present specification, wherein performing a second memory operation comprises: configuring local memory operation logic based on the second memory operation and the second local memory address; and receiving a trigger message generated by the local memory operation logic associated with the second memory operation and the second local memory address to indicate the status of a second remote memory operation request.

[0089] Example 20 may include the method of Example 19 and / or some other examples of the present specification, wherein a trigger message is generated by local memory operation logic when there is another local memory operation to be performed on a second local memory address.

[0090] Example 21 may include the method of Example 20 and / or some other examples of the present specification, wherein another local memory operation is performed on a second local memory address by another application running on the first processor core or by another remote memory operation request.

[0091] Example 22 may include a computing device, wherein the computing device comprises a first processor core—the first processor core generates a remote memory operation request based on a first memory operation for an application running on the first processor core, and the remote memory operation request is associated with a remote memory address local to a second processor core coupled to the first processor core via a network switch—; and a first network controller coupled to the first processor core and the network switch, wherein the first network controller processes the remote memory operation request generated by the first processor core and forwards the remote memory operation request to a second network controller coupled to the second processor core via a network switch, wherein the second processor core and the second network controller perform a second memory operation that extends the first memory operation through the network switch, the first network controller, and the second network controller.

[0092] Example 23 may include the computing device of Example 22 and / or some other examples of the present specification, wherein the remote memory operation request includes a remote memory address in a shared memory address space.

[0093] Example 24 may include the computing device of Example 22 and / or some other examples of the present specification, wherein the remote memory operation request is a request for a remote memory monitor operation, a remote memory standby operation, or a merged remote memory monitor and standby operation.

[0094] Example 25 may include the computing device of Example 22 and / or some other examples of the present specification, wherein a remote memory operation request is generated by a memory coherence agent (CHA) operating on a first processor core, wherein the first network controller further receives a notification regarding the status of the remote memory operation request from a second network controller, and updates the CHA regarding the status of the remote memory operation request received from the second network controller.

[0095] It will be apparent to those skilled in the art that various modifications and variations may be made to the disclosed embodiments of the disclosed devices and related methods without departing from the spirit or scope of the present disclosure. Accordingly, the present disclosure is intended to cover such modifications and variations of the embodiments disclosed above, provided that such modifications and variations fall within the scope of any claims and their equivalents.

Claims

Claim 1 A device comprising: a first network controller coupled to a first processor core, wherein the first network controller: processes a remote memory operation request, wherein the remote memory operation request is generated by a first memory coherence agent (CHA) based on a first memory operation for an application running on the first processor core, and the remote memory operation request is associated with a remote memory address local to a second processor core coupled to the first processor core via a network switch; and forwards the remote memory operation request to a second network controller coupled to the second processor core via the network switch, wherein the second processor core and the second network controller perform a second memory operation at a remote memory address local to the second processor core, and the second memory operation corresponding to the remote memory operation request is generated based on the first memory operation, wherein the remote memory operation request includes the remote memory address in a shared memory address space. Claim 2 delete Claim 3 In paragraph 1, the device wherein the remote memory operation request is a request for a remote memory monitor operation, a remote memory standby operation, or a merged remote memory monitor and standby operation. Claim 4 In claim 1, the device is a host fabric interface (HFI) or a network interconnect card (NIC). Claim 5 In claim 1, the first network controller further comprises: receiving a notification regarding the status of the remote memory operation request from the second network controller; and a device for updating the first memory CHA regarding the status of the remote memory operation request received from the second network controller. Claim 6 In any one of claims 1 and 3 through 5, the remote memory operation request is a first remote memory operation request, the remote memory address is a first remote memory address, and the first network controller further: receives a second remote memory operation request associated with a second remote memory address local to the first processor core from a third network controller coupled to the first network controller through the network switch; and a device for determining whether there are sufficient resources to fulfill the second remote memory operation request. Claim 7 In paragraph 6, the first network controller further comprises: a device that converts the second remote memory address into a local memory address for the first processor core. Claim 8 In claim 7, the first network controller further comprises: a device that, if there are sufficient resources to fulfill the second remote memory operation request, converts the second remote memory operation request into a local memory operation to be performed on the local memory address by the first memory CHA. Claim 9 In claim 8, the first network controller further comprises: configuring local memory operation logic based on the local memory operation and the local memory address; receiving a trigger message generated by the local memory operation logic associated with the local memory operation and the local memory address to indicate the status of the second remote memory operation request; and notifying the third network controller regarding the status of the second remote memory operation request. Claim 10 In claim 9, the trigger message is a device generated by the local memory operation logic when there is another local memory operation to be performed on the local memory address. Claim 11 In paragraph 10, the device wherein the other local memory operation is performed on the local memory address by another application operating on the first processor core or by another remote memory operation request. Claim 12 In claim 6, the device further comprises a local storage for storing multiple remote memory operation requests, and the first network controller determines whether there are sufficient resources to fulfill the second remote memory operation request based on the first processor core and available resources for the stored multiple remote memory operation requests, and the information stored in the local storage includes the status of the remote memory operation request, the identification of the network controller from which the remote memory operation request is received, and a local memory address associated with the remote memory operation request. Claim 13 In paragraph 12, the first network controller further: responds to the third network controller that, based on information stored in the local storage for the multiple remote memory operation requests, there are insufficient resources to fulfill the second remote memory operation request associated with the second remote memory address local to the first processor core. Claim 14 A method for a first processor core comprising: receiving a memory operation instruction for operating on a memory address in a shared memory address space, wherein the memory operation instruction is for an application operating on the first processor core; determining that the memory address is a remote memory address local to a second processor core; generating a remote memory operation request corresponding to the memory operation instruction for operating on the remote memory address; and transmitting the remote memory operation request to a network controller so as to be forwarded to the second processor core, wherein the remote memory operation request includes the remote memory address in a shared memory address space. Claim 15 A method according to claim 14, further comprising: a step of determining that the memory address is a local memory address for the first processor core in response to the step of receiving the memory operation instruction; and a step of executing the memory operation instruction by the first processor core. Claim 16 In paragraph 14, the method wherein the remote memory operation request is a request for a remote memory monitor operation, a remote memory standby operation, or a merged remote memory monitor and standby operation. Claim 17 In paragraph 14, the above instructions further cause the first processor core to: receive the status of the remote memory operation request from the network controller in response to execution by the first processor core—the status being based on a remote memory operation executed on the second processor core, and the remote memory operation being in response to the remote memory operation request—; and further cause the first processor core to report the result of the memory operation instruction for the application based on the status of the remote memory operation request. Claim 18 In any one of claims 14 to 17, the memory operation instruction is a first memory operation instruction, the remote memory operation request is a first remote memory operation request, the remote memory address is a first remote memory address, and the instructions are: causing the first processor core to, in response to execution by the first processor core: receive a second memory operation from the network controller, wherein the second memory operation corresponds to a second remote memory operation request received by the network controller, and the second memory operation is associated with a second local memory address; perform the second memory operation on the second local memory address; and further cause the network controller to be notified of the status of the second memory operation. Claim 19 A method according to claim 18, wherein performing the second memory operation comprises: configuring local memory operation logic based on the second memory operation and the second local memory address; and receiving a trigger message generated by the local memory operation logic associated with the second memory operation and the second local memory address to indicate the status of the second remote memory operation request. Claim 20 In paragraph 19, the trigger message is generated by the local memory operation logic when there is another local memory operation to be performed on the second local memory address. Claim 21 In paragraph 20, the method wherein the other local memory operation is performed on the second local memory address by another application operating on the first processor core or by another remote memory operation request. Claim 22 A computing device comprising: a first processor core - the first processor core generates a remote memory operation request based on a first memory operation for an application running on the first processor core, and the remote memory operation request is associated with a remote memory address local to a second processor core coupled to the first processor core via a network switch -; and a first network controller coupled to the first processor core and the network switch - the first network controller processes the remote memory operation request generated by the first processor core and forwards the remote memory operation request to a second network controller coupled to the second processor core via the network switch, and the second processor core and the second network controller perform a second memory operation extending the first memory operation via the network switch, the first network controller, and the second network controller - wherein the remote memory operation request includes the remote memory address in a shared memory address space. Claim 23 delete Claim 24 In paragraph 22, the above-mentioned remote memory operation request is a request for a remote memory monitor operation, a remote memory standby operation, or a merged remote memory monitor and standby operation, for a computing device. Claim 25 A computing device according to claim 22 or 24, wherein the remote memory operation request is generated by a memory coherence agent (CHA) operating on the first processor core, the first network controller further receives a notification regarding the status of the remote memory operation request from the second network controller, and updates the CHA regarding the status of the remote memory operation request received from the second network controller.

Images