Adaptive Memory Consistency in Distributed Data Centers

Abstract

The data processor includes a fabric-attached memory (FAM) interface for coupling to the data fabric and executing memory access instructions. A requester-side adaptive coherency controller coupled to the FAM interface requests notifications from a fabric manager of the fabric-attached memory regarding changes in requesters authorized to access the FAM region that the data processor is authorized to access. If the notifications indicate that two or more requesters are authorized to access the FAM region, a fence is activated for a selected memory access instruction in a local application.

Description

[Background technology]

[0001] Emerging fabric standards such as Compute Express Link (CXL) 3.0, Gen-Z, or Slingshot, illustrate an approach to datacenter decentralization in which central processing unit (CPU) hosts can access Fabric-Attached Memory (FAM) modules. Such modules contain memory attached to the datacenter fabric and have little or no computational power associated with them. With FAM, hosts are not constrained by the memory capacity limitations of their local servers. Instead, hosts can access vast pools of memory that do not need to be attached to any particular host. FAM is partitioned among hosts, and partitions may be dedicated to one host or shared among multiple hosts.

[0002] Memory consistency is an important consideration in the process of developing software applications for use with a FAM system. Consistency defines how memory instructions (to different memory locations) in a multiprocessor or multithreaded system are ordered, and is performed by reordering independent memory operations according to a consistency model.

[0003] Various consistency models have been developed that impose different ordering constraints on independent memory operations in a single processor's instruction stream where high levels of dependencies are involved. In a simple consistency model known as Sequential Consistency, the processor cannot reorder reads and writes. Another model, known as "Total Store Order" (TSO), allows for store buffering. In this scheme, a store buffer holds store operations that need to be sent to memory until a specified condition is met and the group of operations can be sent to memory. Stores are allowed to go through, but stores are sent to memory in program order. The address of the load operation is checked against addresses in the store buffer, and the store buffer is used to satisfy the load operation if there is an address match.

[0004] Other consistency models, known as relaxed or weak consistency models, rely on some version of a fence (or barrier) operation that defines an area within which reordering of operations is permitted. Release consistency is an example of a weak consistency model, where synchronized accesses are divided into "Acquire", where an operation such as locking must complete before all subsequent memory accesses, and "Release", where an operation such as unlocking must complete along with all memory operations before release can complete. [Brief description of the drawings]

[0005] [Figure 1] 1 is a block diagram of a fabric attached memory (FAM) system according to the prior art. [Diagram 2] FIG. 2 is a block diagram of an accelerated processing unit (APU) suitable for use as a computational unit in an FAM system such as FIG. 1, according to some embodiments. [Diagram 3]1 is a block diagram of certain elements of a FAM memory system including an adaptive consistency controller (ACC), in accordance with some embodiments. [Figure 4] FIG. 4 illustrates a process for operating the ACC of FIG. 3 according to some embodiments. [Diagram 5] FIG. 1 is a flow diagram of a process for managing a memory consistency model in an adaptive consistency controller, in accordance with an example embodiment. [Figure 6] FIG. 1 is a flow diagram of a process for enforcing a memory consistency model, according to some embodiments. [Figure 7] FIG. 13 is a flow diagram of another process for enforcing a memory consistency model, in accordance with some additional embodiments. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0006] In the following description, the use of the same reference numbers in different figures indicates similar or identical items. Unless otherwise noted, the word "coupled" and its related verb forms include both direct and indirect electrical connections by means known in the art, and unless otherwise noted, any description of a direct connection also refers to an alternative embodiment using a suitable form of indirect electrical connection.

[0007] The method is for use with a fabric-attached memory system including a fabric-attached memory and a plurality of requesters coupled to the fabric-attached memory via a fabric. Notification is requested from a fabric manager regarding a change in requesters authorized to access the fabric-attached memory region. In response to notification from the fabric manager indicating that two or more requesters are authorized to access the fabric-attached memory region, a fence is activated for each such authorized requester for a selected memory access instruction in a local application related to the fabric-attached memory region.

[0008] The data processor includes a processing core, a fabric-attached memory interface, and a requester-side adaptive coherence controller. The processing core executes an application. The fabric-attached memory interface is coupled to the processing core and adapted to connect to a data fabric and execute memory access instructions from the processing core to the fabric-attached memory. The requester-side adaptive coherence controller is coupled to the processing core and the fabric-attached memory interface and requests notifications from a fabric manager for the fabric-attached memory regarding changes in requesters authorized to access a fabric-attached memory region that the data processor is authorized to access. In response to a notification from the fabric manager indicating that two or more requesters are authorized to access the fabric-attached memory region, the requester-side adaptive coherence controller activates a fence for a selected memory access instruction in the local application.

[0009] The fabric-attached memory system includes a fabric-attached memory, a data fabric, a fabric manager, and a plurality of data processors. The data fabric is connected to the fabric-attached memory. The fabric manager is connected to the data fabric and is operable to grant and de-authorize requesters to access memory regions of the fabric-attached memory. A plurality of data processors are connected to the data fabric, each including a processing core executing an application, a fabric-attached memory interface, and a requester-side adaptive consistency controller coupled to the processing core and the fabric-attached memory interface. The requester-side adaptive consistency controller requests notification from the fabric manager regarding a change in requesters that are permitted to access the fabric-attached memory regions that the data processors are permitted to access. In response to notification from the fabric manager indicating that two or more requesters are permitted to access the fabric-attached memory regions, the requester-side adaptive consistency controller activates a fence for selected memory access instructions in the local application.

[0010] 1 is a block diagram of a prior art fabric-attached memory (FAM) system 100. The illustrated FAM system 100 is only one example of a data fabric topology among many topologies often used in distributed data centers. The FAM system 100 generally includes a data center fabric 102 and a number of device groups 104 called pods.

[0011] Each pod 104 includes multiple compute nodes "C", multiple memory nodes "M", and an interconnect network "ICN". The compute nodes C are connected to the ICN via router "R". The compute nodes C include multiple CPUs (each with multiple cores) or multiple accelerated processing units (APUs) that are part of the same consistency domain. Each compute node C includes a fabric bridge, such as a network interface card (NIC), CXL interface, or other suitable fabric interface that is a gateway to the data center fabric 102 for the compute nodes C. The memory nodes M are connected to the ICN via router R. Each memory node M includes a similar fabric interface and a media controller that services the requirements for the FAM. The ICN includes switches to interconnect the various compute nodes C with the memory nodes M, and may include routers in some topologies.

[0012] The illustrated topology includes a local data center fabric formed by routers R and ICN, and a global data center fabric, labeled data center fabric 102. In this embodiment, the local data center fabric is within a rack, and the global data center fabric includes multiple racks. However, various fabric topologies may be implemented within a rack or within a data center, and may include compute nodes that access the data center remotely over a network. Note that many topologies have a compute node C that also includes memory that is part of the FAM pool. Such memory may be mapped as fabric-attached memory and made available for use by other compute nodes according to a resource allocation process called "composability."

[0013] The data center fabric 102 provides data interconnection between the pods 104, including switches and routers that combine data traffic over a protocol such as CXL, Gen-Z, or other suitable memory fabric protocol. Note that multiple protocols may be used together in a data center fabric. In this exemplary embodiment, CXL is used to interconnect devices within a rack, and Gen-Z is used to interconnect various racks within the data center.

[0014] Figure 2 is a block diagram of an APU 200 suitable for use as a compute unit C in a FAM system, such as FAM system 100 of Figure 1. APU 200 is an integrated circuit suitable for use as a processor in a host data processing system and generally includes a central processing unit (CPU) core complex 210, a graphics core 220, a set of display engines 222, a data fabric 225, a memory management hub 240, a set of peripheral controllers 260, a set of peripheral bus controllers 270, a system management unit (SMU) 280, and a group of memory interfaces 290.

[0015] CPU core complex 210 includes CPU core 212 and CPU core 214. In this example, CPU core complex 210 includes two CPU cores, but in other embodiments, CPU core complex 210 may include any number of CPU cores. Each of CPU cores 212, 214 is bidirectionally connected to a system management network (SMN) forming a control fabric and to a local data fabric 225, and may provide memory access requests to data fabric 225. Each of CPU cores 212, 214 may be a single core, or may further be a core complex having two or more single cores that share certain resources, such as caches. Each of CPU cores 212, 214 includes μCode 216 that operates to execute certain instructions on the CPU, including implementing certain functions for memory consistency on the data center fabric, as described further below.

[0016] Graphics core 220 is a high-performance graphics processing unit (GPU) capable of performing graphics operations such as vertex processing, fragment processing, shading, texture blending, etc. in a highly integrated and parallel manner. Graphics core 220 is bidirectionally connected to SMN and data fabric 225 and can provide memory access requests to data fabric 225. In this regard, APU 200 can support either a unified memory architecture in which CPU core complex 210 and graphics core 220 share the same memory space, or a memory architecture in which CPU core complex 210 and graphics core 220 share a portion of the memory space, but graphics core 220 also uses private graphics memory that is not accessible by CPU core complex 210. The memory area may be allocated from local memory or from a data center fabric.

[0017] The display engine 222 renders and rasterizes objects generated by the graphics core 220 for display on a monitor. The graphics core 220 and the display engine 222 are bidirectionally connected to a common memory management hub 240 for uniform translation to appropriate addresses in system memory.

[0018] Local data fabric 250 includes a crossbar switch for routing memory access requests and responses between any memory accessing agent and memory management hub 240. The data fabric also includes a system memory map defined by the basic input / output system (BIOS) as well as buffers for each virtual connection to determine the destination of memory accesses based on the system configuration.

[0019] Peripheral controllers 260 include a universal serial bus (USB) controller 262 and a Serial Advanced Technology Attachment (SATA) interface controller 264, each of which is bidirectionally connected to a system hub 266 and the SMN bus. These two controllers are merely exemplary of peripheral controllers that may be used in APU 200.

[0020] Peripheral bus controllers 270 include a system controller or "Southbridge" (SB) 272 and a Peripheral Component Interconnect Express (PCIe) controller 274, each of which is bidirectionally connected to an input / output (I / O) hub 276 and the SMN bus. I / O hub 276 is also bidirectionally connected to system hub 266 and data fabric 225. Thus, for example, a CPU core can program registers in USB controller 262, SATA interface controller 264, SB 272, or PCIe controller 274, with accesses routed by data fabric 225 through I / O hub 276. Software and firmware for APU 200 is stored in a system data drive or system BIOS memory (not shown), which may be any of a variety of non-volatile memory types, such as read-only memory (ROM), flash, electrically erasable programmable ROM (EEPROM), etc. Typically, the BIOS memory is accessed via the PCIe bus, and the system data drives are accessed via the SATA interface.

[0021] The SMU 280 is a local controller that controls the operation of resources on the APU 200 and synchronizes communication between them. The SMU 280 manages the power-up sequencing of the various processors on the APU 200 and controls multiple off-chip devices via reset, enable, and other signals. The SMU 280 includes one or more clock sources (not shown), such as a phase locked loop (PLL), to provide clock signals to each of the components of the APU 200. The SMU 280 also manages power for the various processors and other functional blocks and can receive measured power consumption values ​​from the CPU cores 212, 214 and the graphics core 220 to determine appropriate power states.

[0022] Memory management hub 240 is connected to local data fabric 250, graphics core 220 and display engine 230 and provides direct memory access capabilities to graphics core 220 and display engine 230.

[0023] Memory interface 290 includes two memory controllers 291, 292, DRAM media 293, 294, and FAM memory interface 295. Each of memory controllers 291, 292 is connected to local data fabric 250 and connected to each of DRAM media 293, 294 via a physical layer (PHY) interface. In this embodiment, DRAM media 293, 294 include memory modules based on DDR memory, such as DDR version 5 (DDR5). In other embodiments, other types of DRAM memory, such as low power DDR4 (LPDDR4), graphics DDR version 5 (GDDR5), and high bandwidth memory (HBM), are used.

[0024] FAM memory interface 295 includes fabric bridge 296, adaptive coherency controller (ACC) 297, and fabric PHY 298. Fabric bridge 296 is a fabric-attached memory interface connected to local data fabric 250 for receiving and fulfilling memory requests to a FAM system, such as FAM system 100. Such memory requests may come from CPU core complex 210 or may be direct memory access (DMA) requests from other system components, such as graphics core 220. Fabric bridge 296 is also bidirectionally connected to fabric PHY 298, providing APU 200's connection to the data center fabric. Adaptive coherency controller (ACC) 297 is bidirectionally connected to fabric bridge 296 and provides memory coherency control inputs to fabric bridge 296 and CPU core complex 210, as described further below. During operation, the ACC 297 communicates with the CPU cores in the CPU core complex 210 to receive notification that a specified memory access instruction has been recognized by μCode 216 executing on the CPU cores 212, 214, as described further below. The ACC 297 also provides configuration inputs to the CPU core complex 210 for configuring a memory consistency model.

[0025] 3 is a block diagram of certain elements of a FAM system 300, in accordance with some embodiments. The FAM system 300 generally includes compute nodes 302, a data center fabric 102, a FAM memory node 310, and a fabric manager 320.

[0026] Compute node 302 may be any of many requester compute nodes connected to data center fabric 102 and is typically implemented with an APU, such as APU 200. Compute node 302 may implement an internet server, an application server, a supercomputing node, or another suitable compute node that would benefit from accessing a FAM. To focus on the relevant portions of the system, only the FAM interface components of compute node 302 are illustrated. Compute node 302 includes fabric bridge 296, fabric PHY 298, and ACC 297.

[0027] Fabric bridge 296 is connected to a local data fabric, fabric PHY 298, and ACC 297 as described above with respect to Figure 2. This version of ACC 297 includes a microcontroller (μC) 304. μC 304 performs memory coherency control functions as described below, and is typically also connected to tangible, non-transitory memory for holding firmware that initializes and configures μC 304 to perform its functions.

[0028] μC 304 performs memory consistency control functions as described below, and is typically also connected to tangible, non-transitory memory for holding firmware that initializes μC 304 and configures it to perform its functions.

[0029] The fabric manager 320 is a controller connected to the data center fabric 102 for managing configuration and access to the FAM system 300. The fabric manager 320 executes a data fabric management application for the particular standard used in the data center fabric 102, such as CXL or Gen-Z. The data fabric management application manages and configures data center fabric functions such as authorizing compute nodes, allocating memory space, and managing configurability by identifying and configuring memory resources among various nodes in the FAM system 300. It should be noted that while one FAM memory node 310 and one compute node 302 are shown, the system includes multiple such nodes that may appear in many configurations, such as the exemplary configuration shown in FIG. 1. In some embodiments, the fabric manager 320 has an adaptive consistency controller (ACC) module 297 installed to access the data fabric management application and report data to each ACC 322 on each compute node throughout the FAM system 300.

[0030] The FAM memory node 310 includes a media controller 312, a memory 314. The media controller 312 typically includes a memory controller appropriate for whatever type of memory is selected for use in the memory 314. For example, if the memory 314 is a DRAM memory, a DRAM memory controller is used. The memory 314 may also include persistent memory modules or may be mixed. In some embodiments, the ACC 297 maintains data in a requestor table 306 in the FAM memory node 310 relating to other requestors that are permitted to access the FAM memory region assigned to the compute node 302. The requestor table 306 tracks updates to compute nodes that are permitted to access the same memory region, and includes fields for a "timestamp" reflecting the time of the update, a "region ID" reflecting an identifier for the FAM memory region assigned to the compute node 302, and a "number of requestors" reflecting the number of requestors on the FAM system 300 that were permitted to access the memory region at the time of each update. The requestor table 306 is updated based on reports from the fabric manager 320, as described further below. In this embodiment, the FAM memory node 310 includes a buffer accessible by the media controller 312 that holds the requestor table 306.

[0031] 4 is a diagram 400 illustrating a process for operating an ACC 297, according to some embodiments. Diagram 400 illustrates activity at a requester compute node in a data fabric, showing a CPU core complex 210, an operating system 402, an ACC 297, and a fabric bridge 296 at the requester compute node. Diagram 400 also illustrates a fabric manager 320 and a media controller 312, which may be any of many media controllers on the data fabric.

[0032] When a requester is assigned a particular FAM region to use for system memory by the fabric manager 320, the ACC 297 makes a callback request to the fabric manager 320 to request notification if a change is made to the number of requesters permitted to use the same memory region as the requester compute node, as indicated by the outgoing request labeled "callback." The fabric manager 320 returns a notification to the ACC 297 each time the number of requesters permitted to use the memory region changes, as indicated by the "number of users" response on diagram 400. In this embodiment, the requester table 306 (FIG. 3) in the FAM memory node 310 is updated each time the number of users changes, as indicated by the "number of users" arrow pointing to the media controller 312 on diagram 400. In some embodiments, the ACC 287 maintains the requester table in a local buffer of the ACC 297. In some embodiments, the ACC module 322 (FIG. 3) executing in the fabric manager 320 manages the process of monitoring the requesters that are permitted to access the memory region and sending notifications to the ACC 297. Based on the user count update notifications, the ACC 297 sets the consistency model for use with the allocated FAM memory region. Generally, this process involves setting the consistency model to a first consistency model if the current requester is the only requester permitted to access the memory region, and setting the consistency model to a second consistency model if more than one requester is permitted to access the memory region. Setting the consistency model is indicated by the command "SET MODEL(FAMi)=(WB,FENCED)", where "FAMi" identifies the relevant fabric-attached memory region and "WB,FENCED" indicates which consistency model is activated. In some embodiments, the first consistency mode is characterized as relaxed relative to the second consistency model. An example of this process is further described below with respect to FIG. 5.

[0033] In diagram 400, μCode 216 (FIG. 2) executing in CPU core complex 210 serves to enforce the second consistency model when the second consistency model is active. In particular, μCode 216 recognizes a specified memory access instruction in an application executing in CPU core complex 210 that indicates that data fencing is required for a specified memory instruction that pertains to a fabric-attached memory region assigned to a requester node. μCode 216 has various methods for recognizing the specified instruction, as further described below with respect to FIGS. 6 and 7. Upon recognition of such an instruction, μCode 216 communicates with ACC 297 to send a notification identifying the selected instruction, which is sent to the data fabric via fabric bridge 296. ACC 297 then adds a fence command to the command stream going to media controller 312, as indicated by the outgoing arrow labeled "fence." If the first consistency model is active, μCode 216 does not provide such notification, but instead allows the selected command to execute normally with memory consistency handled by the first consistency model set by operating system 402. In this embodiment, μCode 216 recognizes the specified instruction, although in other embodiments this function is performed by the CPU firmware or a combination of CPU firmware and μCode.

[0034] Although the diagram shows the fence command directed to the media controller 312, in a topology including a local data center fabric and a global data center fabric, the ACC 297 causes the fence command to be sent to the media controllers in both the local data center fabric and the global data center fabric in scenarios involving a memory region accessing both levels of the fabric topology.

[0035] 5 is a flow diagram 500 of a process for managing a memory consistency model in an adaptive consistency controller, according to an example embodiment. The process begins at block 502, where a requester node on the data fabric is authorized to access a specified FAM memory region. Such authorization is typically provided by a fabric manager, but in some embodiments may be configured by other system components. Based on the authorization, a memory region is established within the addressable system memory space of the requester node. The memory region is typically used for applications running on the requester node that may have dependencies with other requester nodes.

[0036] At block 504, the ACC 297 of the requester node requests notification from the fabric manager regarding any changes in the number of requesters allowed to access a particular memory region. In one embodiment, this request has the form of a callback request to the fabric manager to track the number of computational requesters accessing the FAM region. The ACC module 322 (FIG. 3) may be used to receive such requests and either execute the requests in the fabric manager or configure the fabric manager to execute the requests. In other embodiments, the fabric manager may have such capabilities as part of the fabric manager application and may not require an additional module.

[0037] In block 506, the ACC 297 of the requester node receives a notification from the fabric manager in response to the request. Based on the notification, the ACC 297 determines the number of requesters currently authorized to use the FAM region and updates the requester table 306 (FIG. 3). In some embodiments, the notification includes data fields used in the requester table 306, including a timestamp, a region ID, and the number of requesters. In other embodiments, the fabric manager may not provide all the data, but instead only data indicating that a requester permission has been added or removed from the FAM region. In such a case, the ACC 297 updates the data in the requester table 306 based on the current notification data and previous updates to the requester table 306.

[0038] At block 508, if more than one requestor is permitted for the memory region, the process proceeds to block 510. Otherwise, the process proceeds to block 512. At block 510, the process activates a fence for a selected memory access instruction in a local application that pertains to the FAM region. In some embodiments, for each requestor so permitted, if the requestor is configured to issue a local fence command, the process includes causing a local fence command for the selected memory access instruction to be issued and mapping the fabric-attached memory region as a write-through region for that requestor, and if the requestor is not configured to issue a local fence command, the process includes mapping the fabric-attached memory region as non-cacheable. At block 508, a transition is made from a state in which only one requestor is allowed to a state in which two or more requestors are allowed, which process includes deactivating the first memory consistency model at the requestor node and activating the second memory consistency model, as described above with respect to Figure 4. At block 508, if two or more requestors are already allowed, the second memory consistency model is already active and does not need to be changed.

[0039] At block 512, the update notification received at block 506 results in a state in which only one requestor is permitted for the FAM region, and thus the process activates a first memory consistency model. In some embodiments, the first memory consistency model includes mapping the FAM region as a write-back memory for a single requestor. Mapping the FAM region as a write-back memory for the local compute node is preferably accomplished by the ACC 297 sending appropriate messaging to an operating system running on the CPU core complex 210, which then marks one or more memory pages corresponding to the FAM region as write-back in the requestor's page table. Mapping the FAM region as a write-back memory ensures that a normal local consistency scheme (such as an x86 total store order (TSO) consistency scheme) is applied by the compute node for the FAM region. If only one requestor is permitted for the FAM region, no changes need to be made to the application's functionality to use FAM instead of local memory. The process of blocks 506, 508, 510, and 512 is repeated each time the fabric manager sends an update notification for the relevant FAM memory region.

[0040] FIG. 6 is a flow diagram 600 of a process for implementing a memory consistency model, according to some embodiments. In general, applications not originally written for use with a FAM system require modification, if they have any dependencies between compute nodes, in order for the FAM system to properly account for dependencies between compute nodes. Such modifications can be made when compiling the code, or can be applied after the fact by adding instructions to selected memory commands in the application. The process of FIG. 6 is performed on applications to which compile-time instructions have been added to adapt the application for use with the FAM system. Such applications include applications written and compiled with tools for handling consistency in the FAM system, and applications modified with suitable tools adapted for use with the FAM system.

[0041] At block 602, the process begins activating a fence for an application with compile-time instructions included in the application. Block 602 occurs when ACC 297 changes the consistency model of a compute node to activate a second, more relaxed consistency model. Typically, data fabric protocols such as CXL 3.0 and Gen-Z use a relaxed sequential consistency model that is implemented via fences. When the relaxed consistency model is active, ACC 297 attempts to insert data center fabric fences, such as CXL / Gen-Z fences, at critical locations transparently to the application code. This transparency means that the activities performed by ACC 297 do not require any coordination by the applications involved.

[0042] At block 604, the process recognizes compile-time fabric interface instructions for the selected memory access instructions. Typically, the selected memory access instructions are pieces of code such as flags, locks, semaphores, control variables, etc. that implement parallel synchronization between computational threads, and therefore memory accesses need to be ordered. The location of the selected memory access instructions is identified by hints in the application that the software developer can provide to the adaptive controller via several mechanisms.

[0043] Compiler hints such as the C++11 atomic_load / atomic_store consistency constructs or primitives, or runtime hints such as OpenMP flush constructs, can be integrated with development tools such as the “CodeAnalyst” tool (by Advanced Micro Devices, Santa Clara, California) to make consistency hints easy to share with developers. The compiler inserts requestor-side fabric interface markers such as a special “FABRIC_ACQUIRE_FENCE” instruction before the marked control variable and a special “FABRIC_RELEASE_FENCE” instruction after the control variable. These special instructions or markers are converted to no-operation (NOP) by the CPU μCode 216 on non-FAM systems or on FAM systems where only one requestor node has access to the associated FAM region, but if more than one application is allowed to access the associated FAM region, these special instructions or markers are recognized by μCode 216 in block 604.

[0044] In response to recognizing such an instruction, in block 606, the CPU's μCode 216 notifies the ACC 297 that a fabric fence is required for the instruction, as indicated by the "selected instruction" arrow in FIG. 4. In some embodiments, such notification is performed using a message over the local data fabric from the CPU to the ACC 297. The notification includes identification information for the ACC 297, such as the memory addresses of the involved variables, to identify the instruction when it is received over the local data fabric. In other embodiments, the notification may be performed by adding a predefined flag or marker to the memory access instruction to be modified when the instruction is sent by the CPU over the local data fabric to the fabric bridge 296. In some embodiments, such a marker includes the information necessary to determine whether a fence acquire instruction or a fence release instruction is required.

[0045] In response to receiving each notification, the ACC 297 issues acquire / release data center fabric fences in block 608. These fences are commands that are inserted into the command stream going to the data fabric and media controller for the associated FAM domain, as indicated by the "fence" arrow in Figure 4. The media controller then executes the command to provide the fence for the associated variable.

[0046] Although the illustrated process occurs after the application has already been modified to include the FABRIC_ACQUIRE_FENCE and FABRIC_RELEASE_FENCE instructions, in some embodiments the process also includes inserting such hints or markers into the application so that μCode 216 recognizes the selected memory instructions.

[0047] 7 is a flow diagram 700 of another process for implementing a memory consistency model, according to some additional embodiments. The illustrated process is performed for applications that are not modified with compile-time instructions to implement fabric fences. This process has the advantage that applications for which the developer has not provided a version configured for use with the FAM system may still be used with the FAM system without causing the consistency problems that typically arise with such use. Another advantage is that for deployments where a separate or more expensive software license is required to obtain an application version for use with the FAM system, this process may enable the use of non-FAM versions with the FAM system.

[0048] At block 702, the process begins activating a FAM fence for such application. Recognizing a selected memory access instruction that requires a fence command differs in this process from the process of FIG. 6 because the application does not include a specific instruction for a fabric fence. At block 704, the CPU's μCode 216 recognizes a memory access instruction in the application for which a fence is required based on recognizing an instruction in a predefined list of instruction types and instruction prefixes associated with synchronizing data between threads. In some embodiments, a predefined list is provided to μCode 216 to cover all types of applications for which μCode 216 can recognize the selected memory access instruction. The list includes instructions and instruction prefixes associated with dependencies between applications. For example, the list may include the x86 LOCK instruction prefix, the X86 xacquire / xrelease, and the PowerPC SYNC instruction.

[0049] At block 706 , whenever any instruction from this predefined list is called, the CPU's μCode 216 notifies the ACC 297 which then issues a fabric fence as shown in block 708 .

[0050] The FAM memory interface 295, or any portion thereof, such as the ACC 297 or the fabric bridge 296, may be described or represented by a computer-accessible data structure in the form of a database or other data structure that may be read by a program and used directly or indirectly to manufacture an integrated circuit. For example, the data structure may be a behavioral or register-transfer level (RTL) description of the hardware functionality in a high level design language (HDL) such as Verilog or VHDL. The description may be read by a synthesis tool that can synthesize the description to generate a netlist that includes a list of gates from a synthesis library. The netlist includes a set of gates that also represent the functionality of the hardware that comprises the integrated circuit. The netlist may then be placed and routed to generate a data set that describes the geometric shapes to be applied to a mask. The mask may then be used in various semiconductor manufacturing processes to manufacture the integrated circuit. Alternatively, the database on the computer-accessible storage medium may be a netlist (with or without a synthesis library) or a data set, or Graphic Data System (GDS) II data, as desired.

[0051] Although specific embodiments have been described, various modifications to these embodiments will be apparent to those skilled in the art. For example, the internal architecture of the fabric bridge 296 and ACC 297 may differ in different embodiments. The type of FAM and FAM protocol utilized may also differ. Furthermore, the specific fabric architecture may vary from an architecture that uses fabric protocols and transports based on PCIe or CXL to provide distribution within a data node or rack of data nodes, to an architecture that may utilize fiber or copper networks to connect between devices and racks in a data center using protocols such as Gen-Z. Accordingly, the appended claims are intended to cover all modifications of the disclosed embodiments that fall within the scope of the disclosed embodiments.

Claims

1. 1. A method for use in a fabric-attached memory system including a fabric-attached memory and a plurality of requesters coupled to the fabric-attached memory via a fabric, the method comprising: requesting notification from a fabric manager regarding a change in requestors authorized to access the fabric attached memory region; activating a fence for a selected memory access instruction in response to a notification from the fabric manager indicating that two or more requestors are permitted to access the fabric-attached memory region. method.

2. and in response to a notification from the fabric manager indicating that a single requestor is permitted to access the fabric-attached memory region, mapping the fabric-attached memory region as a write-back memory region for the single requestor.

2. The method of claim 1.

3. activating the fence includes, in response to recognizing the selected memory access instruction as indicative of a dependency between requestors, causing a requestor-side adaptive consistency controller to add a fence command for the selected memory access instruction.

2. The method of claim 1.

4. recognizing the selected memory access instruction is performed by processor instruction microcode executing at each requestor granted access to the fabric attached memory region. The method of claim 3.

5. and inserting a requestor fabric interface marker into the application during compilation in response to recognizing the consistency primitives and the consistency constructs. The method of claim 3.

6. recognizing the selected memory access instruction is based on recognizing an instruction within a predefined list of instruction types and instruction prefixes associated with data synchronization between threads. The method of claim 3.

7. and, at each requestor granted access to the fabric-attached memory region, maintaining a table of the requestors granted access to the fabric-attached memory region and updating the table in response to each of the notifications.

2. The method of claim 1.

8. the table includes at least a fabric-attached memory region identifier and a number of compute requesters that are permitted to access the fabric-attached memory region; The method of claim 7.

9. 1. A data processor comprising: a processing core for executing applications; a fabric-attached memory interface coupled to the processing core and adapted to couple to a data fabric and execute memory access instructions from the processing core to a fabric-attached memory; a requester side adaptive coherency controller coupled to the processing core and to the fabric attached memory interface; The requester-side adaptive consistency controller, requesting notification from a fabric manager of the fabric-attached memory regarding changes in requestors permitted to access regions of the fabric-attached memory that the data processor is permitted to access; activating a fence for a selected memory access instruction in a local application in response to a notification from the fabric manager indicating that two or more requestors are permitted to access the fabric attached memory region; [0023] Data processor.

10. the requestor-side adaptive consistency controller is operable, in response to a notification from the fabric manager indicating that a single requestor is permitted to access the fabric-attached memory region, to cause the fabric-attached memory region to be mapped as a write-back memory region.

10. The data processor of claim 9.

11. a processor instruction microcode stored in a tangible non-transitory memory accessible by said processing core and executable by said processing core to recognize said selected memory access instruction; 10. The data processor of claim 9.

12. and in response to recognizing the selected memory access instruction as indicating a dependency between requestors, the processor instruction microcode performs a process of notifying the requestor side adaptive consistency controller of the recognized selected memory access instruction to cause a requestor side fence command to be inserted into a memory command to the fabric-attached memory interface.

12. The data processor of claim 11.

13. recognizing the selected memory access instructions includes recognizing fabric interface markers inserted into the application during compilation.

12. The data processor of claim 11.

14. the requestor-side adaptive consistency controller maintains a table of the requestors that are granted access to the fabric-attached memory region and updates the table in response to the notification.

10. The data processor of claim 9.

15. the table includes at least a fabric-attached memory region identifier and a number of compute requesters that are permitted to access the fabric-attached memory region; 15. A data processor according to claim 14.

16. 1. A fabric-attached memory system, comprising: a fabric attached memory; a data fabric coupled to the fabric attached memory; a fabric manager coupled to the data fabric and operable to grant and de-authorize requesters from accessing memory regions of the fabric-attached memory; a plurality of data processors coupled to the data fabric; Each of the plurality of data processors comprises: a processing core for executing applications; a fabric attached memory interface; a requester side adaptive coherency controller coupled to the processing core and to the fabric attached memory interface; The requester-side adaptive consistency controller: requesting notification from the fabric manager regarding changes in requestors authorized to access fabric-attached memory regions that the data processor is authorized to access; activating a fence for a selected memory access instruction in a local application in response to a notification from the fabric manager indicating that two or more requestors are permitted to access the fabric attached memory region; [0023] Fabric-attached memory system.

17. the requestor-side adaptive consistency controller is operable, in response to a notification from the fabric manager indicating that a single requestor is permitted to access the fabric-attached memory region, to cause the fabric-attached memory region to be mapped as a write-back memory region.

20. The fabric-attached memory system of claim 16.

18. each data processor including processor instruction microcode stored in a tangible non-transitory memory accessible by a processing core of said each data processor and executable by said processing core to recognize said selected memory access instructions; 20. The fabric-attached memory system of claim 16.

19. and in response to recognizing the selected memory access instruction as indicating a dependency between requestors, the processor instruction microcode performs a process of notifying the requestor side adaptive consistency controller of the recognized selected memory access instruction to cause a requestor side fence command to be inserted into a memory command to the fabric-attached memory interface.

20. The fabric-attached memory system of claim 18.

20. recognizing the selected memory access instructions includes recognizing fabric interface markers inserted into the application during compilation.

20. The fabric-attached memory system of claim 18.

21. the requestor-side adaptive consistency controller maintains a table of the requestors that are granted access to the fabric-attached memory region and updates the table in response to the notification.

20. The fabric-attached memory system of claim 16.

22. the table includes at least a fabric-attached memory region identifier and a number of compute requesters that are permitted to access the fabric-attached memory region; 22. The fabric-attached memory system of claim 21.

Images