System Coherency Protocol

Abstract

In an embodiment herein, a coherency protocol for a distributed computing topology that tolerates large scale stalls on various interfaces is described. In one embodiment, the computing topology includes multiple boards, each including multiple processors. When a particular core on a processor wants to access data that is not currently stored in its cache, the core may first initiate a request to search for a cache line in the caches of other cores on the same processor. If the cache line is not found, the cache coherency protocol allows the processor to then broadcast a request to other processors on the same board. If the processors on the same board do not have the data, the processor may then broadcast a request to other boards in the system. The processors in those boards may then search their caches to identify the data.

Description

[Technical field]

[0001] The present invention relates to coherency protocols in shared memory multiprocessing (SMP) computing systems. [Background technology]

[0002] New computing systems rely on highly distributed system topologies with high-speed interfaces between processors (e.g., between processor chips). High interface speeds lead to reliability issues and the need for new bus recovery mechanisms. Interface recovery mechanisms for high-speed interfaces can lead to large stalls and unpredictable message propagation speeds. This creates various challenges in maintaining data integrity in cache-coherent systems. Therefore, distributed system topologies with processors distributed across different boards or platforms require improved cache coherency protocols. Summary of the Invention

[0003] According to one embodiment of the present invention, a method includes providing a cache coherent computing system including a plurality of boards, each board including a plurality of processors, requesting cache data from a first core of a first processor on a first board of the plurality of boards, and upon determining that another core on the first processor does not have the requested cache data, broadcasting the request to other processors on the first board to determine whether one of the other processors is an Intervening Master (IM) for the requested cache data, and upon determining that none of the other processors on the first board is an IM, broadcasting the request to other boards of the plurality of boards to determine whether one of the processors on the other boards is an IM. One advantage is that this cache coherency method is tolerant of delays that change as cache operations move to a higher scope (e.g., board-to-board communication).

[0004] The method may also include receiving, at the first processor, partial responses (PRESP) from other processors on the first board, determining based on the PRESP which of the other processors are still involved in the request and other operations initiated by the other processors on the same cached data, where at least one of the other processors indicates in the PRESP that it is not involved in the request, and sending a combined response (CRESP) to only the other processors that are still involved in the request. One advantage is that this limits the number of processors that continue coherency actions, thereby allowing the uninvolved processors to perform other functions.

[0005] The method also includes including a fork processor in the first board assigned to communicate with each of the multiple boards where other processors are still involved in the request. One advantage is that at this point, the first processor does not yet know whether an IM for the requested data is on the same board, so the fork processor may still be used to send to the other boards.

[0006] The method also includes receiving the request at each branch processor in each of the other boards, broadcasting the request from each branch processor to other processors in the same board, receiving at each branch processor a PRESP from the other processors in the same board reporting other operations in-flight for the same cache data, forwarding the PRESP from the other processors in the other boards to the first processor on the first board, sending a board-scope CRESP from each branch processor to only other processors in the same board that have indicated they are still interested in the request based on the PRESP, and sending a global-scope CRESP from the first board to other processors in the other boards that have indicated they are still interested in the request based on the PRESP. One advantage of the board-scope CRESP and global-scope CRESP is that it includes only processors in the board scope and global scope that are still interested in the cache operation.

[0007] The method also includes a first one of the other processors that indicates it is still involved in the request is an IM for the requested cached data or has an in-flight operation on the same cached data and a second one of the other processors that indicates it is still involved in the request is not an IM for the requested cached data. One advantage is that any processor requesting the same cached data as the first processor will receive the communication and know that the IM has changed.

[0008] According to one embodiment of the present invention, a cache coherent computing system includes a plurality of boards, each board including a plurality of processors, and a first core in a first processor on a first board of the plurality of boards is configured to request cache data that is not stored in a cache assigned to the first core. Upon determining that another core on the first processor does not have the requested cache data, the first processor is configured to broadcast a request to other processors on the first board to determine whether one of the other processors is an Intervening Master (IM) for the requested cache data. Upon determining that none of the other processors on the first board are an IM, the first processor is configured to broadcast a request to other boards of the plurality of boards to determine whether one of the processors of the other boards is an IM. One advantage is that this cache coherency method is tolerant of delays that change as cache operations move to a higher scope (e.g., board-to-board communication).

[0009] According to one embodiment of the present invention, a computer program product for maintaining a cache coherent computing system comprising a plurality of boards, each board including a plurality of processors, the computer program product including a computer readable storage medium having computer readable program code embodied therein, the computer readable program code being executable by one or more computer processors to perform operations. The operations include requesting cache data from a first core of a first processor on a first board of the plurality of boards, and, upon determining that another core on the first processor does not have the requested cache data, broadcasting the request to other processors on the first board to determine whether one of the other processors is an Intervening Master (IM) for the requested cache data, and, upon determining that none of the other processors on the first board is an IM, broadcasting the request to other boards of the plurality of boards to determine whether one of the processors on the other boards is an IM. One advantage is that this cache coherency method is tolerant of delays that change as cache operations move to a higher scope (e.g., board-to-board communication).

[0010] According to one embodiment of the present invention, a cache coherent computing system includes a plurality of boards, each board including a plurality of processors, and is configured such that a first core in a first processor on a first board of the plurality of boards requests cache data that is not stored in a cache assigned to the first core. Upon determining that another core on the first processor does not have the requested cache data, the first processor is configured to broadcast a request to other processors on the first board to determine whether one of the other processors is an Intervening Master (IM) for the requested cache data, receive a partial response (PRESP) from the other processors of the first board, determine which of the other processors are still involved in the request based on the PRESP, where a first one of the other processors indicates in the PRESP that it is an IM and a second one of the other processors indicates in the PRESP that it is not an IM but is still involved in the request, and send a combined response (CRESP) to only the other processors that indicated in the PRESP that they are still involved in the request. One advantage is that this cache coherency method is tolerant of delays that change as cache operations move from intra-processor communication to inter-processor communication between processors on the same board.

[0011] Embodiments of the present invention will now be described, by way of example only, with reference to the accompanying drawings, in which: [Brief description of the drawings]

[0012] [Figure 1] FIG. 1 illustrates a cache coherent computing system having a distributed topology, according to one embodiment. [Diagram 2] FIG. 2 illustrates a processor chip according to one embodiment. [Diagram 3] FIG. 2 illustrates a communication system between processors on a board according to one embodiment. [Figure 4] FIG. 2 illustrates communication between boards in a cache coherent computing system according to one embodiment. [Diagram 5] 1 is a flowchart for identifying cached data in a distributed topology according to one embodiment. [Figure 6A] FIG. 1 illustrates identifying cache data within a board including multiple processors according to one embodiment. [Figure 6B] FIG. 1 illustrates identifying cache data within a board including multiple processors according to one embodiment. [Figure 6C] FIG. 1 illustrates identifying cache data within a board including multiple processors according to one embodiment. [Figure 6D] FIG. 1 illustrates identifying cache data within a board including multiple processors according to one embodiment. [Figure 6E] FIG. 1 illustrates identifying cache data within a board including multiple processors according to one embodiment. [Figure 6F] FIG. 1 illustrates identifying cache data within a board including multiple processors according to one embodiment. [Figure 7A] FIG. 1 illustrates identifying cache data in multiple boards according to one embodiment. [Figure 7B] FIG. 1 illustrates identifying cache data in multiple boards according to one embodiment. [Figure 7C] FIG. 1 illustrates identifying cache data in multiple boards according to one embodiment. [Figure 7D] FIG. 1 illustrates identifying cache data in multiple boards according to one embodiment. [Figure 7E] FIG. 1 illustrates identifying cache data in multiple boards according to one embodiment. [Figure 7F] FIG. 1 illustrates identifying cache data in multiple boards according to one embodiment. [Figure 7G] FIG. 1 illustrates identifying cache data in multiple boards according to one embodiment. [Figure 7H] FIG. 1 illustrates identifying cache data in multiple boards according to one embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] In an embodiment herein, a coherency protocol for a distributed computing topology that tolerates large scale stalls on various interfaces (e.g., interfaces between different motherboards that contain processor chips) has been described. In one embodiment, the computing topology includes multiple boards (e.g., multiple motherboards, also referred to as "drawers"), each having multiple processors (e.g., multiple processor chips). When a particular core on a processor wants to access data that is not currently stored in its cache, the core may first initiate a request to search for a cache line in the caches of other cores on the same processor. If the cache line is not found, the processor then broadcasts a request to other processors on the same board. For example, the processors within a board may be connected via a fully connected fabric. If a processor on the same board does not have the data, the processor may then broadcast a request to other boards in the system. The processors in those boards may then search their caches to identify the data.

[0014] One challenge in performing coherency operations in a multi-scope system is that the latency involved in performing cache operations (e.g., snoops, cache responses, etc.) may vary widely at each level or scope (e.g., searching a cache in the same processor versus searching a cache in a processor on the same board versus searching a cache in a processor on a different board). The multi-scope coherency protocol described herein may ensure fast data sharing at lower scopes (e.g., searching a cache in the same processor) and tolerate increased latency as the request progresses to higher scopes (e.g., searching a cache in a processor on the same board or searching a cache in a processor on a different board). The multi-scope coherency protocol may be tolerant of varying delays as the cache operation moves to higher scopes. Advantageously, the multi-scope coherency protocol may be used in different system topologies and still maintain data integrity.

[0015] 1 illustrates a cache coherent computing system 100 having a distributed topology, according to one embodiment. In this example, the topology of the system 100 is distributed because the processors 110 (e.g., central processing units (CPUs)) are distributed on different boards 105 (e.g., different motherboards or drawers) within the computing system. The communication techniques used to communicate between the processors 110 on the same board 105 may be different than the communication techniques used to communicate between the processors 110 on different boards 105. For example, communicating messages or performing cache operations between the processors 110 on different boards 105 may generally incur more latency than performing cache operations between the processors 110 on the same board 105.

[0016] Computing system 100 may be any computing system, such as a server, a blade server, a rack server, a tower server, a desktop computer, a laptop, etc. In one embodiment, computing system 100 includes a chassis that includes the hardware elements shown in Figure 1. In one embodiment, the chassis may be mountable, such that computing system 100 can be mounted in a rack or tower.

[0017] In this example, computing system 100 includes two boards, boards 105A and 105B, each including multiple processors 110. However, computing system 100 may have any number of boards (e.g., 3, 4, 8, etc.), each including any number of processors (e.g., 2, 4, 6, 8, 16, etc.). Each processor 110 includes one or more (e.g., 1, 4, 8, etc.) cores 115 and caches 120. Some of the caches may be integrated into the cores (e.g., level 1 (L1)) caches, while other caches 120 are external to cores 115 (e.g., level 2 (L2) and level 3 (L3) caches). The embodiments herein are not limited to any particular number of cache levels or cache hierarchy. Although not shown, computing system 100 may also include a main memory (e.g., random access memory), which may be separate from board 105 or may be located on board. The computing system 100 may also include long-term storage, such as a hard drive.

[0018] In a shared memory multiprocessor system such as computing system 100 with an isolated cache 120 for each processor 110, it is possible to have many copies of shared data, e.g., one copy in main memory (not shown) and one copy in the local cache of each processor that requested it. However, maintaining data coherency in a highly distributed topology using high speed interfaces with long term recovery mechanisms can be difficult and can benefit from the techniques described herein. For example, to ensure data integrity of a cache line in transit, the old owner of the cache line should continue to protect the cache line until the new owner begins protecting the data, and any other in-flight operations on the same cache line (e.g., when two cores 115 are requesting the same cache line simultaneously) will be denied.

[0019] Embodiments herein establish several coherency protocol rules to address computing systems with distributed topologies. In this example, the topology of computing system 100 has three distinct scopes: (1) processor scope, (2) board scope, and (3) inter-board scope (or global scope). The multi-scope coherency protocol described herein addresses the heterogeneous nature of the system topology of system 100 with limited resources and allows fast and fair line sharing when communicating between processors on the same board 105 and different boards 105. Embodiments herein introduce a join response that does not depend on another operation to complete (CRESP). That is, no other operation can stop the creation of a CRESP. Additionally, the multi-scope coherency protocol can be time-delay tolerant (e.g., time-independent) by using a coherency protocol (also called a fabric (FBC) protocol) that handles any delays resulting from cache operations transmitted between processors 110 on the same board 105 and different boards 105. The coherency protocol is responsible for handling off-chip broadcasts and all associated messages, as well as coherency aspects that maintain data coherency. Furthermore, an Intervening Master (IM) may complete its tasks without relying on other operations from the same or lower scopes. That is, nothing prevents an IM from completing the handoff of a requested cache line to a new owner. As used herein, an IM is the highest point of coherency for a given scope and contains the most up-to-date cached copy of the cache line. The IM is typically a cache on the processor 110 (or a core 115 within the processor 110), but may also be a memory (e.g., cache 120) if the current core does not own the cache line.Additionally, in one embodiment, the multi-scope coherency protocol has request observability, where an actor outside the current scope cannot block a request from broadcasting a request within its scope.

[0020] 2 illustrates a processor 110 according to one embodiment. In one embodiment, the processor 110 is an integrated circuit (e.g., a single chip). The processor 110 of FIG. 2 is an example of an implementation of the processor 110 shown in FIG.

[0021] Processor 110 includes eight cores (Cores 0-7) and eight caches (Caches 0-7). In one embodiment, the caches are owned by the respective cores. For example, cache 0 is owned by core 0, cache 1 is owned by core 1, etc. As shown, cores have direct connections to their owning caches but may not have direct connections to their non-owning caches. The caches may be L1, L2, or L3 caches.

[0022] Processor 110 also includes an X-FBC 205 and an M-FBC 210. In FIG. 2, the "X" designation indicates circuitry tasked with communicating with processors on the same board but in different sockets, and the "M" designation indicates circuitry tasked with communicating with processors on the board in the same socket as processor 110. These sockets and connections are described in FIG. 3. However, embodiments herein are also applicable to computing systems where the processors are not located (or packaged) in different sockets on a board.

[0023] The FBCs 205, 210 are fabric logic that manages coherency between multiple processors. That is, the processor 110 uses the FBCs 205, 210 when sending and receiving cache operations to other processors. For example, if the processor 110 sends a request for a cache line to a processor in the same socket, it communicates with the processor using the M-FBC 210. However, if the processor 110 sends a request for a cache line to a processor in a different socket, it communicates with the processor using the X-FBC 205.

[0024] Processor 110 also includes X-bus input / output (IO) 215 and M-bus IO 220. Similar to FBCs 205, 210, X-bus IO 215 is an IO interface for communicating with processors on different sockets, and M-bus IO 220 is an IO interface for communicating with processors on the same socket. Again, other topologies may have different configurations that do not package the processor in a socket. Processor 110 also includes A-bus IO 240, which the FBC unit uses to connect to other boards.

[0025] In this example, processor 110 communicates between caches using rings 225 and 230. Two rings are used in FIG. 2 to illustrate that different types of data may be transmitted between caches. For example, outer ring 225 may represent the path used by coherency messages (e.g., snoop and response messages), and inner ring 230 may represent the data bus between caches.

[0026] Assume that core 2 wants to read and process data in a particular cache line. Core 2 first searches its internal cache (if it has one) and if it does not find the cache line (i.e., a cache miss), it queries its own cache, i.e., cache 2. If cache 2 does not own the cache line, core 2 sends a request to other cores to search their caches for the cache line. This request may travel along ring 225 to different caches. In one embodiment, the request may be sent in one direction or in both directions simultaneously. The cache / core may then respond to the request indicating whether or not it owns the requested cache line.

[0027] 2 illustrates the use of a ring topology to perform cache operations in the processor 110, but this is by way of example only. The embodiments herein are not limited to any particular technique for performing cache operations in a single chip / processor, but instead may be used with any other type of technique for performing cache operations in a single processor, such as a mesh network, a crossbar, etc. As such, the rings 225, 230 are but one suitable technique for propagating cache operations within a chip.

[0028] 3 illustrates a communication system between processors 110 on a board 105 according to one embodiment. For example, the computing system 100 of FIG. 1 may have multiple instances of the board 105 illustrated in FIG.

[0029] In this example, the board 105 includes four sockets 305A-D, each of which includes two processors 110. The sockets 305 may then be coupled to a common substrate (e.g., a printed circuit board). For example, the processors 110 in the same socket 305 may be packaged together and disposed on a common substrate (e.g., an interposer). However, embodiments herein are not limited to boards 105 having sockets and may include embodiments in which the processors 110 are mounted directly to the board 105.

[0030] Board 105 includes a full connection fabric 310 for communicatively coupling processors 110 that are located on different sockets 305. In this example, the processor in each socket has a direct connection in full connection fabric 310 to each processor in a different socket. That is, processor CP-0 has a direct connection to processors CP-2 / CP-3 in socket 305B, processors CP-4 / CP-5 in socket 305C, and processors CP-6 / CP-7 in socket 305D. Referring again to FIG. 2, the six connections to these six processors can be made using six X-bus IOs 215 on the top of processor 110.

[0031] Board 105 also includes inter-socket connections 315 for connecting processors in the same socket 305. That is, processor CP-0 communicates with processor CP-1 using inter-socket connection 315, processor CP-2 communicates with processor CP-3 using inter-socket connection 315, etc. Referring again to Figure 2, inter-socket connections to processors in the same socket may be made using one of the lower M-bus IOs 220 of processor 110.

[0032] The fully-connected fabric 310 and inter-socket connections 315 ensure that each processor 110 has a direct communication link to every other processor 110 on the same board 105. That is, resources are guaranteed that one processor can use to communicate with every other processor in the same board. In this example, "direct communication link" means that a processor 110 can communicate with another processor on the same board 105 without using another processor as an intermediary (e.g., as would be the case with a spine-leaf topology). Additionally, the fully-connected fabric 310 may be time-independent in that cache coherence may be maintained in the presence of additional latency or transmission errors.

[0033] FIG. 4 illustrates communication between boards in a cache coherent computing system, according to one embodiment. Each board 105 (boards 0-3) includes four sockets and eight processors, as shown in FIG. 3. The straight lines indicate the inter-board connections between the boards. In this example, each board has sockets (i.e., two processors) assigned to communicate with different boards, called fork processors. Using board 0 as an example, processors CP-2 / CP-3 are connected to a pair of processors on board 2, processors CP-4 / CP-5 are connected to a pair of processors on board 1, and processors CP-6 / CP-7 are connected to a pair of processors on board 3. When a processor wants to send a cache message to a different board, it may reach the board using a fork processor. For example, if processor CP-0 on board 0 wants to send a response to a processor on board 3, processor CP-0 will first send the response to one of processors CP-6 / CP-7 on board 0, which will then forward the response to processor CP-6 / CP-7 on board 3.

[0034] In one embodiment, a processor assigned to communicate with other boards may have a ticket manager that manages the cache operations flowing between the boards. The ticket manager may use a queue to determine which cache operation to send next.

[0035] Unlike FIG. 3, where a processor has a direct connection to every other processor, in FIG. 4, a processor on one board may not have a direct communication link to a processor on a different board. That is, a processor may have to rely on one or two processors (e.g., a fork processor) to communicate a cache request or response to a processor on a different board. Thus, FIG. 4 can be referred to as a pass-through topology, in which a processor is used as an intermediary between other processors on different boards. For example, CP-0 on board 0 relies on processors CP-2 / CP-3 to reach board 2, relies on processors CP-4 / CP-5 to reach board 1, and relies on processors CP-6 / CP-7 to reach board 3.

[0036] FIG. 5 is a flow chart of a method 500 for identifying cached data in a distributed topology, according to one embodiment. At block 505, a requestor (e.g., a core on a processor) determines whether a cache on the processor has the requested data (e.g., a particular cache line). For example, the requestor may search an internal L1 cache (if any) and an assigned L2 cache (e.g., caches 0-7 of FIG. 2) to determine whether these caches have the most recent copy of the requested data. For example, the requesting core may use the processor's communication network (e.g., the ring topology or mesh network of FIG. 2) to determine whether caches assigned to other cores in the processor contain the requested data. If so, method 500 proceeds to block 510, where the requestor retrieves the most recent version from a cache on the processor.

[0037] However, if none of the caches in the processor have the most recent version of the requested data, method 500 proceeds to block 515, where the processor broadcasts the request to processors on the same board. For example, the processor may broadcast the request to processors in the same socket and different sockets on board 105 using full connectivity fabric 310 and inter-socket connections 315 of Figure 3. However, the connections shown in Figure 3 are just one example of connecting processors located on the same board.

[0038] In block 520, a processor determines whether it is an IM for the requested data, i.e., after receiving the broadcast, each processor determines whether it is an IM for that data, i.e., whether the processor is at the highest point of coherency and contains the most up-to-date cached copy of the requested data.

[0039] Details regarding searching for processors within the same board to determine if one of these processors is the IM for the requested data are described below in FIGS. 6A-6F.

[0040] If one of the processors on the same board is the IM, the method proceeds to block 520, where the IM is changed to the requesting processor. Doing so places the onus on the requesting processor to protect the cache line, as described in more detail below. Additionally, at block 510, the requested data may be copied from a cache in the processor that was previously the IM to the requesting processor.

[0041] If none of the processors on the board are IM for the requested data, the method proceeds to block 530, where the requesting processor broadcasts the request to processors on other boards to determine whether they are IM for the data (or whether the data is not owned by any core but instead by a cache within the processor).

[0042] Details of searching the processors on different boards to determine if one of these processors is the IM for the requested data are described below in Figures 7A-7H.

[0043] Method 500 assumes that one of the processors on the different board is the IM for the requested data. At block 520, the IM is changed to the requesting processor, and at block 510, the requested data is copied from a cache in the processor that was previously the IM to the requesting processor. However, if no processor in the computer system is the IM for the requested data, the computing system may instead retrieve the data from main memory or long-term storage (e.g., a hard drive).

[0044] Figures 6A-6F illustrate identifying cache data within a board 105 including multiple processors 110, according to one embodiment. For ease of explanation, board 105 has the same configuration of sockets and processors as shown in Figures 3 and 4, except that the eight processors are labeled CP_A through CP_H. Figures 6A-6F illustrate steps that may be performed in blocks 515 and 520 to identify whether a processor on the board (e.g., at the board scope or level) is an IM for the requested data.

[0045] In this example, processor CP_E is the home processor requesting data (e.g., a cache line). That is, processor CP_E wants to become the IM of the data so that processor CP_E can ensure that it has the most up-to-date copy of the data. FIG. 6A shows processor CP_E broadcasting the request to all other processors in the same board 105. For example, processor CP_E may broadcast the request to each of processors CP_A, CP_B, CP_C, CP_D, CP_F, CP_G, and CP_H using the fully connected fabric 310 and inter-socket connections 315 of FIG. 3.

[0046] 6B shows processor CP_E receiving a partial response (PRESP) from each of the processors. The PRESPs may be received at different times. As discussed above, the board scope may be time tolerant such that the PRESPs from the processors may be received at different times based, for example, on the available bandwidth of the connection and the usage of the processors. In one embodiment, processor CP_E waits until it receives all PRESPs before proceeding.

[0047] In one embodiment, the PRESP indicates whether the processor is an IM or if the IMs are on different boards, the processor may be tasked with identifying the IM. In this case, processor CP_B is the IM for the requested data (labeled board IM1 (BIM1)). Thus, its PRESP indicates to processor CP_E that processor CP_B is the IM for the data. Additionally, processors CP_A, CP-C, and CP_G are labeled as "fork" in Figures 6A-6F, indicating that these processors are assigned to be used to communicate with other boards. That is, if the IMs are not located on board 105, processors CP_A, CP-C, and CP_G would then be tasked with broadcasting the request to other boards in the computing system, as described in Figures 7A-7H. For example, processor CP_A may be responsible for exchanging cache operation messages with board 1, processor CP_C may be responsible for exchanging cache operation messages with board 2, and processor CP_G may be responsible for exchanging cache operation messages with board 3.

[0048] 6B, the fork processors do not yet know whether the IM is located on board 105 when they send their PRESPs, so their PRESPs indicate that the fork processors will remain in a standby state in case processor CP_E must broadcast a request for data to other boards. That is, the fork processors have not indicated in their PRESPs to processor CP_E that their portion of the cache operation is complete.

[0049] In contrast, processors CP_D, CP_F, and CP_H are neither the IM for the requested data nor the fork processors used to communicate with other boards, so the PRESP sent by processors CP_D, CP_F, and CP_H indicates to processor CP_E that their part of the cache operation is complete and that they are no longer involved in the request.

[0050] 6C illustrates processor CP_E sending a combined response (CRESP) in response to receiving all PRESPs (i.e., partial responses from other processors on board 105). As illustrated, the CRESP is not broadcast to all processors on board 105, but only to those processors that have indicated they are still part of the cache operation (i.e., a processor that is the IM for the requested data, the home memory controller for the data, a processor that has an active outstanding request for the same cache line, or a fork processor used to communicate with other boards in the computing system). Thus, processors CP_D, CP_F, and CP_H do not receive the CRESP.

[0051] CRESP informs the fork processors CP_A, CP-C, and CP_G that it has found an IM for the data, so they do not need to send messages to other boards. In other words, the fork processors are no longer needed to complete the cache operation. In response, CRESP informs the IM (i.e., processor CP_B) that it still needs an IM to complete its operation.

[0052] Figure 6D shows that fork processors CP_A, CP_C, and CP_G send reset responses (RRESPs) to processor CP_E. These RRESPs inform processor CP_E that processors CP_A, CP_C, and CP_G have finished their cache operations.

[0053] Figure 6E shows processor CP_E sending an end-of-coherency response (ERESP) to processor CP_B. ERESP informs processor CP_B that it is safe to stop protecting the requested data and that the home processor CP_B is now responsible for protecting the requested data. In other words, ERESP indicates that it is safe for the old IM (i.e., processor CP_B) to reset the coherency protection. The old IM can stop rejecting new requests for the data. ERESP indicates that the new IM (i.e., processor CP_E) is now capable of handling further requests for the data. Furthermore, ERESP informs the old IM that the new IM will ensure that all potential observers are satisfied.

[0054] In Figure 6F, in response to receiving ERESP, processor CP_B sends RRESP to processor CP_E indicating that it has completed the cache operation. Additionally, ERESP indicates to processor CP_E that any other requesters of the same data (e.g., another processor on that board or a different board) should have observed the transfer and that the data is being transferred. Thus, this prevents conflicts where processors requesting the same data in parallel may not know which processor is currently the IM for that data.

[0055] Figures 7A-7H illustrate identifying cached data across multiple boards, according to one embodiment. For ease of illustration, each board 105A-D in Figures 7A-7H has the same configuration of sockets and processors as board 105 in Figures 6A-6F. Figures 7A-7H illustrate steps that may be performed in block 530 to identify whether a processor on a different board (e.g., inter-board scope or level) is an IM for the requested data.

[0056] 7A-7H assume that blocks 505-525 of method 500 have already been performed and the requesting processor has determined that (i) it does not have a current cached copy of the requested data and (ii) the IM for the data is not on the same board. Thus, the requesting processor broadcasts a request to other boards in the system to identify the IM.

[0057] In Figure 7A, similar to Figure 6A, processor CP_E is the requesting processor that is retrieving the IM of the requested data. However, unlike Figure 6A, the IM of the data is not on the same board (i.e., board 105A) but on a different board. Specifically, the IM of the requested data (labeled global IM (GIM1)) is processor CP_F on board 105B.

[0058] Processor CP_E determines that none of the processors on board 105A are IMs for the requested data, and therefore instructs fork processors CP_A, CP_C, and CP_G on board 105A to broadcast requests for the IMs to the other boards 105B-D, as shown in FIG. 7A. The request sent to board 105B is received by processor CP_G (e.g., fork processor), the request sent to board 105C is received by processor CP_C (e.g., fork processor), and the request sent to board 105D is received by processor CP_A (e.g., fork processor). Each of these fork processors on the three boards 105B-D then broadcasts a request for the IM to the processors on the same board. That is, each fork processor broadcasts a request in the same manner as shown in FIG. 6A.

[0059] Figure 7B shows that the bifurcation processor on each board 105B-D collects the partial responses PRESP from each processor on the same board. As described in Figure 6B, the other processors can communicate to the processor broadcasting the request (e.g., the bifurcation processor) whether they are the IM for the data.

[0060] 7B further shows that the fork processor forwards the PRESP to fork processors CP_A, CP_C, and CP_G on board 105A, which in turn forward the PRESP to the requesting processor CP_E. Based on the responses, the requesting processor CP_E may determine that processor CP_F on board 105B is the IM for the requested data.

[0061] Figure 7C illustrates a board-scoped CRESP. As shown, each bifurcation processor on boards 105B-D sends a CRESP based on the PRESPs it receives from other processors in the same board. These board-scoped CRESPs may contain the same information as the CRESPs described in Figure 6C above.

[0062] FIG. 7D illustrates a global scope CRESP (or inter-board scope CRESP). The global CRESP originates from the requesting processor CP_E on board 105A. The CRESP is sent only to the fork processor in board 105A, which in turn forwards the CRESP to each board. As illustrated, the fork processors on boards 105B-105D forward the CRESP only to those processors that are still part of the operation. That is, since only processor CP_F in board 105B is an IM for the data, the PRESP sent by the other processors in board 105B indicates that those processors are not IM and therefore do not need to be involved in the cache operation any more. Thus, only processor CP_F and the fork processor in board 105B receive the CRESP. On board 105C, the CRESP is forwarded from fork processor CP_C to only processor CP_F, which functions as LFAR-X. LFAR-X indicates that CP_F has an ongoing request for the same line (e.g., LFAR-X indicates "compare" or "conflict detect"). On board 105D, CRESP is forwarded from branch processor CP_A to only processor CP_F, which serves as the memory home for the requested data. Processor CP_F remained part of the cache operation because the memory home for the requested data could be the IM for the data if another processor has not yet assigned an IM. In this case, since processor CP_F on board 105B was the IM, CRESP informs processor CP_F on board 105D that the IM has been found and that it should no longer be involved in the cache operation. One advantage of board-scope and global-scope CRESP is that it only includes the board-scope and global-scope processors that are still involved in the cache operation.

[0063] Because CP_F on board 105C has a request for the same line in flight, the controller handling the request originating from board 105A waits until board 105C receives a denial CRESP before sending an RRESP back to the CP_E chip on board 105A. The comparison is reported on a PRESP, and the home processor waits for an RRESP from all chips that reported the comparison on a PRESP before sending an ERESP to the GIM1 processor. Figure 7E shows the home processor CP_E on board 105 receiving an RRESP from a non-IM processor (e.g., CP_F on board 105C) indicating the completion of the comparison operation. In Figures 7A-7E, the IM has moved from CP_F on board 105B to CP_E on board 105A. LFAR_X on CP_F on board 105C also wants to access this cache line and maintain coherency to ensure that the system observes the IM movement. This is done by preventing LFAR_X on CP_E of board 105A from completing until it sees that LFAR_X on CP_F of board 105C has been rejected by either CP_F of board 105B or CP_E of board 105E. At time PRESP in FIG. 7B, CP_F of board 105C reports a comparison and the state machine is loaded depending on LFAR_X. Once LFAR_X on CP_F of board 105C is rejected, the dependency is resolved and the state machine sends an RRESP. Once LFAR_X on CF_E of board 105A has received all RRESPs from non-IM processors, it knows that all dependencies encountered on the snoop have been resolved and all other requests have observed IM transfers.

[0064] 7F shows the requested data's IM (i.e., processor CP_F on board 105B) and the data's memory home (i.e., processor CP_F on board 105D) sending an RRESP to the requesting processor. In one embodiment, before sending an ERESP to the current IM, the requesting processor waits until it receives an RRESP from all non-IM processors that indicated in their PRESP that they have outstanding requests for the same line.

[0065] Figure 7G shows the requesting processor CP_E sending a coherency termination ERESP to IM, which informs processor CP_F on board 105B that it is safe to stop protecting the requested data, and that the home processor CP_E is now responsible for protecting the requested data.

[0066] In Figure 7H, in response to receiving ERESP, processor CP_F on board 105B sends RRESP to processor CP_E on board 105A indicating that it has completed the cache operation. In addition, ERESP indicates to processor CP_E that any other requesters of the same data (e.g., another processor on that board or a different board) should have observed the transfer and that the data is being transferred. This prevents conflicts where processors requesting the same data in parallel may not know which processor is currently the IM for that data.

[0067] Further, Figure 7H shows that the memory home of the data (i.e., processor CP_F on board 105D) and the global IM of the data (i.e., processor CP_F on board 105B) send an RRESP to the home processor CP_E on board 105. Now, all processors involved in the operation are sending an RRESP to the home processor. This is used to determine when a remote resource is available for use by another operation.

[0068] For performance reasons, the system may speculatively begin accessing memory before it knows if the memory data is needed. At snoop time in Figure 7A, the system loads the state machine and begins the memory access. The global CRESP indicates to the memory home (i.e., processor CP_F on board 105D) that the memory data is not needed, but that the state machine on the memory chip should still send a message indicating when it is done.

[0069] By waiting for any in-flight operations to get CRESP on all processor chips that reported the comparison before returning an RRESP to the home chip, and having the home chip delay ERESP to the global IM processor until the home chip has received all RRESPs from all chips that reported the comparison, it is guaranteed that all operations active on the same cache line observe the movement of the global IM to the new processor.

[0070] In one embodiment, there are no other operations that may block the IM from requesting the original, from fetches that hit the IM, and from castouts / evictions of the IM. Whenever one or more requests are attempting to acquire a cache line, at least one is guaranteed to win. In one embodiment, the protocol has a built-in mechanism to ensure acquisition of additional resources required to complete the operation. If resources are held by a non-IM requester or a resource queue is forming, a mechanism exists to cancel the non-IM requester so that it can open resources for the IM requester. Additionally, CRESP may include information about whether the requester has IM permission for the current scope or a higher scope.

[0071] In one embodiment, there are no other operations that may block the request from proceeding to the higher broadcast scope. That is, the protocol allows all new requesters to be broadcast or observed by other operations in flight. Before sending an ERESP in a scope, a request may wait until it receives an RRESP from all targets in the scope that have outstanding requests for the same cache line. Additionally, the requesting processor may honor any address comparisons in the requesting FBC itself. This is accomplished by setting the controller's scope-based pending action and system-based pending action, as well as setting need-to-wait actions for each other when appropriate to ensure system coherency. Additionally, the requesting processor may have to receive a CRESP for the higher scope broadcast, if necessary.

[0072] The description of various embodiments of the present invention is presented for illustrative purposes, but is not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terms used in this specification are selected to best explain the principles of the embodiments, practical applications, or technical improvements to the technology found in the market, or to enable other skilled in the art to understand the embodiments disclosed herein.

[0073] As mentioned above, reference is made to the embodiments presented in this disclosure. However, the scope of the disclosure is not limited to the specific described embodiments. Instead, any combination of features and elements, whether related to different embodiments or not, is contemplated to implement and practice the contemplated embodiments. Furthermore, while the embodiments disclosed herein may achieve advantages over other possible solutions or the prior art, whether or not a particular advantage is achieved by a given embodiment does not limit the scope of the disclosure. Thus, the aspects, features, embodiments, and advantages discussed herein are merely exemplary and are not considered elements or limitations of the appended claims unless expressly recited in the claims. Similarly, references to "the present invention" are not to be construed as a generalization of any inventive subject matter disclosed herein and are not to be considered elements or limitations of the appended claims unless expressly recited in the claims.

[0074] Aspects of the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment (including firmware, resident software, microcode, etc.), or an embodiment combining software and hardware aspects, all of which may be generally referred to herein as a "circuit," "module," or "system."

[0075] The present invention may be a system, method, or computer program product, or combination thereof, in any possible level of technical detail integration. The computer program product may include a computer readable storage medium (or multiple computer readable storage media) having computer readable program instructions for causing a processor to implement aspects of the present invention.

[0076] A computer readable storage medium may be a tangible device capable of holding and storing instructions for use by an instruction execution device. A computer readable storage medium may be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. A non-exhaustive list of more specific examples of computer readable storage media includes portable computer diskettes, hard disks, random access memory (RAM), read only memory (ROM), erasable programmable read only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disk read only memory (CD-ROM), digital versatile disk (DVD), memory sticks, floppy disks, mechanically encoded devices such as punch cards or grooved structures having instructions recorded thereon, and any suitable combination of the foregoing. As used herein, a computer-readable storage medium should not be construed as a transitory signal itself, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission medium (e.g., light pulses passing through a fiber optic cable), or electrical signals transmitted over electrical wires.

[0077] The computer readable program instructions described herein may be downloaded from a computer readable storage medium to the respective computing / processing device or to an external computer or storage device via a network, such as the Internet, a local area network, a wide area network, or a wireless network, or a combination thereof. The network may include copper transmission cables, optical transmission fiber, wireless transmission, routers, firewalls, switches, gateway computers, or edge servers, or a combination thereof. A network adapter card or network interface of each computing / processing device receives the computer readable program instructions from the network and forwards the computer readable program instructions for storage in a computer readable storage medium within the respective computing / processing device.

[0078] The computer readable program instructions for carrying out the operations of the present invention may be either source code or object code written in any combination of one or more programming languages, including assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state configuration data, configuration data for integrated circuits, or object oriented programming languages ​​such as Smalltalk, C++, and procedural programming languages ​​such as the "C" programming language or similar programming languages. The computer readable program instructions may be executed entirely on the user's computer, partly on the user's computer as a standalone software package, partly on the user's computer and partly on a remote computer, or entirely on a remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or wide area network (WAN), or the connection may be made to an external computer (e.g., via the Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example, a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA) can execute computer readable program instructions to individualize the electronic circuitry by utilizing state information of the computer readable program instructions to perform aspects of the invention.

[0079] Aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer readable program instructions.

[0080] These computer readable program instructions may be provided to a processor of a computer or other programmable data processing apparatus to produce a machine, such that the instructions, which execute on the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / acts specified in one or more blocks of the flowcharts and / or block diagrams. These computer readable program instructions may also be stored on a computer readable storage medium having instructions stored thereon, directing a computer, programmable data processing apparatus, or other device, or combination thereof, to function in a particular manner, such that the computer readable storage medium having the instructions stored thereon comprises an article of manufacture including instructions for implementing aspects of the functions / acts specified in one or more blocks of the flowcharts and / or block diagrams.

[0081] The computer readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause the computer, other programmable apparatus, or other device to perform a series of operable steps to create a computer-implemented process such that the instructions, which execute on the computer, other programmable apparatus, or other device, implement the functions / acts specified in one or more blocks of the flowcharts and / or block diagrams.

[0082] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block of the flowcharts or block diagrams may represent a module, segment, or portion of instructions, including one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions shown in the blocks may occur out of the order shown in the figures. For example, two blocks shown in succession may actually be performed as one step, or may be executed simultaneously, substantially simultaneously, partially or fully in a time-overlapping manner, or the blocks may sometimes be executed in reverse order depending on the functionality involved. It should also be noted that each block of the block diagrams and / or flowchart illustrations, and combinations of blocks in the block diagrams and / or flowchart illustrations, may be implemented by a special-purpose hardware-based system that performs the specified functions or actions or executes a combination of special-purpose hardware and computer instructions.

[0083] While the forgoing is directed to embodiments of the present invention, other and further embodiments of the invention may be devised without departing from the basic scope thereof, which scope is determined by the following claims.

Claims

1. Providing a cache coherent computing system including a plurality of boards, each board including a plurality of processors; requesting cache data from a first core of a first processor on a first board of the plurality of boards; if it determines that another core on the first processor does not have the requested cached data, broadcasting a request to other processors on the first board to determine whether one of the other processors is an Intervening Master (IM) of the requested cached data; upon determining that none of the other processors on the first board are the IM, broadcasting a request to other boards of the plurality of boards to determine whether one of the processors on the other boards is the IM; The method includes:

2. broadcasting the request to the other processors on the first board includes: receiving, in the first processor, a partial response (PRESP) from the other processor on the first board; determining which of the other processors are still engaged in the request and other operations initiated by the other processors on the same cached data based on the PRESP, wherein at least one of the other processors indicates in the PRESP that it is not engaged in the request; sending a combined response (CRESP) to only those other processors that are still participating in the request; The method of claim 1 further comprising:

3. 3. The method of claim 2, wherein the other processors still participating in the request include a fork processor in the first board assigned to communicate with a respective board of the plurality of boards.

4. 4. The method of claim 3, wherein the at least one of the other processors not involved in the request is a processor on the first board that is not a fork processor, does not have an in-flight operation on the same cached data, and is not the IM of the requested cached data.

5. broadcasting the request to the other boards of the plurality of boards receiving said request at a respective bifurcation processor on each of said other boards; broadcasting said request from each of said bifurcated processors to said other processors within the same board; receiving, at each of said branch processors, a PRESP reporting another operation in-flight on said same cache data from said other processor in said same board; transferring the PRESP from the other processor in the other board to the first processor on the first board; sending a board-scope CRESP from each of said branch processors to only those other processors in said same board that have indicated based on said PRESP that they are still interested in said request; sending a global scope CRESP from said first board to said other processors in said other boards that have indicated based on said PRESP that they are still interested in said request; The method of claim 1 further comprising:

6. 6. The method of claim 5, wherein a first of the other processors that indicates it is still involved in the request is the IM of the requested cached data or has an in-flight operation on the same cached data, and a second of the other processors that indicates it is still involved in the request is not the IM of the requested cached data.

7. receiving, at the first processor, a reset response (RRESP) from the second one of the other processors that is not the IM of the requested cache data; sending, after receiving the RRESP, from the first processor, an end of coherency response (ERESP) to the first of the other processors that is the IM of the requested cache data; The method of claim 6 further comprising:

8. 8. The method of claim 7, wherein the ERESP is sent from the first processor to the first of the other processors only after the first processor has received a respective RRESP from all of the other processors indicating that they are still involved in the request but are not the IM of the requested cached data or have another operation in-flight for the same cached data.

9. receiving an RRESP from a first one of the other processors, the first processor becoming the IM for the requested cached data; receiving an RRESP from one of the other processor that is the memory home of the requested cached data and any remaining processors participating in the request that did not have another operation in-flight on the same cached data; The method of claim 7 further comprising:

10. A cache coherent computing system including a plurality of boards, each board including a plurality of processors; a first core of a first processor on a first board of the plurality of boards is configured to request cache data that is not stored in a cache assigned to the first core; configured, upon determining that another core on the first processor does not have the requested cached data, to broadcast a request to other processors on the first board to determine whether one of the other processors is an Intervening Master (IM) for the requested cached data; and upon determining that none of the other processors on the first board is the IM, the first processor is configured to broadcast a request to other boards of the plurality of boards to determine whether one of the processors of the other boards is the IM. Cache coherent computing system.

11. broadcasting the request to the other processors on the first board includes: receiving, in the first processor, a partial response (PRESP) from the other processor on the first board; determining, based on said PRESP, which of said other processors are still involved in said request and other operations initiated by said other processors on the same cached data, wherein at least one of said other processors indicates in said PRESP that it is not involved in said request; sending a combined response (CRESP) to only those other processors that are still participating in the request; 11. The cache coherent computing system of claim 10, further comprising:

12. the other processors still participating in the request include a fork processor in the first board assigned to communicate with each board of the plurality of boards; the at least one of the other processors not involved in the request is a processor on the first board that is not a fork processor, does not have an in-flight operation on the same cached data, and is not the IM of the requested cached data; 12. The cache coherent computing system of claim 11.

13. broadcasting the request to the other boards of the plurality of boards receiving said request at a respective bifurcation processor on each of said other boards; broadcasting said request from each of said bifurcated processors to said other processors within the same board; receiving, at each of said branch processors, a PRESP reporting another operation in-flight on said same cache data from said other processor in said same board; transferring the PRESP from the other processor in the other board to the first processor on the first board; sending a board-scope CRESP from each of said branch processors to only those other processors in said same board that have indicated based on said PRESP that they are still interested in said request; sending a global scope CRESP from said first board to said other processors in said other boards that have indicated based on said PRESP that they are still interested in said request; 11. The cache coherent computing system of claim 10, further comprising:

14. 14. A cache coherent computing system as described in claim 13, wherein a first of the other processors that indicates it is still involved in the request is the IM of the requested cached data or has an in-flight operation on the same cached data, and a second of the other processors that indicates it is still involved in the request is not the IM of the requested cached data.

15. The first processor, receiving a reset response (RRESP) from the second one of the other processors that is not the IM of the requested cache data; After receiving the RRESP, the first processor transmits a coherency end response (ERESP) to the first of the other processors that is the IM of the requested cache data.

15. The cache coherent computing system of claim 14, configured to:

16. 1. A computer program product for maintaining a cache coherent computing system having a plurality of boards, each board including a plurality of processors, the computer program product comprising: A computer readable storage medium having computer readable program code embodied therein, said computer readable program code being executable by one or more computer processors to perform operations, said operations including: requesting cache data from a first core of a first processor on a first board of the plurality of boards; if it determines that another core on the first processor does not have the requested cached data, broadcasting a request to other processors on the first board to determine whether one of the other processors is an Intervening Master (IM) of the requested cached data; if determining that none of the other processors on the first board are the IM, broadcasting a request to other boards of the plurality of boards to determine whether one of the processors on the other boards is the IM; 2. A computer program product comprising:

17. A cache coherent computing system including a plurality of boards, each board including a plurality of processors; a first core of a first processor on a first board of the plurality of boards is configured to request cache data that is not stored in a cache assigned to the first core; Upon determining that another core on the first processor does not have the requested cached data, the first processor: broadcasting a request to other processors on the first board to determine whether one of the other processors is an Intervening Master (IM) for the requested cache data; receiving a partial response (PRESP) from the other processor of the first board; determining which of the other processors are still involved in the request based on the PRESP, where a first one of the other processors indicates in the PRESP that it is the IM and a second one of the other processors indicates in the PRESP that it is not the IM but is still involved in the request; sending a combined response (CRESP) to only those other processors that indicate in the PRESP that they are still participating in the request; 1. A cache coherent computing system configured to:

18. 20. The cache coherent computing system of claim 17, wherein the second of the other processors is a fork processor in the first board assigned to communicate with a second of the plurality of boards on behalf of the processor on the first board.

19. 20. The cache coherent computing system of claim 18, wherein the second one of the other processors is configured to send a reset response (RRESP) to the first processor after receiving the CRESP from the first processor, the CRESP indicating to the second one of the other processors that the IM has been found and that the second one of the other processors does not need to communicate with the second board.

20. 20. The cache coherent computing system of claim 19, wherein the first processor is configured to send an end coherency response (ERESP) to the first one of the other processors after receiving the RRESP from the second one of the other processors and from all processors that have reported PRESPs indicating that they have another operation in-flight on the same cached data.

21. 21. The cache coherent computing system of claim 20, wherein the first of the other processors is configured to send an RRESP to the first processor after receiving the ERESP, the first processor being a new IM for the requested cached data.

Images