System Coherence Protocol

Abstract

In an embodiment herein, a coherency protocol for a distributed computing topology that tolerates large scale stalls on various interfaces is described. In one embodiment, the computing topology includes multiple boards, each including multiple processors. When a particular core on a processor wants to access data that is not currently stored in its cache, the core may first initiate a request to search for a cache line in the caches of other cores on the same processor. If the cache line is not found, the cache coherency protocol allows the processor to then broadcast a request to other processors on the same board. If the processors on the same board do not have the data, the processor may then broadcast a request to other boards in the system. The processors in those boards may then search their caches to identify the data.

Description

Technical Field

[0001] The present invention relates to a coherence protocol in a shared memory multi-processing (SMP) computing system.

Background Art

[0002] New computing systems rely on advanced distributed system topologies with high-speed interfaces between processors (e.g., between processor chips). High interface speeds lead to reliability issues and the need for new bus recovery mechanisms. The interface recovery mechanism of a high-speed interface can lead to large-scale stalls and unpredictable message propagation speeds. This causes various problems in maintaining data integrity in a cache coherent system. Therefore, in a distributed system topology with processors distributed across different boards or platforms, it is necessary to improve the cache coherence protocol.

Summary of the Invention

[0003] According to one embodiment of the present invention, the method provides a cache-coherent computing system comprising a plurality of boards, each board comprising a plurality of processors; requests cache data from a first core of a first processor on a first board of the plurality of boards; if another core on the first processor does not have the requested cache data, broadcasts the request to the other processors on the first board to determine whether one of the other processors is the intervention master (IM) for the requested cache data; and if none of the other processors on the first board are IMs, broadcasts the request to the other boards of the plurality of boards to determine whether one of the processors on the other boards is an IM. One advantage is that this cache-coherency method is tolerant of delays that change as cache operations move to higher scopes (e.g., board-to-board communication).

[0004] The method may also include, in the first processor, receiving partial responses (PRESPs) from other processors on the first board, determining, based on the PRESPs, which of the other processors are still involved in the request and other operations initiated by the other processors for the same cache data, determining that at least one of the other processors is not involved in the request, and sending coupled responses (CRESPs) only to the other processors that are still involved in the request. One advantage is that this limits the number of processors that continue the coherency action, thereby allowing the processors that are not involved to perform other functions.

[0005] The method also includes including fork processors within the first board that are assigned to communicate with each of the boards of the multiple boards, as other processors still involved in the request. One advantage is that, at this point, the first processor does not yet know whether the IM for the requested data is on the same board, so the fork processors may still be used to send to the other boards.

[0006] The method also includes receiving a request at each branch processor on each of the other boards, broadcasting the request from each branch processor to other processors on the same board, receiving a PRESP at each branch processor reporting other in-flight operations for the same cache data from other processors on the same board, forwarding the PRESP from other processors on the other boards to the first processor on the first board, sending a board-scoped CRESP from each branch processor only to other processors on the same board that have indicated, based on the PRESP, that are still involved in the request, and sending a global-scoped CRESP from the first board to other processors on the other boards that have indicated, based on the PRESP, that are still involved in the request. One advantage of board-scoped and global-scoped CRESPs is that they include only processors that are still in board-scoped and global-scoped and are still involved in cache operations.

[0007] The method also includes the first processor, among other processors still indicating involvement in the request, being the IM for the requested cache data, or a second processor, among other processors having in-flight action on the same cache data and still indicating involvement in the request, not being the IM for the requested cache data. One advantage is that any processor requesting the same cache data as the first processor receives the communication and knows that the IM has changed.

[0008] According to one embodiment of the present invention, a cache-coherent computing system comprises a plurality of boards, each board comprising a plurality of processors, wherein a first core in a first processor on a first board of the plurality of boards is configured to request cache data not stored in the cache allocated to the first core. If the first processor determines that another core on the first processor does not have the requested cache data, the first processor is configured to broadcast the request to the other processors on the first board to determine whether one of the other processors is the intervention master (IM) for the requested cache data. If the first processor determines that none of the other processors on the first board are IMs, the first processor is configured to broadcast the request to the other boards of the plurality of boards to determine whether one of the processors on the other boards is an IM. One advantage is that this cache-coherency method is tolerant of the delays that change as cache operations move to higher scopes (e.g., board-to-board communication).

[0009] According to one embodiment of the present invention, a computer program product for maintaining a cache-coherent computing system comprising multiple boards, wherein each board comprises multiple processors, and the computer program product comprises a computer-readable storage medium having embodied computer-readable program code, the computer-readable program code being executable for one or more computer processors to perform operations. The operation includes requesting cache data from a first core of a first processor on a first board of the multiple boards, and if it is determined that another core on the first processor does not have the requested cache data, broadcasting the request to other processors on the first board to determine whether one of the other processors is an intervention master (IM) for the requested cache data, and if it is determined that none of the other processors on the first board are IMs, broadcasting the request to other boards of the multiple boards to determine whether one of the processors on the other boards is an IM. One advantage is that this cache-coherency method is tolerant of the delays that change as cache operations move to higher scopes (e.g., board-to-board communication).

[0010] According to one embodiment of the present invention, a cache-coherent computing system comprises a plurality of boards, each board comprising a plurality of processors, wherein a first core in a first processor on a first board of the plurality of boards is configured to request cache data not stored in the cache allocated to the first core. When the first processor determines that another core on the first processor does not have the requested cache data, the first processor broadcasts the request to the other processors on the first board to determine whether one of the other processors is the intervention master (IM) for the requested cache data, receives a partial response (PRESP) from the other processors on the first board, and determines, based on the PRESP, which of the other processors is still involved in the request, by determining that the first processor among the other processors indicates in the PRESP that it is the IM, and the second processor among the other processors indicates in the PRESP that it is not the IM but is still involved in the request, and sends a combined response (CRESP) only to the other processors that indicate in the PRESP that they are still involved in the request. One advantage is that this cache coherency method is tolerant of the latency that changes as cache operations shift from intra-processor communication to inter-processor communication between processors on the same board.

[0011] Herein, embodiments of the present invention will be described as mere examples with reference to the accompanying drawings. [Brief explanation of the drawing]

[0012] [Figure 1] This figure shows a cache-coherent computing system with a distributed topology according to one embodiment. [Figure 2] This figure shows a processor chip according to one embodiment. [Figure 3] This figure shows a communication system between processors on a board according to one embodiment. [Figure 4] This figure shows the communication between boards in a cache-coherent computing system according to one embodiment. [Figure 5] This is a flowchart for identifying cache data in a distributed topology, according to one embodiment. [Figure 6A] This figure shows how cache data is identified within a board including multiple processors, according to one embodiment. [Figure 6B] This figure shows how cache data is identified within a board including multiple processors, according to one embodiment. [Figure 6C] This figure shows how cache data is identified within a board including multiple processors, according to one embodiment. [Figure 6D] This figure shows how cache data is identified within a board including multiple processors, according to one embodiment. [Figure 6E] This figure shows how cache data is identified within a board including multiple processors, according to one embodiment. [Figure 6F] This figure shows how cache data is identified within a board including multiple processors, according to one embodiment. [Figure 7A] This figure shows how cache data is identified within a multi-board according to one embodiment. [Figure 7B] This figure shows how cache data is identified within a multi-board according to one embodiment. [Figure 7C] This figure shows how cache data is identified within a multi-board according to one embodiment. [Figure 7D] This figure shows how cache data is identified within a multi-board according to one embodiment. [Figure 7E] This figure shows how cache data is identified within a multi-board according to one embodiment. [Figure 7F] This figure shows how cache data is identified within a multi-board according to one embodiment. [Figure 7G] This figure shows how cache data is identified within a multi-board according to one embodiment. [Figure 7H] This figure shows how cache data is identified within a multi-board according to one embodiment. [Modes for carrying out the invention]

[0013] Embodiments of this specification describe a coherence protocol for a distributed computing topology that allows for large-scale stalls on various interfaces (e.g., interfaces between different motherboards, including processor chips). In one embodiment, the computing topology includes multiple boards (e.g., multiple motherboards, also called "drawers"), each having multiple processors (e.g., multiple processor chips). When a particular core on a processor wants to access data that is not currently stored in its cache, the core may first initiate a request to search for a cache line in the cache of other cores on the same processor. If no cache line is found, the processor then broadcasts the request to other processors on the same board. For example, processors on a board may be connected via a fully connected fabric. If a processor on the same board does not have the data, the processor then may broadcast the request to other boards in the system. Processors on those boards may then search their caches to identify the data.

[0014] One problem with performing coherent operations in a multi-scope system is that the latency associated with performing cache operations (e.g., snooping, cache responses, etc.) can vary significantly at each level or scope (e.g., contrasting searching a cache within the same processor, searching a cache within a processor on the same board, and searching a cache within a processor on a different board). The multi-scope coherence protocol described herein guarantees fast data sharing at lower scopes (e.g., when searching a cache within the same processor) and may tolerate increased latency as requests proceed to higher scopes (e.g., searching a cache within a processor on the same board or searching a cache within a processor on a different board). The multi-scope coherence protocol may be tolerant of the varying delays as cache operations move to higher scopes. Advantageously, the multi-scope coherence protocol may be used in different system topologies and still maintain data integrity.

[0015] FIG. 1 shows a cache-coherent computing system 100 having a distributed topology, according to one embodiment. In this example, the topology of system 100 is distributed because processors 110 (e.g., central processing units (CPUs)) are distributed across different boards 105 (e.g., different motherboards or drawers) within the computing system. The communication technology used to communicate between processors 110 on the same board 105 may be different from the communication technology used to communicate between processors 110 on different boards 105. For example, message communication or performing cache operations between processors 110 on different boards 105 may generally incur more latency than performing cache operations between processors 110 on the same board 105.

[0016] Computing system 100 may be any computing system such as a server, blade server, rack server, tower server, desktop computer, laptop, etc. In one embodiment, computing system 100 includes a chassis that includes the hardware elements shown in FIG. 1. In one embodiment, the chassis may be attachable, whereby computing system 100 can be attached to a rack or tower.

[0017] In this example, computing system 100 includes two boards, boards 105A and 105B, each including a multi-processor 110. However, computing system 100 may have any number (e.g., 3, 4, 8, etc.) of boards, each including any number (e.g., 2, 4, 6, 8, 16, etc.) of processors. Each processor 110 includes one or more (e.g., 1, 4, 8, etc.) cores 115 and a cache 120. Some of the cache may be integrated into the core (e.g., level 1 (L1)) cache, and other caches 120 may be external to the core 115 (e.g., level 2 (L2) and level 3 (L3) caches). Embodiments herein are not limited to any particular number of cache levels or cache hierarchies. Although not shown, computing system 100 may also include main memory (e.g., random access memory) that may be separated from or disposed on the board 105. Computing system 100 may also include long-term storage such as a hard drive.

[0018] In a shared-memory multiprocessor system such as computing system 100, which has isolated caches 120 for each processor 110, it is possible to have many copies of shared data, for example, one copy in main memory (not shown) and one copy in the local cache of each processor that requested it. However, maintaining data coherence in a highly distributed topology using high-speed interfaces with long-term recovery mechanisms is difficult, and one can benefit from the techniques described herein. For example, to ensure data integrity of a cache line in transit, the former owner of the cache line should continue to protect the cache line until the new owner begins protecting the data, and all other in-flight operations on the same cache line (for example, when two cores 115 are requesting the same cache line simultaneously) are rejected.

[0019] Embodiments herein establish several coherency protocol rules to address computing systems having a distributed topology. In this example, the topology of computing system 100 has three distinct scopes: (1) processor scope, (2) board scope, and (3) inter-board scope (or global scope). The multi-scope coherency protocol described herein addresses the heterogeneous nature of the system topology of system 100, which has limited resources, and enables fast and fair line sharing when communicating between processors on the same board 105 and different boards 105. Embodiments herein introduce coupled responses (CRESPs) that do not depend on other operations to be completed; that is, no other operation can stop the creation of a CRESP. Furthermore, the multi-scope coherency protocol can be time-delay-tolerant (e.g., time-independent) by using a coherency protocol (also called a fabric (FBC) protocol) that handles any delays arising from cache operations transmitted between processors 110 on the same board 105 and different boards 105. The coherency protocol is responsible for handling off-chip broadcasts and all associated messages, as well as coherency modes that maintain data coherency. Furthermore, the intervention master (IM) can complete its task without depending on other operations from the same or lower scopes; that is, nothing prevents the IM from completing the handoff of the requested cache line to a new owner. As used herein, the IM is the highest point of coherency for a given scope and contains the most recent cache copy of the cache line. The IM is typically a cache on processor 110 (or core 115 within processor 110), but may be memory (e.g., cache 120) if the current core does not own the cache line.Furthermore, in one embodiment, the multiscope coherency protocol has request observability, in which case an actor outside the current scope cannot prevent a request from broadcasting requests within that scope.

[0020] Figure 2 shows a processor 110 according to one embodiment. In this embodiment, the processor 110 is an integrated circuit (e.g., a single chip). The processor 110 in Figure 2 is an example of an embodiment of the processor 110 shown in Figure 1.

[0021] The processor 110 includes eight cores (cores 0-7) and eight caches (caches 0-7). In one embodiment, each cache is owned by a core. For example, cache 0 is owned by core 0, cache 1 is owned by core 1, and so on. As shown in the figure, a core has direct connections to the caches it owns, but does not have direct connections to the caches it does not own. The caches may be L1, L2, or L3 caches.

[0022] Processor 110 also includes X-FBC205 and M-FBC210. ​​In Figure 2, the designation "X" indicates a circuit tasked with communicating with a processor on the same board, but in a different socket, and the designation "M" indicates a circuit on the board tasked with communicating with a processor in the same socket as processor 110. These sockets and connections are described in Figure 3. However, the embodiments described herein are also applicable to computing systems, in which case the processors are not located (or packaged) in different sockets on the board.

[0023] FBC205 and 210 are fabric logic that manages coherence between multiple processors. That is, processor 110 uses FBC205 and 210 when sending and receiving cache operations with other processors. For example, when processor 110 sends a request for a cache line to a processor on the same socket, it communicates with the processor using M-FBC210. ​​However, when processor 110 sends a request for a cache line to a processor on a different socket, it communicates with the processor using X-FBC205.

[0024] Processor 110 also includes X-bus I / O (IO) 215 and M-bus IO 220. Similar to FBC 205 and 210, X-bus IO 215 is an I / O interface for communicating with processors on different sockets, and M-bus IO 220 is an I / O interface for communicating with processors on the same socket. Again, other topologies may have different configurations that do not package the processors in sockets. Processor 110 also includes A-bus IO 240, which the FBC unit uses to connect with other boards.

[0025] In this example, processor 110 uses rings 225 and 230 to communicate between caches. The two rings are used in Figure 2 to illustrate that different types of data can be transmitted between caches. For example, the outer ring 225 may represent the path used by coherence messages (e.g., snoop and response messages), and the inner ring 230 may represent the data bus between caches.

[0026] Suppose Core 2 wants to read and process data in a specific cache line. Core 2 first searches its internal cache (if any), and if the cache line is not found (i.e., a cache miss), it queries its own cache, i.e., Cache 2. If Cache 2 does not own the cache line, Core 2 sends a request to other cores to search those caches for the cache line. This request may travel to different caches along ring 225. In one embodiment, the request may be sent unidirectionally or simultaneously in both directions. The cache / core may then respond to the request by indicating whether or not it owns the requested cache line.

[0027] Figure 2 shows the use of a ring topology for performing cache operations in processor 110, but this is only an example. The embodiments herein are not limited to any particular technique for performing cache operations in a single chip / processor, but can instead be used in conjunction with any other type of technique for performing cache operations in a single processor, such as mesh networks or crossbars. Thus, rings 225, 230 are just one technique suitable for propagating cache operations within a chip.

[0028] Figure 3 shows a communication system between processors 110 on board 105 according to one embodiment. For example, the computing system 100 in Figure 1 may have multiple instances of board 105 shown in Figure 3.

[0029] In this example, the board 105 includes four sockets 305A to D, each containing two processors 110. The sockets 305 may be coupled to a common board (e.g., a printed circuit board). For example, processors 110 within the same socket 305 may be packaged together and arranged on a common board (e.g., an interposer). However, embodiments described herein are not limited to boards 105 having sockets and may include embodiments in which the processors 110 are directly mounted to the board 105.

[0030] Board 105 includes a fully connected fabric 310 for communicatively coupling processors 110 located on different sockets 305. In this example, each processor in each socket has direct connections via the fully connected fabric 310 to each processor in a different socket. That is, processor CP-0 has direct connections to processors CP-2 / CP-3 in socket 305B, processors CP-4 / CP-5 in socket 305C, and processors CP-6 / CP-7 in socket 305D. Referring again to Figure 2, these six connections to six processors can be made using the six X-bus IO215s on top of processor 110.

[0031] Board 105 also includes an inter-socket connection 315 for connecting processors within the same socket 305. That is, processor CP-0 communicates with processor CP-1 using inter-socket connection 315, processor CP-2 communicates with processor CP-3 using inter-socket connection 315, and so on. Referring again to Figure 2, inter-socket connections to processors within the same socket may be made using one of the M-bus IO220s at the bottom of processor 110.

[0032] The fully connected fabric 310 and the inter-socket connections 315 ensure that each processor 110 has a direct communication link to any other processor 110 on the same board 105. That is, resources are guaranteed that one processor can use to communicate with all other processors on the same board. In this example, “direct communication link” means that a processor 110 can communicate with another processor on the same board 105 without using another processor as an intermediary (for example, even in a spine-leaf topology). Furthermore, the fully connected fabric 310 can be time-independent in that cache coherence can be maintained even when there is additional latency or transmission errors.

[0033] Figure 4 shows board-to-board communication in a cache-coherent computing system according to one embodiment. Each board 105 (boards 0-3) contains four sockets and eight processors, as shown in Figure 3. Straight lines indicate board-to-board connections between boards. In this example, each board has sockets (i.e., two processors) assigned to communicate with different boards, called fork processors. Using board 0 as an example, processors CP-2 / CP-3 are connected to the pair of processors on board 2, processors CP-4 / CP-5 are connected to the pair of processors on board 1, and processors CP-6 / CP-7 are connected to the pair of processors on board 3. When a processor wants to send a cache message to a different board, it can use a fork processor to reach that board. For example, if processor CP-0 on board 0 wants to send a response to the processor on board 3, processor CP-0 first sends the response to one of the processors CP-6 / CP-7 on board 0, and then processor CP-6 / CP-7 on board 0 forwards that response to processor CP-6 / CP-7 on board 3.

[0034] In one embodiment, a processor assigned to communicate with other boards may have a ticket manager that manages cache operations flowing between boards. The ticket manager may use a queue to determine which cache operation to send next.

[0035] Unlike Figure 3, where a processor has direct connections to any other processor, in Figure 4, a processor on one board does not necessarily have direct communication links to processors on different boards. That is, a processor may have to rely on one or two other processors (e.g., fork processors) to communicate cache requests or responses to processors on different boards. Thus, Figure 4 can be called a pass-through topology, in which one processor is used as an intermediary between other processors on different boards. For example, CP-0 on board 0 relies on processors CP-2 / CP-3 to reach board 2, on processors CP-4 / CP-5 to reach board 1, and on processors CP-6 / CP-7 to reach board 3.

[0036] Figure 5 is a flowchart of method 500 for identifying cache data in a distributed topology according to one embodiment. In block 505, the requester (e.g., a core on the processor) determines whether the cache on the processor has the requested data (e.g., a specific cache line). For example, the requester may search the internal L1 cache (if any) and allocated L2 caches (e.g., caches 0-7 in Figure 2) to determine whether these caches have the latest copy of the requested data. For example, the requesting core may use the processor's communication network (e.g., a ring topology or mesh network in Figure 2) to determine whether caches allocated to other cores in the processor contain the requested data. If the data is found to be present, method 500 proceeds to block 510, where the requester retrieves the latest version from the cache in the processor.

[0037] However, if no cache within the processor has the latest version of the requested data, method 500 proceeds to block 515, where the processor broadcasts the request to other processors on the same board. For example, a processor may broadcast a request to processors on the same socket and different sockets on board 105 using the fully connected fabric 310 and inter-socket connections 315 shown in Figure 3. However, the connections shown in Figure 3 are just one example of connecting processors located on the same board.

[0038] In block 520, the processor determines whether it is the IM for the requested data. That is, after receiving a broadcast, each processor determines whether it is the IM for that data. That is, the processor determines whether it is the highest point of coherency and contains the latest cached copy of the requested data.

[0039] Details on how to search for processors on the same board and determine whether one of these processors is the IM for the requested data are explained in Figures 6A to 6F below.

[0040] If one of the processors on the same board is the IM, the method proceeds to block 520, where the IM is changed to the requesting processor. In doing so, the requesting processor is obligated to protect the cache lines, as will be described in more detail later. In addition, in block 510, the requested data may be copied from the cache in the processor that was previously the IM to the requesting processor.

[0041] If none of the processors on the board is an IM for the requested data, the method proceeds to block 530, where the requesting processor broadcasts the request to other processors on the board to determine whether those processors are IMs for the data (or whether the data is not owned by any core but is instead owned by a cache within the processor).

[0042] Details on searching for processors on different boards and determining whether one of these processors is the IM for the requested data are explained in Figures 7A to 7H below.

[0043] Method 500 assumes that one of the processors on a different board is the information processor (IM) for the requested data. In block 520, the IM is changed to the requesting processor, and in block 510, the requested data is copied from the cache in the processor that was previously the IM to the requesting processor. However, if no processor in the computer system is the IM for the requested data, the computing system may instead retrieve the data from main memory or long-term storage (e.g., a hard drive).

[0044] Figures 6A to 6F illustrate the identification of cache data within a board 105 containing a multiple processor 110 according to one embodiment. For ease of explanation, the board 105 has the same configuration of sockets and processors as shown in Figures 3 and 4, except that the eight processors are labeled CP_A to CP_H. Figures 6A to 6F illustrate steps that may be performed in blocks 515 and 520 to identify whether a processor on the board (e.g., at the board scope or level) is the IM of the requested data.

[0045] In this example, processor CP_E is the home processor requesting data (e.g., cache lines). That is, processor CP_E wants to become the IME of the data so that it can be sure that it has the most up-to-date copy of the data. Figure 6A shows that processor CP_E broadcasts the request to all other processors on the same board 105. For example, processor CP_E may broadcast the request to processors CP_A, CP_B, CP_C, CP_D, CP_F, CP_G, and CP_H, respectively, using the fully connected fabric 310 and inter-socket connections 315 in Figure 3.

[0046] Figure 6B shows that processor CP_E receives partial responses (PRESPs) from each of the processors. PRESPs may be received at different times. As mentioned above, board scope can be time-tolerant so that PRESPs from processors may be received at different times, for example, based on the available bandwidth of the connection and the processor usage. In one embodiment, processor CP_E waits until all PRESPs have been received before proceeding with processing.

[0047] In one embodiment, PRESP indicates whether a processor is an IM, or if the IM is on a different board, whether the processor may be required to identify the IM. In this case, processor CP_B is the IM for the requested data (labeled as board IM1 (BIM1)). Therefore, its PRESP indicates to processor CP_E that processor CP_B is the IM for the data. Furthermore, processors CP_A, CP-C, and CP_G are labeled "forks" in Figures 6A to 6F, indicating that these processors are assigned to be used for communication with other boards. That is, if the IM is not located on board 105, processors CP_A, CP-C, and CP_G will then be required to broadcast requests to other boards in the computing system, as illustrated in Figures 7A to 7H. For example, processor CP_A may be responsible for exchanging cache operation messages with board 1, processor CP_C may be responsible for exchanging cache operation messages with board 2, and processor CP_G may be responsible for exchanging cache operation messages with board 3.

[0048] At the point shown in Figure 6B, the fork processor does not yet know whether the IM is located on board 105 when sending the PRESP; therefore, these PRESPs indicate that the fork processor will remain in standby mode while processor CP_E broadcasts a request for data to other boards. In other words, the fork processor does not indicate in the PRESP to processor CP_E that its part of the cache operation is complete.

[0049] In contrast, processors CP_D, CP_F, and CP_H are neither IMs for the requested data nor fork processors used for communication with other boards. Therefore, the PRESP sent by processors CP_D, CP_F, and CP_H indicates to processor CP_E that their part in the cache operation is complete and those processors are no longer involved in the request.

[0050] Figure 6C shows that processor CP_E sends a Combined Response (CRESP) in response to receiving all PRESPs (i.e., partial responses from other processors on board 105). As illustrated, the CRESP is not broadcast to all processors on board 105, but only to processors that have indicated they are still part of the cache operation (i.e., the processor that is the IM of the requested data, the home memory controller of the data, a processor with an active outstanding request for the same cache line, or a fork processor used to communicate with other boards in the computing system). Therefore, processors CP_D, CP_F, and CP_H do not receive the CRESP.

[0051] CRESP notifies the fork processors CP_A, CP-C, and CP_G that the data IM has been found, so that these processors no longer need to send messages to the other boards. In other words, the fork processors are no longer needed to complete the cache operation. On the other hand, CRESP notifies the IM (i.e., processor CP_B) that it is still needed to complete the operation.

[0052] Figure 6D shows that the fork processors CP_A, CP_C, and CP_G send a reset response (RRESP) to processor CP_E. These RRESPs inform processor CP_E that processors CP_A, CP_C, and CP_G have finished their cache operations.

[0053] Figure 6E shows that processor CP_E sends an end-of-coherency response (ERESP) to processor CP_B. The ERESP informs processor CP_B that it is safe to stop protecting the requested data and that the home processor CP_B is now responsible for protecting the requested data. In other words, the ERESP indicates that it is safe for the old IM (i.e., processor CP_B) to reset coherency protection. The old IM can then stop rejecting new requests for the data. The ERESP indicates that the new IM (i.e., processor CP_E) is now capable of handling further requests for the data. Furthermore, the ERESP informs the old IM that the new IM will ensure that all potential observers are satisfied.

[0054] In Figure 6F, in response to receiving ERESP, processor CP_B sends RRESP to processor CP_E, indicating that the cache operation has ended. Furthermore, ERESP indicates to processor CP_E that any other requester for the same data (e.g., another processor on that board or a different board) should have observed the transfer and that the data is being transferred. This thus prevents conflicts when processors requesting the same data in parallel may not know which processor is currently the IM for that data.

[0055] Figures 7A to 7H illustrate the identification of cache data in a multiplexed board according to one embodiment. For ease of explanation, each board 105A to D in Figures 7A to 7H has the same socket and processor configuration as board 105 in Figures 6A to 6F. Figures 7A to 7H illustrate steps that may be performed in block 530 to identify whether a processor on a different board (e.g., inter-board scope or level) is the IM of the requested data.

[0056] Figures 7A to 7H assume that blocks 505 to 525 of method 500 have already been executed and the requesting processor has determined that (i) it does not have the latest cache copy of the requested data and (ii) the IM of the data is not on the same board. Therefore, the requesting processor broadcasts a request to other boards in the system to identify the IM.

[0057] In Figure 7A, as in Figure 6A, processor CP_E is the requesting processor searching for the IM of the requested data. However, unlike in Figure 6A, the IM of the data is not on the same board (i.e., board 105A), but on a different board. Specifically, the IM of the requested data (labeled as global IM (GIM1)) is on processor CP_F on board 105B.

[0058] Processor CP_E determined that none of the processors on board 105A had the requested data as an IM (Input Method), and therefore instructed the fork processors CP_A, CP_C, and CP_G on board 105A to broadcast the request for the IM to the other boards 105B-D, as shown in Figure 7A. The request sent to board 105B was received by processor CP_G (e.g., the branch processor), the request sent to board 105C was received by processor CP_C (e.g., the branch processor), and the request sent to board 105D was received by processor CP_A (e.g., the branch processor). Each of these branch processors on the three boards 105B-D then broadcasts the request for the IM to the processors on the same board. That is, each branch processor broadcasts the request as shown in Figure 6A.

[0059] Figure 7B shows that branch processors on each board 105B-D collect partial response PRESPs from each processor on the same board. As explained in Figure 6B, other processors can tell the processor broadcasting the request (e.g., a branch processor) whether they are the data IM.

[0060] Figure 7B further shows that the branch processor forwards the PRESP to the fork processors CP_A, CP_C, and CP_G on board 105A, and that the fork processors CP_A, CP_C, and CP_G similarly forward the PRESP to the requesting processor CP_E. Based on the response, the requesting processor CP_E may determine that processor CP_F on board 105B is the IM for the requested data.

[0061] Figure 7C shows a board-scope CRESP. As shown, each branch processor on boards 105B-D sends a CRESP based on a PRESP received from another processor on the same board. These board-scope CRESPs may contain the same information as the CRESP described in Figure 6C above.

[0062] Figure 7D shows a global-scope CRESP (or cross-board-scope CRESP). The global CRESP originates from the requesting processor CP_E on board 105A. The CRESP is sent only to the fork processors within board 105A, and the fork processors similarly forward the CRESP to each board. As illustrated, the branch processors on boards 105B-105D forward the CRESP only to the processors that are still part of the operation. That is, since only processor CP_F within board 105B is the IM for the data, PRESPs sent by other processors within board 105B indicate that those processors are no longer IMs and therefore do not need to be involved in the cache operation. Thus, only processor CP_F and the branch processors within board 105B receive the CRESP. On board 105C, the CRESP is forwarded only from branch processor CP_C to processor CP_F, which functions as an LFAR-X. An LFAR-X indicates that CP_F has an ongoing request for the same line (e.g., LFAR-X indicates "comparison" or "conflict detection"). On board 105D, the CRESP is forwarded from branch processor CP_A only to processor CP_F, which acts as the memory home for the requested data. Processor CP_F remained part of the cache operation because the memory home for the requested data could be the IM for the data if another processor had not yet been allocated the IM. In this case, since processor CP_F on board 105B was the IM, the CRESP notifies processor CP_F on board 105D that the IM has been found and it should no longer be involved in the cache operation. One of the advantages of board-scope and global-scope CRESPs is that they only include board-scope and global-scope processors that are still involved in the cache operation.

[0063] Since CP_F on board 105C has a request for the same line in flight, the controller handling the request originating from board 105A waits for board 105C to receive a rejection CRESP before sending an RRESP back to the CP_E chip on 105A. The comparison is reported on PRESP, and the home processor waits for RRESPs from all chips that reported comparisons on PRESP before sending an ERESP to the GIM1 processor. Figure 7E shows that the home processor CP_E on board 105 receives an RRESP from a non-IM processor (e.g., CP_F on board 105C) indicating the end of the comparison operation. In Figures 7A-7E, the IM moves from CP_F on board 105B to CP_E on board 105A. LFAR_X on CP_F on board 105C also accesses this cache line, hoping to maintain coherence so that the system can reliably observe the IM's movement. This is done by preventing LFAR_X on board 105A's CP_E from completing until it is determined that LFAR_X on board 105C's CP_F has been rejected by either board 105B's CP_F or board 105E's CP_E. At the time of the PRESP in Figure 7B, board 105C's CP_F reports the comparison, and the state machine is loaded depending on LFAR_X. When LFAR_X on board 105C's CP_F is rejected, the dependency is resolved, and the state machine sends an RRESP. When LFAR_X on board 105A's CF_E receives all RRESPs from non-IM processors, all dependencies encountered on the snoop are resolved, and all other requests observe an IM transfer.

[0064] Figure 7F shows that the requesting data IM (i.e., processor CP_F on board 105B) and the data memory home (i.e., processor CP_F on board 105D) send an RRESP to the requesting processor. In one embodiment, before sending an ERESP to the current IM, the requesting processor waits until it receives RRESPs from all non-IM processors that indicate in a PRESP that they have outstanding requests for the same line.

[0065] Figure 7G shows that the requesting processor CP_E sends a coherency termination ERESP to IM. The ERESP informs processor CP_F on board 105B that it is safe to stop protecting the requested data and that the home processor CP_E is currently responsible for protecting the requested data.

[0066] In Figure 7H, in response to receiving the ERESP, processor CP_F on board 105B sends an RRESP to processor CP_E on board 105A, indicating that it has completed its cache operation. Furthermore, the ERESP indicates to processor CP_E that any other requester for the same data (e.g., another processor on that board or a different board) should have observed the transfer and that the data is being transferred. This prevents conflicts that may occur when processors requesting the same data in parallel do not know which processor is currently the IM for that data.

[0067] Furthermore, Figure 7H shows that the data memory home (i.e., processor CP_F on board 105D) and the data global IM (i.e., processor CP_F on board 105B) send an RRESP to the home processor CP_E on board 105. Here, all processors involved in the operation send an RRESP to the home processor. This is used to determine when the remote resource is available for use by another operation.

[0068] For performance reasons, the system may speculatively initiate memory access before knowing whether memory data is needed. At the snoop time in Figure 7A, the system loads the state machine and initiates memory access. The global CRESP indicates to the memory home (i.e., processor CP_F on board 105D) that memory data is not needed, but a message should still be sent indicating when the state machine on the memory chip is complete.

[0069] Before returning the RRESP to the home chip, it is ensured that all operations active on the same cache line observe the movement of the global IM to the new processor by waiting for any in-flight operations to acquire a CRESP on all processor chips that reported the comparison, and by having a home-chip delayed ERESP to the global IM processor until the home chip receives all RRESPs from all chips that reported the comparison.

[0070] In one embodiment, there are no other actions that could block the IM from requesting the original, fetching the IM, or casting out / eviction the IM. Whenever one or more requests are attempting to retrieve a cache line, it is guaranteed that at least one will win. In one embodiment, the protocol has a built-in mechanism to ensure the acquisition of any additional resources necessary to complete the operation. If a resource is held by a non-IM requester or a resource queue has been formed, there is a mechanism to cancel a non-IM requester so that the non-IM requester can free up the resource for an IM requester. Furthermore, the CRESP may include information on whether the requester has an IM license for the current scope or a higher scope.

[0071] In one embodiment, there are no other actions that could block a request from progressing to a higher broadcast scope. That is, the protocol allows all new requesters to broadcast or be observed by other in-flight actions. A request may wait until it receives an RRESP from all targets in the scope that have outstanding requests for the same cache line before sending an ERESP in the scope. Furthermore, the requesting processor may respect any address comparisons in the requesting FBC itself. This is achieved by setting scope-based and system-based pending actions of the controller, as well as by setting need-to-wait actions for each other when appropriate to ensure system coherence. Furthermore, the requesting processor may need to receive a CRESP for broadcasts in a higher scope if necessary.

[0072] The descriptions of various embodiments of the present invention are presented for illustrative purposes only and are not intended to be exhaustive or to limit oneself to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein has been selected to best describe the principles of the embodiments, their practical applications, or technical improvements to the art found in the market, or to enable those else skilled in the art to understand the embodiments disclosed herein.

[0073] As stated above, references are made to the embodiments presented in this disclosure. However, the scope of this disclosure is not limited to any specific described embodiments. Rather, any combination of features and elements, whether related to a different embodiment or not, is intended to carry out and practice the intended embodiments. Furthermore, while embodiments disclosed herein may achieve advantages over other possible solutions or prior art, whether or not a particular advantage is achieved by a given embodiment does not limit the scope of this disclosure. Accordingly, the aspects, features, embodiments, and advantages discussed herein are merely illustrative and shall not be considered elements or limitations of the appended claims unless expressly enumerated in the claims. Similarly, references to “the present invention” shall not be construed as generalizations of the subject matter of any invention disclosed herein and shall not be considered elements or limitations of the appended claims unless expressly enumerated in the claims.

[0074] Aspects of the present invention may take the form of a complete hardware embodiment, a complete software embodiment (including firmware, resident software, microcode, etc.), or an embodiment combining software and hardware embodiments, all of which may generally be referred to herein as “circuits,” “modules,” or “systems.”

[0075] The present invention may, in any possible level of technical detail, be a system, a method, a computer program product, or a combination thereof. The computer program product may include a computer-readable storage medium (or a plurality of computer-readable storage media) having computer-readable program instructions for causing a processor to carry out aspects of the present invention.

[0076] A computer-readable storage medium can be a tangible device capable of holding and storing instructions for use by an instruction execution device. A computer-readable storage medium may be, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. A non-exhaustive enumeration of more specific examples of computer-readable storage media includes portable computer diskettes, hard disks, random-access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random-access memory (SRAM), portable compact disk read-only memory (CD-ROM), digital versatile disks (DVDs), memory sticks, floppy disks, mechanically encoded devices such as punched cards or grooved structures on which instructions are recorded, and any suitable combination thereof. As used herein, computer-readable storage media should not be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses passing through optical fiber cables), or electrical signals transmitted through wires.

[0077] The computer-readable program instructions described herein may be downloaded from a computer-readable storage medium to each computing / processing device, or to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, or a wireless network, or a combination thereof. The network may include copper transmission cables, optical transmission fibers, wireless transmission, routers, firewalls, switches, gateway computers, or edge servers, or a combination thereof. The network adapter card or network interface of each computing / processing device receives computer-readable program instructions from the network and transfers the computer-readable program instructions for storage on a computer-readable storage medium within each computing / processing device.

[0078] The computer-readable program instructions for performing the operations of the present invention may be either source code or object code written in any combination of one or more programming languages, including assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, configuration data for integrated circuits, or object-oriented programming languages ​​such as Smalltalk®, C++, and procedural programming languages ​​such as the "C" programming language or similar programming languages. The computer-readable program instructions may be executed entirely on the user's computer, partially as a standalone software package on the user's computer, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or on a server. In the latter scenario, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection may be made to an external computer (for example, via the Internet using an Internet service provider). In some embodiments, for example, an electronic circuit including a programmable logic circuit, a field-programmable gate array (FPGA), or a programmable logic array (PLA) can be personalized by executing computer-readable program instructions by utilizing state information of computer-readable program instructions in order to perform aspects of the present invention.

[0079] Aspects of the present invention will be described herein with reference to flowcharts or block diagrams, or both, of methods, apparatus (systems), and computer program products according to embodiments of the present invention. It will be understood that each block in a flowchart or block diagram, or both, and combinations of blocks in a flowchart or block diagram, or both, can be implemented by computer-readable program instructions.

[0080] These computer-readable program instructions may be provided to a computer or other programmable data processing device processor to create a machine, which is executed via the processor of the computer or other programmable data processing device, in order to create means for implementing functions / operations specified in one or more blocks of a flowchart or block diagram or both. These computer-readable program instructions may also be stored in a computer-readable storage medium that stores and holds the instructions, instructing a computer, programmable data processing device, or other device or combination thereof to function in a particular way, so as to provide a product containing instructions for implementing modes of functions / operations specified in one or more blocks of a flowchart or block diagram or both.

[0081] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing device, or other device to create a computer implementation process in which instructions executed on a computer, other programmable device, or other device implement a function / action specified in one or more blocks of a flowchart or block diagram or both, causing the computer, other programmable device, or other device to execute a series of operable steps.

[0082] The flowcharts and block diagrams in the drawings illustrate the architecture, functionality, and operation of possible implementations of the system, method, and computer program product according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of instructions, containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions shown in a block may occur in a different order than shown in the drawings. For example, two consecutively shown blocks may actually be performed as a single step, simultaneously, substantially simultaneously, partially or entirely in overlapping time, or the blocks may sometimes be performed in reverse order depending on the functionality involved. It should also be noted that each block in a block diagram or flowchart, or both, and combinations of blocks in a block diagram or flowchart, or both, may be implemented by a special-purpose hardware-based system that performs a specified function or operation, or executes a combination of special-purpose hardware and computer instructions.

[0083] While the above is intended for embodiments of the present invention, other embodiments and further embodiments of the present invention may be devised without departing from its basic scope, which is determined by the following claims.

Claims

1. To provide a cache-coherent computing system including multiple boards, wherein each board includes multiple processors, Requesting cache data from the first core of the first processor on the first board among the plurality of boards, If it is determined that another core on the first processor does not have the requested cache data, it broadcasts the request to other processors on the first board to determine whether one of the other processors is the intervention master (IM) for the requested cache data, If it is determined that none of the other processors on the first board is the IM, a request is broadcast to the other boards among the plurality of boards to determine whether one of the processors on the other board is the IM. A method that includes this.

2. Broadcasting the request to the other processor on the first board is: The first processor receives a partial response (PRESP) from the other processor on the first board, Based on the PRESP, determine which of the other processors is still involved in the request and other operations initiated by the other processors with respect to the same cache data, wherein the PRESP indicates that at least one of the other processors is not involved in the request. Sending a binding response (CRESP) only to the other processors still involved in the aforementioned request. The method according to claim 1, further comprising:

3. The method according to claim 2, wherein the other processors still involved in the request include a fork processor in the first board assigned to communicate with each of the boards of the plurality of boards.

4. The method of claim 3, wherein at least one of the other processors not involved in the request is a processor on the first board that is not a fork processor, does not have in-flight operations for the same cache data, and is not the IM of the requested cache data.

5. Broadcasting the request to the other boards among the plurality of boards is The request is received in each branch processor on each of the other boards, Broadcasting the request from each of the aforementioned branch processors to the other processors on the same board, Each of the aforementioned branch processors receives a PRESP from the other processors on the same board reporting other in-flight operations regarding the same cache data, Transferring the PRESP from the other processor on the other board to the first processor on the first board, From each of the aforementioned branch processors, a board-scoped CRESP is sent only to the other processors on the same board that have indicated they are still involved in the request based on the PRESP, Sending a global-scoped CRESP from the first board to the other processor in the other board that has indicated it is still involved in the request based on the PRESP, and The method according to claim 1, further comprising:

6. The method of claim 5, wherein a first processor among the other processors indicating that it is still involved in the request is the IM of the requested cache data, or a second processor among the other processors indicating that it has an in-flight operation on the same cache data and is still involved in the request is not the IM of the requested cache data.

7. The first processor receives a reset response (RRESP) from the second processor, which is one of the other processors that is not the IM of the requested cache data, After receiving the RRESP, the first processor sends a coherency termination response (ERESP) to the first processor among the other processors which is the IM of the requested cache data. The method according to claim 6, further comprising:

8. The method according to claim 7, wherein the ERESP is transmitted from the first processor to the first processor among the other processors only after the first processor has received an RRESP from each of the other processors indicating that it is still involved in the request but is not the IM of the requested cache data or has another in-flight operation on the same cache data.

9. Receiving an RRESP from the first processor among the other processors, wherein the first processor receives the requested cache data as the IM, Receiving an RRESP from one of the other processors that is the memory home of the requested cache data, and all the remaining processors involved in the request that did not have another in-flight operation on the same cache data. The method according to claim 7, further comprising:

10. A cache-coherent computing system comprising multiple boards, each board comprising multiple processors, The first core of the first processor on the first board among the plurality of boards is configured to request cache data that is not stored in the cache allocated to the first core. If the first processor determines that another core on the first processor does not have the requested cache data, the first processor is configured to broadcast a request to other processors on the first board to determine whether one of the other processors is the intervention master (IM) for the requested cache data. If the first processor determines that none of the other processors on the first board is the IM, the first processor is configured to broadcast a request to the other boards among the plurality of boards to determine whether one of the processors on the other boards is the IM. Cache-coherent computing system.

11. Broadcasting the request to the other processor on the first board is: The first processor receives a partial response (PRESP) from the other processor on the first board, Based on the aforementioned PRESP, determine which of the other processors is still involved in the request and other operations initiated by the other processors with respect to the same cache data, wherein the PRESP indicates that at least one of the other processors is not involved in the request. Sending a binding response (CRESP) only to the other processors still involved in the aforementioned request. The cache-coherent computing system according to claim 10, further comprising:

12. The other processors still involved in the aforementioned request include a fork processor in the first board that is assigned to communicate with each of the boards of the plurality of boards, At least one of the other processors not involved in the request is a processor on the first board that is not a fork processor, does not have in-flight operations on the same cache data, and is not the IM of the requested cache data. The cache-coherent computing system according to claim 11.

13. Broadcasting the request to the other boards among the plurality of boards is The request is received in each branch processor on each of the other boards, Broadcasting the request from each of the aforementioned branch processors to the other processors on the same board, Each of the aforementioned branch processors receives a PRESP from the other processors on the same board reporting other in-flight operations regarding the same cache data, Transferring the PRESP from the other processor on the other board to the first processor on the first board, From each of the aforementioned branch processors, a board-scoped CRESP is sent only to the other processors on the same board that have indicated they are still involved in the request based on the PRESP, Sending a global-scoped CRESP from the first board to the other processor in the other board that has indicated it is still involved in the request based on the PRESP, and The cache-coherent computing system according to claim 10, further comprising:

14. A cache-coherent computing system according to claim 13, wherein a first processor among the other processors indicating that it is still involved in the request is the IM of the requested cache data, or a second processor among the other processors having in-flight operations on the same cache data and indicating that it is still involved in the request is not the IM of the requested cache data.

15. The first processor is, A reset response (RRESP) is received from the second processor among the other processors, not the IM, for the requested cache data. After receiving the RRESP, the first processor sends a coherency termination response (ERESP) to the first processor among the other processors which is the IM of the requested cache data. A cache-coherent computing system according to claim 14, configured as described above.

16. A computer program for maintaining a cache-coherent computing system comprising multiple boards, wherein each board includes multiple processors, and the computer program is: It includes a computer-readable storage medium having embodied computer-readable program code, wherein the computer-readable program code is executable for performing an operation by one or more computer processors, and the operation is Requesting cache data from the first core of the first processor on the first board among the plurality of boards, If it is determined that another core on the first processor does not have the requested cache data, it broadcasts the request to other processors on the first board to determine whether one of the other processors is the intervention master (IM) for the requested cache data, If it is determined that none of the other processors on the first board is the IM, a request is broadcast to the other boards of the plurality of boards to determine whether one of the processors on the other boards is the IM. Computer programs that include [this].

17. A cache-coherent computing system comprising multiple boards, each board comprising multiple processors, The first core of the first processor on the first board among the plurality of boards is configured to request cache data that is not stored in the cache allocated to the first core. If the first processor determines that another core on the first processor does not have the requested cache data, the first processor will: Broadcasting a request to other processors on the first board to determine whether one of the other processors is the intervention master (IM) for the requested cache data, The first board receives a partial response (PRESP) from the other processor, Based on the PRESP, determine which of the other processors is still involved in the request, wherein the first processor among the other processors indicates in the PRESP that it is the IM, and the second processor among the other processors indicates in the PRESP that it is not the IM but is still involved in the request. Sending a coupled response (CRESP) only to the other processors that indicate in the PRESP that they are still involved in the request. A cache-coherent computing system configured to perform the following actions.

18. The cache-coherent computing system according to claim 17, wherein the second processor among the other processors is a fork processor on the first board assigned to communicate with the second board of the plurality of boards on behalf of the processor on the first board.

19. The cache-coherent computing system according to claim 18, wherein the second processor among the other processors is configured to send a reset response (RRESP) to the first processor after receiving the CRESP from the first processor, the CRESP indicating to the second processor among the other processors that the IM has been found and that the second processor among the other processors does not need to communicate with the second board.

20. The cache-coherent computing system according to claim 19, wherein the first processor is configured to send a coherency termination response (ERESP) to the first processor among the other processors after receiving the RRESP from the second processor among the other processors and from all processors that have reported a PRESP indicating that they have a different in-flight operation for the same cache data.

21. The cache-coherent computing system according to claim 20, wherein the first processor among the other processors is configured to send an RRESP to the first processor after receiving the ERESP, and the first processor is a new IM of the requested cache data.

Images