System coherency protocol

By broadcasting requests in a cache coherence computing system on a multi-board cache coherence computing system to determine the IM and limiting the number of processors participating in a consistent operation, the data integrity and delay problems in the cache coherence protocol under a high-speed interface are solved, and the data integrity and delay problems in the cache coherence protocol under a high-speed interface are achieved. Cache operation efficiency and consistency in different system topology.

CN118974714APending Publication Date: 2024-11-15INTERNATIONAL BUSINESS MACHINE CORPORATION
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202380032259.1
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2022-04-04
Filing Date
2023-03-20
Publication Date
2024-11-15

AI Technical Summary

Technical Problem

In the distributed system topology of high-speed interfaces, cache coherence protocols are difficult to maintain data integrity, especially due to unpredictable communication latency between processors and pauses caused by interface recovery mechanisms.

Method used

By in cache coherence computing systems of multiple boards, the request processor broadcasts a request to other boards to determine whether there is an intervention master IM and is based on the processor's Responsively, limit the number of processors that continue to participate in consistent operations, thereby freeing up unparticipated processors to perform other functions.

Benefits of technology

This approach tolerate variable latency, ensures data integrity, and maintains efficiency and consistency of cache operations across different system topology.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118974714A_ABST
    Figure CN118974714A_ABST
Patent Text Reader

Abstract

Embodiments herein describe a consistency protocol for a distributed computing topology that allows large scale pauses over various interfaces. In one embodiment, a computing topology includes a plurality of plates, each plate including a plurality of processors. When a particular core on a processor wants to access data that is not currently stored in its cache, the core may first initiate a request to search in the cache for cache lines of other cores on the same processor. If no cache line is found, the cache coherency protocol allows the processor to broadcast requests to other processors on the same board. If the processor on the same board has no data, the processor may broadcast the request to other boards in the system. The processors in these boards may then search their caches to identify data.
Need to check novelty before this filing date? Find Prior Art

Description

Background Art

[0001] The present invention relates to a coherence protocol in a shared memory multiprocessing (SMP) computing system.

[0002] New computing systems rely on highly distributed system topologies with high-speed interfaces between processors (e.g., between processor chips). High interface speeds can cause reliability issues and require new bus recovery mechanisms. Interface recovery mechanisms on high-speed interfaces can cause large stalls and unpredictable message propagation speeds. This creates various challenges for maintaining data integrity in cache coherent systems. Therefore, there is a need for improved cache coherence protocols in distributed system topologies where processors are distributed on different boards or platforms. Summary of the invention

[0003] According to one embodiment of the present invention, a method includes: providing a cache coherent computing system including multiple boards, each board including multiple processors; requesting cache data of a first core in a first processor on a first board among the multiple boards; upon determining that another core on the first processor does not have the requested cache data, broadcasting a request to other processors on the first board to determine whether one of the other processors is an intervening master IM of the requested cache data; and upon determining that none of the other processors on the first board is the IM, broadcasting a request to other boards among the multiple boards to determine whether one of the processors in the other boards is the IM. One advantage is that this cache coherent method can tolerate variable latency when cache operations move to a higher range (e.g., board-to-board communication).

[0004] The method may also include: receiving, at the first processor, a partial response PRESP from the other processors on the first board, determining, based on the PRESP, which of the other processors are still participating in the request and other operations initiated by the other processors for the same cache data, wherein at least one of the other processors indicates in its PRESP that it is not participating in the request, and sending a combined response CRESP only to the other processors that are still participating in the request. One advantage is that this limits the number of processors that continue to perform coherence operations, thereby freeing up non-participating processors to perform other functions.

[0005] The method also includes: the other processors still participating in the request include a fork processor in the first board, the fork processor being assigned to communicate with a corresponding board in the plurality of boards. One advantage is that at this point, the fork processor can still be used to send to other boards, because the first processor does not yet know whether the IM of the requested data is on the same board.

[0006] The method further comprises: receiving the request at a corresponding branch processor of each of the other boards, broadcasting the request from the corresponding branch processor to the other processors in the same board, receiving at the corresponding branch processor a PRESP from the other processors in the same board, the PRESP reporting other operations being performed on the same cache data, forwarding the PRESP from the other processors in the other boards to the first processor on the first board, sending a board-wide CRESP from the corresponding branch processor only to the other processors in the same board indicating that they are still participating in the request based on the PRESP, and sending a global-scope CRESP from the first board to the other processors in the other boards indicating that they are still participating in the request based on the PRESP. An advantage of board-wide CRESP and global-scope CRESP is that they involve only board-wide and global-scope processors that are still participating in cache operations.

[0007] The method further comprises: indicating that a first one of the other processors that are still participating in the request is the IM of the requested cache data or has an ongoing operation for the same cache data, and indicating that a second one of the other processors that are still participating in the request is not the IM of the requested cache data. One advantage is that any processor requesting the same cache data as the first processor receives the communication and knows that the IM has changed.

[0008] According to one embodiment of the present invention, a cache coherent computing system includes a plurality of boards, each board including a plurality of processors, wherein a first core in a first processor on a first board among the plurality of boards is configured to request cache data that is not stored in a cache assigned to the first core. Upon determining that another core on the first processor does not have the requested cache data, the first processor is configured to broadcast a request to other processors on the first board to determine whether one of the other processors is an intervening master IM of the requested cache data. Upon determining that none of the other processors on the first board is the IM, the first processor is configured to broadcast a request to other boards among the plurality of boards to determine whether one of the processors in the other boards is the IM. One advantage is that this cache coherence method can tolerate variable latency when cache operations move to a higher range (e.g., board-to-board communication).

[0009] According to one embodiment of the present invention, a computer program product for maintaining a cache consistency computing system is provided, the cache consistency computing system comprising a plurality of boards, each board comprising a plurality of processors, the computer program product comprising a computer-readable storage medium having a computer-readable program code contained therein, the computer-readable program code being executable by one or more computer processors to perform operations. The operations include: requesting cache data of a first core in a first processor on a first board among the plurality of boards; when it is determined that another core on the first processor does not have the requested cache data, broadcasting a request to other processors on the first board to determine whether one of the other processors is an intervening master IM of the requested cache data; and when it is determined that none of the other processors on the first board is the IM, broadcasting a request to other boards among the plurality of boards to determine whether one of the processors in the other boards is the IM. One advantage is that this cache consistency method can tolerate variable latency when cache operations move to a higher range (e.g., board-to-board communication).

[0010] According to one embodiment of the present invention, a cache coherent computing system includes a plurality of boards, each board including a plurality of processors, wherein a first core in a first processor on a first board of the plurality of boards is configured to request cache data that is not stored in a cache allocated to the first core. Upon determining that another core on the first processor does not have the requested cache data, the first processor is configured to: broadcast a request to other processors on the first board to determine whether one of the other processors is an intervening master IM of the requested cache data; receive a partial response PRESP from the other processors on the first board; based on the PRESP, determine which other processors of the other processors are still involved in the request, wherein a first other processor of the other processors indicates in its PRESP that it is the IM, and a second other processor of the other processors indicates in its PRESP that it is not the IM but is still involved in the request; and send a combined response CRESP only to the other processors that indicate in their PRESP that they are still involved in the request. One advantage is that this cache coherence method can tolerate variable latency when cache operations are transferred from intra-processor communication to inter-processor communication between processors on the same board. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Embodiments of the present invention will now be described, by way of example only, with reference to the accompanying drawings, in which:

[0012] Figure 1A cache coherent computing system with a distributed topology is shown according to one embodiment.

[0013] Figure 2 A processor chip according to one embodiment is shown.

[0014] Figure 3 A communication system between on-board processors according to one embodiment is shown.

[0015] Figure 4 Communication between boards in a cache coherent computing system is shown according to one embodiment.

[0016] Figure 5 is a flow chart for identifying cached data in a distributed topology according to one embodiment.

[0017] 6A to 6F Identifying cache data in a board containing multiple processors according to one embodiment is shown.

[0018] 7A to 7H Identifying cache data in multiple boards according to one embodiment is shown. DETAILED DESCRIPTION

[0019] Embodiments herein describe a consistency protocol for a distributed computing topology that allows large-scale pauses on various interfaces (e.g., interfaces between different motherboards containing processor chips). In one embodiment, the computing topology includes multiple boards (e.g., multiple motherboards, also referred to as "drawers"), each board containing multiple processors (e.g., multiple processor chips). When a specific core on a processor wants to access data that is not currently stored in its cache, the core can first initiate a request to search for cache lines in the caches of other cores on the same processor. If the cache line is not found, the processor broadcasts the request to other processors on the same board. For example, the processors in the board can be connected via a fully connected structure. If the processors on the same board do not have data, the processor can broadcast the request to other boards in the system. The processors in these boards can then search their caches to identify the data.

[0020] One problem with performing coherence operations in a multi-scope system is that the latency involved in performing cache operations (e.g., snooping, cache responses, etc.) can vary significantly at each level or scope (e.g., searching a cache in the same processor versus searching a cache in processors on the same board versus searching a cache in processors on different boards). The multi-scope coherence protocol described herein can ensure that data is shared quickly at lower scopes (e.g., when searching a cache in the same processor) and can tolerate increased latency as requests progress to higher scopes (e.g., searching a cache in processors on the same board or searching a cache in processors on different boards). The multi-scope coherence protocol can tolerate variable latency as cache operations move to higher scopes. Advantageously, the multi-scope coherence protocol can be used in different system topologies and still maintain data integrity.

[0021] Figure 1 A cache coherent computing system 100 having a distributed topology is shown according to one embodiment. In this example, the topology of the system 100 is distributed because processors 110 (e.g., central processing units (CPUs)) are distributed across different boards 105 (e.g., different motherboards or drawers) in the computing system. The communication technology used to communicate between processors 110 on the same board 105 may be different than the communication technology used to communicate between processors 110 on different boards 105. For example, passing messages or performing cache operations between processors 110 on different boards 105 may generally incur greater latency than performing cache operations between processors 110 on the same board 105.

[0022] The computing system 100 may be any computing system, such as a server, a blade server, a rack server, a tower server, a desktop computer, a laptop computer, etc. In one embodiment, the computing system 100 includes a chassis, and the chassis includes Figure 1 In one embodiment, the chassis may be mountable so that the computing system 100 may be mounted in a rack or tower server.

[0023] In this example, computing system 100 includes two boards: boards 105A and 105B, each board including multiple processors 110. However, computing system 100 can have any number of boards (e.g., three, four, eight, etc.), each board including any number of processors (e.g., two, four, six, eight, sixteen, etc.). Each processor 110 includes one or more cores 115 (e.g., one, four, eight, etc.) and a cache 120. Some caches can be integrated into the core (e.g., level 1 (L1) cache), while other caches 120 are located outside the core 115 (e.g., level 2 (L2) and level 3 (L3) cache). The embodiments herein are not limited to any particular number of cache levels or cache hierarchies. Although not shown, computing system 100 may also include a main memory (e.g., random access memory), which may be separate from board 105 or disposed on the board. Computing system 100 may also include a long-term storage device, such as a hard disk.

[0024] In a shared memory multiprocessor system (such as computing system 100, where each processor 110 has a separate cache 120), multiple copies of shared data can be held, for example, one copy in main memory (not shown) and one copy in the local cache of each processor that requests it. However, maintaining data consistency in a highly distributed topology that uses high-speed interfaces and long recovery mechanisms is difficult and can benefit from the techniques discussed herein. For example, to ensure data integrity of a cache line as it is moved, the old owner of the cache line should maintain protection of the cache line until the new owner begins protecting the data and all other operations in progress on the same cache line are rejected (e.g., when two cores 115 request the same cache line simultaneously).

[0025] Embodiments herein establish several consistency protocol rules to address computing systems with distributed topologies. In this example, the topology of computing system 100 has three different scopes: (1) processor scope, (2) board scope, and (3) inter-board scope (or global scope). The multi-scope consistency protocol described herein addresses the heterogeneous nature of system topology in system 100, which has limited resources and can quickly and fairly share lines when communicating between processors on the same board 105 and different boards 105. Embodiments herein introduce a combined response (CRESP) that is not dependent on another operation to complete. That is, no other operation can stop the generation of CRESP. In addition, the multi-scope consistency protocol can tolerate time delays (e.g., time independence) by using a consistency protocol (also known as a structure (FBC) protocol) that handles any delays caused by cache operations sent between processors 110 on the same board 105 and on different boards 105. The consistency protocol is responsible for handling off-chip broadcasts and all related messages and maintaining the consistency aspects of data consistency. In addition, an intervention master (IM) can complete its tasks without relying on other operations from the same or lower scope. That is, there is nothing preventing the IM from completing the handover of the requested cache line to the new owner. As used herein, the IM is the highest consistency point for a given scope and contains the most recent cached copy of the cache line. The IM is typically a cache on processor 110 (or core 115 within processor 110), but can be memory (e.g., cache 120) if no core currently owns the cache line. Additionally, in one embodiment, the multi-scope coherency protocol has request observability, where no participant outside the current scope can prevent a request from being broadcast within that scope.

[0026] Figure 2 A processor 110 is shown according to one embodiment. In one embodiment, the processor 110 is an integrated circuit (eg, a single chip). Figure 2 The processor 110 is Figure 1 An example implementation of processor 110 is shown in .

[0027] Processor 110 includes eight cores (cores 0-7) and eight caches (cache 0-7). In one embodiment, the caches are owned by the corresponding cores - for example, cache 0 is owned by core 0, cache 1 is owned by core 1, and so on. As shown, a core has a direct connection to a cache it owns, but may not have a direct connection to a cache it does not own. The caches may be L1, L2, or L3 caches.

[0028] Processor 110 also includes X-FBC 205 and M-FBC 210. Figure 2In FIG. 1 , an “X” mark indicates a circuit responsible for communicating with a processor located in a different socket on the same circuit board, while an “M” mark indicates a circuit responsible for communicating with a processor located in the same socket on the same circuit board as processor 110. These sockets and connections are Figure 3 However, the embodiments herein may also be applicable to computing systems where processors are not arranged (or packaged) in different sockets on a circuit board.

[0029] FBC 205, 210 is a fabric logic that manages consistency between multiple processors. That is, processor 110 uses FBC 205, 210 when sending and receiving cache operations with other processors. For example, if processor 110 sends a request for a cache line to a processor in the same socket, it communicates with that processor using M-FBC 210. However, if processor 110 sends a request for a cache line to a processor in a different socket, it communicates with that processor using X-FBC 205.

[0030] The processor 110 also includes an X-bus input / output (IO) 215 and an M-bus IO 220. Like the FBCs 205, 210, the X-bus IO 215 is an IO interface for communicating with processors on different sockets, while the M-bus IO 220 is an IO interface for communicating with processors on the same socket. Likewise, other topologies may have different arrangements that do not package the processors into sockets. The processor 110 also includes an A-bus IO 240, which the FBC unit uses to connect to other boards.

[0031] In this example, processor 110 uses rings 225 and 230 to communicate between caches. Figure 2 Two rings are used in 200 to illustrate that different types of data can be sent between caches. For example, outer ring 225 can represent the path used by coherence messages (e.g., snoop and response messages), while inner ring 230 can represent the data bus between caches.

[0032] Assume that core 2 wants to read and process data on a particular cache line. Core 2 first searches its internal cache (if any), and if the cache line is not found (i.e., a cache miss), it queries its own cache - i.e., cache 2. If cache 2 does not own the cache line, core 2 sends a request to other cores to search for the cache line in their caches. The request can travel along ring 225 to different caches. In one embodiment, the request can be sent unidirectionally or bidirectionally at the same time. The caches / cores can then respond to the request, indicating whether they own the requested cache line.

[0033] Although Figure 2The use of a ring topology to perform cache operations in processor 110 is shown, but this is only an example. The embodiments herein are not limited to any particular technique for performing cache operations in a single chip / processor, but can be used with any other type of technique for performing cache operations in a single processor, such as a mesh network, a cross bar, etc. Thus, rings 225, 230 are just one suitable technique for propagating cache operations in a chip.

[0034] Figure 3 1 shows a communication system between processors 110 on board 105 according to one embodiment. For example, Figure 1 The computing system 100 in may have Figure 3 Multiple instances of plate 105 are shown in FIG.

[0035] In this example, the board 105 includes four sockets 305A-D, each of which contains two processors 110. The sockets 305 can then be coupled to a common substrate (e.g., a printed circuit board). For example, the processors 110 in the same socket 305 can be packaged together and can be arranged on a common substrate (e.g., an interposer). However, embodiments herein are not limited to boards 105 with sockets, and can include embodiments in which the processors 110 are directly attached to the board 105.

[0036] The board 105 includes a fully connected fabric 310 for communicatively coupling together the processors 110 located on different sockets 305. In this example, the processor in each socket has a direct connection with each processor in a different socket in the fully connected fabric 310. That is, processor CP-0 has a direct connection with processors CP-2 / CP-3 in socket 305B, processors CP-4 / CP-5 in socket 305C, and processors CP-6 / CP-7 in socket 305D. Return to Reference Figure 2 , six connections to these six processors can be established using the six X-bus IOs 215 on top of the processor 110.

[0037] Board 105 also includes inter-socket connections 315 for connecting processors in the same socket 305. That is, processor CP-0 communicates with processor CP-1 using inter-socket connection 315, processor CP-2 communicates with processor CP-3 using inter-socket connection 315, and so on. Figure 2 , one of the M-bus IOs 220 at the bottom of processor 110 may be used to establish an inter-socket connection with a processor in the same socket.

[0038] The fully connected fabric 310 and the inter-socket connections 315 guarantee that each processor 110 has a direct communication link with each other processor 110 on the same board 105. That is, there are guaranteed resources available for one processor to communicate with all other processors in the same board. In this example, a "direct communication link" means that a processor 110 can communicate with another processor on the same board 105 without using another processor as an intermediary (e.g., in the case of a spine-leaf topology). In addition, the fully connected fabric 310 can be time-independent because cache coherence can be maintained when there are additional delays or transmission errors.

[0039] Figure 4 1 shows communication between boards in a cache coherent computing system according to one embodiment. Figure 3 As shown, each board 105 (board 0-3) includes four slots and eight processors. These lines show the inter-board connections between the boards. In this example, each board has a slot (i.e., two processors), which are assigned to communicate with different boards, and these boards are called fork processors. Taking board 0 as an example, processor CP-2 / CP-3 is connected to a pair of processors in board 2, processor CP-4 / CP-5 is connected to a pair of processors in board 1, and processor CP-6 / CP-7 is connected to a pair of processors in board 3. When a processor wants to send a cache message to a different board, it can use a fork processor to reach the board. For example, if processor CP-0 in board 0 wants to send a response to a processor in board 3, processor CP-0 first sends the response to one of processors CP-6 / CP-7 in board 0, and then the processor forwards the response to processor CP-6 / CP-7 in board 3.

[0040] In one embodiment, the processor assigned to communicate with other boards can have a ticket manager to manage cache operations flowing between boards. The ticket manager can use the queue to determine which cache operations to send next.

[0041] and Figure 3 processors are directly connected to each other processor, unlike Figure 4 In a , a processor on one board may not have a direct communication link with a processor on another board. That is, a processor may have to rely on one or two processors (e.g., a fork processor) to communicate a cache request or response to a processor on a different board. Figure 4This can be called a pass-through topology, where some processors act as intermediaries between other processors on different boards. For example, CP-0 in board 0 relies on processors CP-2 / CP-3 to reach board 2, relies on processors CP-4 / CP-5 to reach board 1, and relies on processors CP-6 / CP-7 to reach board 3.

[0042] Figure 5 is a flow chart of a method 500 for identifying cached data in a distributed topology according to one embodiment. At block 505, a requester (e.g., a core on a processor) determines whether a cache on the processor has the requested data (e.g., a particular cache line). For example, the requester may search an internal L1 cache (if any) and an assigned L2 cache (e.g., Figure 2 0-7) in the processor to determine whether those caches have the latest copy of the requested data. For example, the requesting core may use a communication network in the processor (e.g., Figure 2 The method 500 may be used to determine whether the caches allocated to other cores in the processor contain the requested data, using a ring topology or mesh network in the processor. If so, the method 500 proceeds to block 510, where the requester retrieves the latest version from the cache in the processor.

[0043] However, if any cache in the processor does not have the latest version of the requested data, then method 500 proceeds to block 515, where the processor broadcasts the request to processors on the same board. For example, the processor may use Figure 3 The fully connected fabric 310 and inter-socket connections 315 in the board 105 broadcast the request to processors in the same socket and different sockets. Figure 3 The connections shown in are just one example of connecting processors together that are on the same board.

[0044] At block 520, the processors determine whether they are the IM for the requested data. That is, after receiving the broadcast, each processor determines whether it is the IM for the data. That is, the processors determine whether they are the highest point of coherence and contain the most recent cached copy of the requested data.

[0045] The following Figures 6A-6F Details of searching processors in the same board to determine if one of the processors in the IM has the requested data are discussed.

[0046] If one of the processors on the same board is an IM, the method proceeds to block 520 where the IM is changed to the requesting processor. Doing so requires the requesting processor to protect the cache line, as discussed in more detail below. Additionally, at block 510, the requested data may be copied from the cache in the processor that was previously the IM to the requesting processor.

[0047] If no processor on the board is the IM for the requested data, the method proceeds to block 530 where the requesting processor broadcasts the request to processors on other boards to determine if those processors are the IM for the data (or if the data does not belong to any core but rather to a cache in a processor).

[0048] The following Figures 7A-7H Details of searching processors on different boards to determine if one of those processors is the IM for the requested data are discussed.

[0049] Method 500 assumes that one of the processors on the different board is the IM for the requested data. At block 520, the IM changes to the requesting processor, and at block 510, the requested data is copied from the cache in the processor that was previously the IM to the requesting processor. However, if no processor in the computing system is the IM for the requested data, the computing system may retrieve the data from main memory or a long-term storage device (e.g., a hard disk).

[0050] Figures 6A-6F 1 shows identifying cache data in a board 105 containing a plurality of processors 110 according to one embodiment. For ease of explanation, the board 105 has Figure 3 and Figure 4 The same socket and processor arrangement as shown in , except that the eight processors are labeled CP_A through CP_H. Figures 6A-6F Steps are shown that may be performed at blocks 515 and 520 to identify whether a processor on the board (eg, at a board-wide or board-level) is the IM for the requested data.

[0051] In this example, processor CP_E is the home processor that is requesting data (e.g., a cache line). That is, processor CP_E wants to become the IM of the data in order to ensure that it has the latest copy of the data. Fig. 6A A processor CP_E is shown, which broadcasts a request to all other processors in the same board 105. For example, processor CP_E may use Figure 3 The fully connected structure 310 and inter-socket connections 315 in the processor are used to broadcast the request to each processor CP_A, CP_B, CP_C, CP_D, CP_F, CP_G and CP_H.

[0052] Figure 6BA processor CP_E is shown which receives partial responses (PRESPs) from each processor. These PRESPs may be received at different times. As described above, the board scope may be time tolerant so that PRESPs from the processors may be received at different times depending on, for example, the available bandwidth of the connection and the usage of the processors. In one embodiment, the processor CP_E waits until it receives all PRESPs before continuing.

[0053] In one embodiment, PRESP indicates whether the processor is an IM, or whether the processor is responsible for identifying the IM if the IM is on a different board. In this case, processor CP_B is the IM for the requested data (labeled as board IM 1 (BIM1)). Therefore, its PRESP indicates to processor CP_E that it is the IM for the data. In addition, processors CP_A, CP-C, and CP_G are Figures 6A-6F , which indicates that these processors are assigned to communicate with other boards. That is, if IM is not set on board 105, processors CP_A, CP-C and CP_G will be responsible for broadcasting requests to other boards in the computing system, which will be in Figures 7A-7H For example, processor CP_A may be responsible for exchanging cache operation messages with board 1, processor CP_C may be responsible for exchanging cache operation messages with board 2, and processor CP_G may be responsible for exchanging cache operation messages with board 3.

[0054] Because in Figure 6B At the time shown, the fork processors do not yet know whether the IM has been set up on board 105 when sending their PRESPS, so their PRESPS indicate that they will remain on standby in case processor CP_E has to broadcast data requests to other boards. That is, the fork processors will not indicate to processor CP_E in their PRESPs that their part in the cache operation is complete.

[0055] In contrast, processors CP_D, CP_F, and CP_H are neither IMs requesting data nor fork processors for communicating with other boards. Therefore, the PRESPs sent by processors CP_D, CP_F, and CP_H indicate to processor CP_E that their part in the cache operation is complete and they are no longer involved in the request.

[0056] Figure 6CProcessor CP_E is shown sending a combined response (CRESP) after receiving all PRESPs (i.e., partial responses from other processors on board 105). As shown, CRESP is not broadcast to all processors on board 105, but only to processors that indicate they are still part of the cache operation (i.e., processors that are IMs requesting data, home memory controllers for data, processors with active outstanding requests for the same cache line, or fork processors used to communicate with other boards in the computing system). Therefore, processors CP_D, CP_F, and CP_H do not receive CRESP.

[0057] CRESP notifies the IM that the fork processors CP_A, CP-C, and CP_G have found the data, so they do not need to send messages to other boards. In other words, the fork processor no longer needs to complete the cache operation. Instead, CRESP notifies the IM (i.e., processor CP_B) that it still needs it to complete the operation.

[0058] Fig.6D The fork processors CP_A, CP-C and CP_G are shown sending reset responses (RRESP) to the processor CP_E. These RRESPs inform the processor CP_E that the processors CP_A, CP-C and CP_G have completed the cache operation.

[0059] Fig. 6E Processor CP_E is shown sending an End of Coherence Response (ERESP) to processor CP_B. ERESP tells processor CP_B that it is safe to stop protecting the requested data and that the home processor CP_B is now responsible for protecting the requested data. In other words, ERESP indicates that the old IM (i.e., processor CP_B) can safely reset the coherence protection. The old IM can stop rejecting new requests for data. ERESP indicates that the new IM (i.e., processor CP_E) is now able to process further requests for data. In addition, ERESP notifies the old IM that the new IM is confident that all potential observers are satisfied.

[0060] In response to receiving ERESP, Fig. 6F In the example above, processor CP_B sends RRESP to processor CP_E indicating that it has completed the cache operation. In addition, ERESP indicates to processor CP_E that any other requestor of the same data (e.g., another processor on the same board or on a different board) should have observed the transfer and that the data is being sent. This therefore prevents conflicts from occurring where processors requesting the same data in parallel may not know which processor is currently the IM of the data.

[0061] Figures 7A-7H Identifying cache data in multiple boards according to one embodiment is shown. For ease of explanation, Figures 7A-7H Each of the plates 105A-D in the Figures 6A-6F The same socket and processor arrangement as in board 105. Figures 7A-7H Steps are shown that may be performed at block 530 to identify whether a processor on a different board (eg, an inter-board range or level) is the IM for the requested data.

[0062] Figures 7A-7H Assume that blocks 505-525 of method 500 have been executed and the requesting processor has determined that (i) it does not have a recent cached copy of the requested data and (ii) the IM for the data is not on the same board. Therefore, the requesting processor broadcasts a request to identify the IM to the other boards in the system.

[0063] exist Fig. 7A In, with Fig. 6A Similarly, processor CP_E is the requesting processor and is searching for the IM for the requested data. Fig. 6A Differently, the IM of the data is not on the same board (ie board 105A), but on a different board. Specifically, the IM of the requested data (labeled as global IM (GIM1)) is processor CP_F on board 105B.

[0064] Processor CP_E has determined that none of the processors on board 105A are IMs for the requested data, and therefore has instructed fork processors CP_A, CP_C, and CP_G on board 105A to broadcast the IM request to other boards 105B-D, such as Fig. 7A As shown. The request sent to board 105B is received by processor CP_G (e.g., a branch processor), the request sent to board 105C is received by processor CP_C (e.g., a branch processor), and the request sent to board 105D is received by processor CP_A (e.g., a branch processor). Then, each branch processor on the three boards 105B-D broadcasts the IM request to the processors on the same board. That is, each branch processor is like Fig. 6A Broadcast the request as shown.

[0065] Figure 7B The branch processors on each of the boards 105B-D are shown collecting partial responses PRESP from each processor on the same board. Figure 6B As described in, other processors can tell the processor broadcasting the request (eg, a branch processor) whether they are the IM of the data.

[0066] Figure 7BThe branch processor is shown forwarding PRESP to fork processors CP_A, CP_C, and CP_G on board 105A, which in turn forward PRESP to requesting processor CP_E. Based on the response, requesting processor CP_E can determine that processor CP_F on board 105B is the IM for the requested data.

[0067] Figure 7C As shown, each branch processor on board 105B-D sends a CRESP based on the PRESP received from other processors in the same board. These board-wide CRESPs can contain the same Figure 6C Same information discussed in CRESP.

[0068] Fig.7D CRESP of global scope (or CRESP of inter-board scope) is shown. Global CRESP originates from request processor CP_E on board 105A. CRESP is sent only to fork processors in board 105A, which in turn forward CRESP to corresponding boards. As shown, branch processors in boards 105B-105D only forward CRESP to processors that are still part of the operation. That is, since only processor CP_F in board 105B is the IM of data, the PRESP sent by other processors in board 105B indicates that they no longer need to participate in cache operations because they are not IM. Therefore, only processor CP_F and branch processors in board 105B receive CRESP. In board 105C, CRESP is forwarded from branch processor CP_C to processor CP_F that only serves as LFAR-X. LFAR-X indicates that CP_F is making a request to the same line (e.g., LFAR-X indicates "comparison" or "contention detected"). In board 105D, CRESP is forwarded from branch processor CP_A to processor CP_F that only serves as the memory home page of the requested data. If another processor has not been assigned an IM, processor CP_F is still part of the cache operation because the memory home of the requested data may be the IM of the data. In this case, since processor CP_F on board 105B is the IM, CRESP notifies processor CP_F on board 105D that an IM has been found and it should no longer participate in the cache operation. One advantage of board-wide CRESP and global-scope CRESP is that they involve only board-wide and global-scope processors that are still participating in the cache operation.

[0069] Since CP_F on board 105C requested the same row that was in progress, the controller processing the request from board 105A waits for board 105C to receive the rejection CRESP before sending RRESP back to the CP_E chip on 105A.

[0070] Comparisons are reported on PRESP, and the Home Processor waits for RRESPs from all chips that reported comparisons on PRESP before sending ERESPs to the GIM1 processor. Fig. 7E The home processor CP_E on board 105 is shown receiving RRESP from a non-IM processor (e.g., CP_F on board 105C), indicating the end of the comparison operation. Figures 7A-7E In , IM is moving from CP_F on board 105B to CP_E on board 105A. LFAR_X on CP_F on board 105C also wants to access this cache line, and for consistency the system ensures that it observes the IM move. This is accomplished by preventing LFAR_X on CP_E board 105A from completing until we know that LFAR_X on CP_F on board 105C has been rejected by either CP_F board 105B or CP_E board 105E. Figure 7B 105C, the CP_F on board 105C will report the comparison and the state machine will load the dependencies on LFAR_X. Once LFAR_X on CP_F on board 105C is rejected, the dependencies are resolved and the state machine will send RRESPs. When LFAR_X on CF_E on board 105A receives all RRESPs from non-IM processors, it knows that all dependencies encountered in snooping have been resolved and all other requests have observed IM transmissions.

[0071] Figure 7F The IM requesting the data (i.e., processor CP_F on board 105B) and the memory home of the data (i.e., processor CP_F on board 105D) are shown sending a RRESP to the requesting processor. In one embodiment, the requesting processor waits until it receives RRESPs from all non-IM processors indicating in their PRESPs that they have outstanding requests for the same row before sending an ERESP to the current IM.

[0072] Figure 7G The request processor CP_E is shown sending a consistency end ERESP to the IM. ERESP tells the processor CP_F on board 105B that it is safe to stop protecting the requested data and that the home processor CP_E is now responsible for protecting the requested data.

[0073] In response to receiving ERESP, Figure 7H, processor CP_F on board 105B sends RRESP to processor CP_E on board 105A indicating that it has completed the cache operation. In addition, ERESP indicates to processor CP_E that any other requester of the same data (e.g., another processor on the board or on a different board) should have observed the transfer and that the data is being sent. This prevents conflicts from occurring, where processors requesting the same data in parallel may not know which processor is currently the IM of the data.

[0074] also, Figure 7H The memory home of the data (i.e., processor CP_F on board 105D) and the global IM of the data (i.e., processor CP_F on board 105B) are shown sending RRESP to the home processor CP_E on board 105. All processors participating in the operation have now sent RRESP to the home processor. This is used to determine when a remote resource is available for use by another operation.

[0075] For performance reasons, the system can begin speculatively accessing memory before knowing whether the memory data will be needed. Fig. 7A During the snoop time in , the system loads the state machine and begins memory access. The global CRESP indicates to the memory master (i.e., processor CP_F on board 105D) that the memory data is not needed, but the state machine on the memory chip should still send a message to indicate when it is done.

[0076] Wait for any ongoing operations to get CRESP on all processor chips reporting the compare before returning RRESP to the home chip, and have the home chip delay ERESP to the global IM processor until the home chip has received all RRESPs from all chips reporting the compare, ensuring that all operations active on the same cache line observe the global IM move to the new processor.

[0077] In one embodiment, there is no other operation that can prevent the IM from requesting original content from the IM, getting a hit IM, and evicting / evicting the IM. Whenever one or more requests attempt to get a cache line, at least one request will win. In one embodiment, the protocol has a built-in mechanism to ensure that additional resources required to complete the operation are acquired. If resources are held by non-IM requesters or resource queues are formed, there is a mechanism to cancel the non-IM requesters so that they can free up resources for the IM requesters. In addition, the CRESP can contain information if the requester has IM rights for the current scope or a higher scope.

[0078] In one embodiment, no other operations can prevent a request from advancing to a higher broadcast scope. That is, the protocol allows all new requesters to be broadcast or observed by other ongoing operations. Before sending an ERESP within a scope, a request can wait until RRESPs are received from all targets in the scope that have outstanding requests for the same cache line. In addition, the requesting processor can honor any address comparisons in the request FBC itself. This is accomplished by setting up scope-based pending operations and system-based pending operations in the controller, and setting operations that need to wait for each other when appropriate to ensure system consistency. In addition, the requesting processor may have to receive CRESPs broadcast by a higher scope if necessary.

[0079] The description of various embodiments of the present invention is for illustrative purposes only and is not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The terms used herein are selected to best explain the principles of the embodiments, practical applications, or technical improvements over technologies found on the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

[0080] The embodiments presented in the present disclosure are mentioned above. However, the scope of the present disclosure is not limited to the specifically described embodiments. On the contrary, any combination of features and elements, whether or not related to different embodiments, may be considered to implement and practice the contemplated embodiments. In addition, although the embodiments disclosed herein may achieve advantages over other possible solutions or over the prior art, whether a given embodiment achieves a particular advantage does not limit the scope of the present disclosure. Therefore, the aspects, features, embodiments, and advantages discussed herein are merely illustrative and are not considered elements or limitations of the appended claims unless expressly stated in the claims. Similarly, references to the "present invention" should not be interpreted as a generalization of any inventive subject matter disclosed herein, and should not be considered elements or limitations of the appended claims unless expressly stated in the claims.

[0081] Aspects of the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment (including firmware, resident software, micro-code, etc.), or an embodiment combining software and hardware aspects, which may generally be referred to herein as a "circuit", "module" or "system".

[0082] The present invention may be a system, method and / or computer program product at any possible level of technical detail integration. The computer program product may include a computer-readable storage medium (or media) having computer-readable program instructions thereon for causing a processor to perform aspects of the present invention.

[0083] A computer-readable storage medium may be a tangible device that can retain and store instructions for use by an instruction execution device. A computer-readable storage medium may be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the above. A non-exhaustive list of more specific examples of computer-readable storage media includes the following: a portable computer floppy disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disc (DVD), a memory stick, a floppy disk, a mechanical encoding device (e.g., a punched card or a raised structure in a groove on which instructions are recorded), and any suitable combination of the above. The computer-readable storage medium used herein should not be interpreted as a transient signal itself, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse through a fiber optic cable), or an electrical signal sent through a wire.

[0084] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to a corresponding computing / processing device, or downloaded to an external computer or external storage device via a network (e.g., the Internet, a local area network, a wide area network, and / or a wireless network). The network can include copper transmission cables, optical transmission fibers, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions to be stored in a computer-readable storage medium in the corresponding computing / processing device.

[0085] The computer-readable program instructions for performing the operation of the present invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, configuration data of an integrated circuit, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, such as Smalltalk, C++, etc., and procedural programming languages, such as "C" programming language or similar programming languages. The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, executed as an independent software package, partially on the user's computer, partially on a remote computer, or entirely on a remote computer or server. In the latter case, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, an electronic circuit (including, for example, a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA)) may be personalized electronic circuits by utilizing the state information of the computer-readable program instructions, thereby executing the computer-readable program instructions to perform aspects of the present invention.

[0086] This article describes aspects of the present invention with reference to the flow chart and / or block diagram of the method, device (system) and computer program product according to an embodiment of the present invention. It should be understood that each block in the flow chart and / or block diagram and the combination of blocks in the flow chart and / or block diagram can be implemented by computer-readable program instructions.

[0087] These computer-readable program instructions may be provided to a computer processor, or other programmable data processing device, to produce a machine, such that the instructions executed by the computer processor or other programmable data processing device create means for implementing the functions / actions specified in the flowchart and / or block diagram blocks. These computer-readable program instructions may also be stored in a computer-readable storage medium, which may instruct a computer, programmable data processing device, and / or other device to operate in a particular manner, such that the computer-readable storage medium storing the instructions includes an article of manufacture, which includes instructions for implementing various aspects of the functions / actions specified in the flowchart and / or block diagram blocks.

[0088] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device, so that a series of operational steps are performed on the computer, other programmable apparatus, or other device to produce a computer-implemented process, so that the instructions executed on the computer, other programmable apparatus, or other device implement the functions / actions specified in the flowchart and / or block diagram blocks.

[0089] Flowchart and block diagram in the figure show the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention.In this respect, each block in the flow chart or block diagram can represent a part of a module, segment or instruction, which includes one or more executable instructions for realizing a specified logical function.In some alternative implementations, the function pointed out in the block may not occur in the order pointed out in the figure.For example, two blocks shown continuously can actually be completed as a step, executed simultaneously, substantially simultaneously, executed in a way that part or all of the time overlaps, or sometimes can be executed in reverse order, depending on the function involved.It should also be noted that each block in the block diagram and / or flow chart and the combination of the blocks in the block diagram and / or flow chart can be realized by a dedicated hardware system that performs a specified function or action or performs a combination of dedicated hardware and computer instructions.

[0090] While the foregoing is directed to embodiments of the present invention, other and further embodiments of the invention may be devised without departing from the basic scope thereof, and the scope of the invention is determined by the claims that follow.

Claims

1. A method comprising: Providing a cache coherent computing system including a plurality of boards, each board including a plurality of processors; requesting cache data of a first core in a first processor on a first board among the plurality of boards; When it is determined that another core on the first processor does not have the requested cache data, broadcast a request to other processors on the first board to determine whether one of the other processors is an intervening master IM of the requested cache data; as well as Upon determining that none of the other processors on the first board is the IM, a request is broadcasted to other boards in the plurality of boards to determine whether one of the processors in the other boards is the IM.

2. The method according to claim 1, wherein: Broadcasting the request to the other processors on the first board further comprises: receiving, at the first processor, a partial response PRESP from the other processors on the first board; determining, based on the PRESP, which other of the other processors still participate in the request and other operations initiated by the other processors for the same cache data, wherein at least one of the other processors indicates in its PRESP that it does not participate in the request; and The combined response CRESP is sent only to the other processors still participating in the request.

3. The method according to claim 2, wherein: The other processors still participating in the request include a fork processor in the first board, the fork processor being assigned to communicate with a corresponding board in the plurality of boards.

4. The method according to claim 3, wherein: At least one of the other processors not participating in the request is a processor on the first board, is not a fork processor, has no ongoing operation for the same cache data, and is not the IM of the requested cache data.

5. The method according to claim 1, wherein: Broadcasting the request to the other boards in the plurality of boards further comprises: receiving, at a corresponding branch processor of each of the other boards, the request; broadcasting the request from the corresponding branch processor to the other processors in the same board; receiving, at the corresponding branch processor, a PRESP from the other processors in the same board, the PRESP reporting other operations being performed on the same cache data; forwarding the PRESP from the other processors in the other boards to the first processor on the first board; sending a board-wide CRESP from the corresponding branch processor to only the other processors in the same board indicating that they are still participating in the request based on the PRESP; and Based on the PRESP, a global scope CRESP is sent from the first board to the other processors in the other boards indicating that they are still participating in the request.

6. The method according to claim 5, wherein: A first one of the other processors indicating that it is still participating in the request is the IM of the requested cache data or has an ongoing operation for the same cache data, and a second one of the other processors indicating that it is still participating in the request is not the IM of the requested cache data.

7. The method according to claim 6, further comprising: receiving, at the first processor, a reset response RRESP from the second other processor of the other processors that is not the IM of the requested cache data; as well as After receiving the RRESP, a coherence end response ERESP is sent from the first processor to the first other processor of the other processors that is the IM of the requested cache data.

8. The method according to claim 7, wherein: The ERESP is sent from the first processor to the first of the other processors only after the first processor receives a corresponding RRESP from each of the other processors indicating that it is still involved in the request and that the IM is not the requested cache data or that another operation is ongoing for the same cache data.

9. The method according to claim 7, further comprising: receiving a RRESP from said first one of said other processors, wherein said first processor becomes said IM of the requested cache data; and The RRESP is received from the processor among the other processors that is the memory home of the requested cache data and all remaining processors participating in the request that have no other operations ongoing for the same cache data.

10. A cache coherent computing system, comprising: multiple boards, each board including multiple processors, wherein a first core in a first processor on a first board among the plurality of boards is configured to request cache data that is not stored in a cache allocated to the first core; Upon determining that another core on the first processor does not have the requested cache data, the first processor is configured to broadcast a request to other processors on the first board to determine whether one of the other processors is an intervening master IM for the requested cache data; and Upon determining that none of the other processors on the first board is the IM, the first processor is configured to broadcast a request to other boards of the plurality of boards to determine whether one of the processors in the other boards is the IM.

11. The cache coherent computing system according to claim 10, wherein: Broadcasting the request to the other processors on the first board further comprises: receiving, at the first processor, a partial response PRESP from the other processors on the first board; determining, based on the PRESP, which other of the other processors still participate in the request and other operations initiated by the other processors for the same cache data, wherein at least one of the other processors indicates in its PRESP that it does not participate in the request; and The combined response CRESP is sent only to the other processors still participating in the request.

12. The cache coherent computing system according to claim 11, wherein: The other processors still participating in the request include a fork processor in the first board, the fork processor being assigned to communicate with a corresponding board in the plurality of boards, At least one of the other processors not participating in the request is a processor on the first board, is not a fork processor, has no ongoing operation for the same cache data, and is not the IM of the requested cache data.

13. The cache coherent computing system according to claim 10, wherein: Broadcasting the request to the other boards in the plurality of boards further comprises: receiving, at a corresponding branch processor at each of the other boards, the request; broadcasting the request from the corresponding branch processor to the other processors in the same board; receiving, at the corresponding branch processor, a PRESP from the other processors in the same board, the PRESP reporting other operations being performed on the same cache data; forwarding the PRESP from the other processors in the other boards to the first processor on the first board; sending, based on the PRESP, a board-wide CRESP from the corresponding branch processor only to the other processors in the same board indicating that they are still participating in the request; and Based on the PRESP, a global scope CRESP is sent from the first board to the other processors in the other boards indicating that they are still participating in the request.

14. The cache coherent computing system according to claim 13, wherein: A first one of the other processors indicating that it is still participating in the request is the IM of the requested cache data or has an ongoing operation for the same cache data, and a second one of the other processors indicating that it is still participating in the request is not the IM of the requested cache data.

15. The cache coherent computing system of claim 14, wherein: The first processor is configured to: receiving, at the first processor, a reset response RRESP from the second other processor of the other processors that is not the IM of the requested cache data; as well as After receiving the RRESP, a coherence end response ERESP is sent from the first processor to the first other processor of the other processors that is the IM of the requested cache data.

16. A computer program product for maintaining a cache coherent computing system, the cache coherent computing system comprising a plurality of boards, each board comprising a plurality of processors, the computer program product comprising: A computer readable storage medium having computer readable program code embodied therein, the computer readable program code being executable by one or more computer processors to perform operations comprising: requesting cache data of a first core in a first processor on a first board among the plurality of boards; When it is determined that another core on the first processor does not have the requested cache data, broadcast a request to other processors on the first board to determine whether one of the other processors is an intervening master IM of the requested cache data; and Upon determining that none of the other processors on the first board is the IM, a request is broadcasted to other boards in the plurality of boards to determine whether one of the processors in the other boards is the IM.

17. A cache coherent computing system, comprising: multiple boards, each board including multiple processors, wherein a first core in a first processor on a first board among the plurality of boards is configured to request cache data that is not stored in a cache allocated to the first core; Upon determining that another core on the first processor does not have the requested cache data, the first processor is configured to: broadcasting a request to other processors on the first board to determine whether one of the other processors is an intervening master IM of the requested cache data; receiving a partial response PRESP from said other processor on said first board; determining which other processors of the other processors are still participating in the request based on the PRESP, wherein a first other processor of the other processors indicates in its PRESP that it is the IM and a second other processor of the other processors indicates in its PRESP that it is not the IM but is still participating in the request; and A combined response CRESP is sent only to those other processors which have indicated in their PRESP that they are still involved in the request.

18. The cache coherent computing system of claim 17, wherein: The second one of the other processors is a fork processor in the first board, the fork processor being assigned to communicate with a second one of the plurality of boards on behalf of the processor on the first board.

19. The cache coherent computing system of claim 18, wherein: The second other processor among the other processors is configured to: after receiving the CRESP from the first processor, send a reset response RRESP to the first processor, wherein the CRESP indicates to the second other processor among the other processors that the IM has been found and that the second other processor among the other processors does not need to communicate with the second board.

20. The cache coherent computing system of claim 19, wherein: The first processor is configured to send a consistency end response ERESP to the first one of the other processors after receiving the RRESP from the second one of the other processors and all processors reporting PRESP indicating that they have another operation in progress for the same cache data.

21. The cache coherent computing system of claim 20, wherein: The first of the other processors is configured to, after receiving the ERESP, send a RRESP to the first processor, wherein the first processor is a new IM for the requested cache data.