Server system, switch module, and switching method
The server system with cache-coherent switches and advanced cache coherence management addresses the limitations of next-generation interconnects by ensuring efficient cache consistency and data processing, enhancing performance and scalability for large data applications.
Patent Information
- Application Number
- PCT/KR2023/013786
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-08-22
- Filing Date
- 2023-09-14
- Publication Date
- 2024-12-05
AI Technical Summary
Next-generation interconnect technologies like CXL and CCIX face challenges in processing large data workloads due to limitations in cache coherence management, leading to increased delays and memory requirements that are difficult to meet, especially as the number of connected devices increases.
A server system and switch module with cache-coherent switches interconnected in a topology, featuring a cache coherence management unit that manages cache coherence by updating state information, adjusting sharer counts, and reconfiguring connection paths to ensure efficient data processing and cache consistency across devices.
The solution enables quick and efficient switching operations, reducing overhead and improving large data processing performance by maintaining cache consistency and expanding the number of connectable devices beyond physical link limitations.
Smart Images

Figure KR2023013786_05122024_PF_FP_ABST
Abstract
Description
Server systems, switch modules, and switching methods
[0001] The present invention relates to a server system, a switch module, and a switching method in which cache coherence is managed.
[0002] As the application of big data and artificial intelligence expands and the use of over-the-top (OTT) services becomes more commonplace, servers and data centers are faced with the challenge of rapidly processing large data workloads. Meanwhile, existing interconnect technologies, such as Peripheral Component Interconnect Express (PCIe), which only allow for simple data movement, are reaching their limits in handling user requests.
[0003] Accordingly, the development and application of next-generation interconnects such as Compute Express Link (CXL), OpenCAPI (Open Coherent Accelerator Processor Interface), CCIX (Cache Coherent Interconnect for Accelerator), and Gen-Z are actively underway. While interconnects based on CXL, OpenCAPI, and CCIX focus on connections between devices within a node, Gen-Z-based interconnects focus on connections between memory pools, CPUs, and devices between racks. In addition, while interconnects based on CXL and CCIX directly apply the physical layer of PCIe, interconnects based on OpenCAPI and Gen-Z specify their own physical layer.
[0004] While these next-generation interconnect technologies offer speed and performance improvements over PCIe, they still present limitations in processing large amounts of data. For example, cache-coherent non-uniform memory access (ccNUMA) interconnects such as CXL and CCIX support cache coherence within the system. This can significantly increase the latency required to process user requests as the number of devices connected to the system increases.
[0005] Accordingly, the memory requirements of large-scale data applications may be difficult to meet. One study found an 875-fold gap between the memory size required to run an application and the actual memory size available by 2022. Considering the rapidly increasing memory requirements of large-scale data applications, latency issues in next-generation interconnects could become even more severe as the number of connected devices increases.
[0006] The present invention is intended to solve the above-described problem, and provides a server system, a switch module, and a switching method that enable an interconnect that guarantees cache consistency between devices to perform switching operations quickly and efficiently.
[0007] A switch module according to an embodiment of the present invention for solving the above technical problem comprises a plurality of cache-coherent switches interconnected in a first topology, each of the plurality of cache-coherent switches comprising: a switch cache including data, an address, and status information; and a cache coherence management unit for managing cache coherence of the switch cache.
[0008] Each of the plurality of cache coherence switches outputs data together with a memory response to a memory request when data corresponding to a memory request received from the outside exists in the switch cache, and the memory request can be forwarded to another cache coherence switch among the plurality of cache coherence switches or to the outside.
[0009] The above cache coherence management unit can receive a cache coherent message and update the status information.
[0010] The above cache coherence management unit may include a switch agent that changes the state information to a temporary state when data of the switch cache is output to the outside.
[0011] The cache coherency management unit may include an eviction monitor that adjusts a sharer count in response to a cache coherency message received from the outside.
[0012] The cache coherency management unit may include a topology manager that generates a local message specified between the plurality of cache coherency switches connected in the first topology.
[0013] The above local message can be generated by setting the attribute value of the reserved bit of a cache coherence message received from the outside.
[0014] The cache coherency switch may further include a fabric manager that connects a plurality of the cache coherency switches into a second topology different from the first topology in response to a topology selection signal.
[0015] The fabric manager may include a control signal interface for receiving the topology selection signal; a reconfigurable routing table for reconfiguring connection path values of a plurality of cache coherence switches in response to the topology selection signal; and a reconfigurable port for reconfiguring the number of cache coherence switches connected among the plurality of cache coherence switches in response to the topology selection signal.
[0016] The above state information may indicate a modified state, an exclusive state, a shared state, an invalid state, a valid state, a temporary state depending on whether a cache hit occurred, or a temporary state depending on whether a cache hit occurred for a leaf switch among a plurality of cache coherence switches.
[0017] The plurality of cache coherency switches may be Compute Express Link (CXL) switches or Cache Coherent Interconnect for Accelerator (CCIX) switches.
[0018] A server system according to an embodiment of the present invention for solving the above technical problem comprises: a plurality of devices; a host processor; and a switch module for connecting the plurality of devices and the host processor or connecting between the plurality of devices using an interconnect protocol in which cache coherency is controlled, and for transmitting data corresponding to a memory request received from a first device among the plurality of devices to the first device and then transmitting the memory request to the host processor.
[0019] The switch module may include a plurality of cache coherence switches interconnected in a first topology, each having a switch cache and a cache coherence management unit.
[0020] The cache coherence management unit may include at least one of a switch agent that changes the state information to a temporary state when data of the switch cache is transmitted to the first device; a removal monitor that adjusts a sharer count in response to the cache coherence message; and a topology manager that generates a local message defined among the plurality of cache coherence switches connected in the first topology by setting an attribute value of a reserved bit of the cache coherence message.
[0021] Each of the plurality of cache coherency switches may further include a fabric manager, responsive to a topology selection signal, interconnecting the plurality of cache coherency switches in a second topology different from the first topology.
[0022] A switching method according to an embodiment of the present invention for solving the above technical problem includes: a step of determining whether data corresponding to the read request exists in a switch cache included in the first switch; a step of, when a cache hit occurs in the first switch, the first switch transferring the data in the switch cache to the first device and forwarding the read request to another switch of the switch module; a step of, a second switch of the switch module to which the read request is forwarded, transferring the read request to a home agent of a host processor connected via the interconnect protocol; a step of, the home agent generating a host response and a cache coherence message in response to the read request and transferring the host response and a cache coherence message to the second switch; and a step of, the first switch changing state information so that cache coherence is maintained in response to the cache coherence message transferred via the second switch.
[0023] The first switch may further include a step of omitting transmission of the host response generated by the home agent to the first device when a cache hit occurs in the first switch; and a step of converting a local message received from the second switch into the host response and transmitting the same to the first device when a cache miss occurs in the first switch.
[0024] The step of setting the first switch to a temporary state from the time of forwarding the read request until receiving the host response from the home agent may be further included; and the step of transmitting the invalid message to the first device when an invalid message is received in the temporary state may be further included.
[0025] The first switch may further include a step of setting a temporary state from the time the read request is forwarded until the host response is received from the home agent; a step of decrementing a sharer count by the first switch when a removal message is received in the temporary state; and a step of transmitting an invalid message to the first device by the first switch when the sharer count becomes a first value.
[0026] The step of changing the interconnection method in response to a topology selection signal may further be included.
[0027] According to the server system, switch module and switching method according to an embodiment of the present invention, the cache-coherent switch of the switch module includes switch cache and cache coherence management logic, thereby increasing the performance and efficiency of interconnect operation.
[0028] FIG. 1 is a drawing showing a server system according to an embodiment of the present invention.
[0029] FIG. 2 is a drawing for explaining a second topology according to an embodiment of the present invention.
[0030] FIG. 3 is a diagram illustrating a cache coherency switch according to an embodiment of the present invention.
[0031] Figure 4 is a diagram showing status information according to an embodiment of the present invention.
[0032] Figure 5 is a flowchart showing a switching method according to an embodiment of the present invention.
[0033] FIG. 6 is a diagram showing a server system operating with a switching method according to an embodiment of the present invention.
[0034] FIG. 7a and FIG. 7b are drawings for explaining the operation of a switch agent according to an embodiment of the present invention.
[0035] FIG. 8a and FIG. 8b are drawings for explaining the operation of a removal monitor according to an embodiment of the present invention.
[0036] FIGS. 9A to 9C are diagrams for explaining the operation of a topology manager according to an embodiment of the present invention.
[0037] FIG. 10 is a diagram illustrating a cache coherency switch according to an embodiment of the present invention.
[0038] FIG. 11 is a table for explaining the state transition operation of a cache coherence switch according to an embodiment of the present invention.
[0039] FIG. 12 is a diagram showing the overall switching architecture in a server system according to an embodiment of the present invention.
[0040] FIG. 13 is a diagram illustrating a scheduling algorithm of a cache coherence switch according to an embodiment of the present invention.
[0041] Hereinafter, specific embodiments of the present invention will be described with reference to the drawings. The following detailed description is provided to facilitate a comprehensive understanding of the methods, devices, and / or systems described herein. However, these are merely examples and the present invention is not limited thereto.
[0042] In describing embodiments of the present invention, detailed descriptions of known technologies related to the present invention will be omitted if they are deemed to unnecessarily obscure the gist of the invention. Furthermore, the terms described below are defined based on their functions in the present invention and may vary depending on the intent or custom of the user or operator. Therefore, their definitions should be based on the overall content of this specification.
[0043] The terminology used in the detailed description is for the purpose of describing embodiments of the present invention only and is not intended to be limiting. Unless expressly stated otherwise, singular forms include plural references. In this description, the terms "comprises" or "having" are intended to indicate certain features, numbers, steps, operations, elements, portions, or combinations thereof, and should not be construed to exclude the presence or possibility of one or more other features, numbers, steps, operations, elements, portions, or combinations thereof other than those described.
[0044] While terms like "first" and "second" may be used to describe various components, these components should not be limited by these terms. These terms may be used to distinguish one component from another. For example, without departing from the scope of the present invention, a first component could be referred to as a "second component," and similarly, a second component could also be referred to as a "first component."
[0045] Hereinafter, embodiments of the present invention will be described clearly and in detail to the extent that a person having ordinary skill in the art can easily practice the present invention.
[0046] FIG. 1 is a drawing showing a server system (100) according to an embodiment of the present invention.
[0047] Referring to FIG. 1, a server system (100) according to an embodiment of the present invention includes a plurality of devices (DEV), a host processor (HP), and a switch module (SM).
[0048] A plurality of devices (DEVs) may be accelerators that perform various functions of the server system (100). For example, the devices (DEVs) may be a GPU (Graphics Processing Unit), an FPGA (Field Programmable Gate Array)-based accelerator, a TPU (Tensor Processing Unit), a DPU (Data Processing Unit), a SmartNIC (Smart Network Interface Card), etc.
[0049] The host processor (HP) may include at least one CPU core, not shown, to perform computing operations for processing a workload, and may control access to system memory for computing operations through a memory controller. In addition, the host processor (HP) may be connected to a device (DEV) through a host input / output interface (HIO). At this time, the host processor (HP) and the device (DEV) may be connected through an interconnect that manages cache coherency. In particular, the host processor (HP) and the device (DEV) may be connected through a cache-coherent non-uniform memory access (ccNUMA) interconnect. For example, the host processor (HP) and the device (DEV) may communicate through the CXL or CCIX protocol.
[0050] The home agent (HA) performs control operations to ensure cache coherence in communication between a host processor (HP) and a device (DEV) within a server system (100) or between devices (DEV). Details on this will be described later.
[0051] At this time, the number of devices (DEV) connectable to the host input / output interface (HIO) may be limited. The switch module (SM) according to an embodiment of the present invention can increase the number of devices (DEV) connectable to the host input / output interface (HIO). That is, even if the number of physical links of the host input / output interface (HIO) is limited, the switch module (SM) is provided to expand the connectable devices (DEV) within the server system (100).
[0052] To this end, the switch module (SM) includes a plurality of cache-coherent switches (CSWs) interconnected in a first topology. The plurality of cache-coherent switches (SWs) may be CXL switches or CCIX switches.
[0053] Fig. 1 illustrates an example in which the first topology is a tree topology. Each cache coherence switch (SW) may be connected to a cache coherence switch (SW) of an upper node of the tree by one link and may be connected to a cache coherence switch (SW) of a lower node by multiple links. Although Fig. 1 illustrates that each cache coherence switch (SW) is connected to a cache coherence switch (SW) of a lower node by two links, the present invention is not limited thereto. The cache coherence switch (SW) of the uppermost node of the tree may be connected to a home agent (HA) of a host processor (HP), and the cache coherence switch (SW) of a lowermost node may be connected to a device (DEV). The cache coherence switch (SW) of the lowermost node that is directly connected to the device (DEV) may be referred to as a leaf switch, and the remaining cache coherence switches (SW) may be referred to as internal switches.
[0054] According to an embodiment of the present invention, a switch module (SM) is configured such that a plurality of cache coherence switches (SW) are connected in a tree structure, making it easy to add devices (DEV), and the basic structure of the server system (100) can be maintained even when a new device (DEV) is added.
[0055] The cache coherency switches (SWs) of the switch module (SM) according to an embodiment of the present invention may be connected in a second topology different from the first topology.
[0056] FIG. 2 is a drawing for explaining a second topology according to an embodiment of the present invention.
[0057] Referring to FIG. 2, in a switch module (SM) according to an embodiment of the present invention, a plurality of cache coherence switches (SWs) may be interconnected in a second topology different from the first topology. FIG. 2 illustrates an example in which the second topology is an all-to-all topology.
[0058] In the case of the all-to-all topology, the maximum number of cache coherency switches (SW) that must be passed for communication between the host processor (HP) and the device (DEV) or between devices (DEV) is small at 3 (Max hop count), whereas the average bandwidth is small at M / 5 because one cache coherency switch (SW) is connected by 5 links. This means that in the case of the tree topology of Fig. 1, the maximum number of cache coherency switches (SW) that must be passed for communication between the host processor (HP) and the device (DEV) or between devices (DEV) is large at 6, whereas the average bandwidth is large at M / 3. In this way, there may be a trade-off depending on how multiple cache coherency switches (SW) are connected.
[0059] Referring back to FIG. 1, a switch module (SM) according to an embodiment of the present invention connects a plurality of devices (DEV) and a host processor (HP) with a cache coherency-controlled interconnect, or connects the devices (DEV) to each other, and then forwards the memory request to the host processor (HP) after transmitting data corresponding to a memory request to the corresponding device. Furthermore, the switch module (SM) can independently handle part of the cache coherency management. To this end, a plurality of cache coherency switches (SW) included in the switch module (SM) each include a switch cache (SW) and a cache coherency management unit (CC).
[0060] Accordingly, the server system (100) according to an embodiment of the present invention can improve large-capacity data processing performance by processing memory requests from devices (DEV) more quickly and reducing the overhead required for cache consistency management even when the number of devices (DEV) included in the server system (100) increases. This will be described in detail below.
[0061] FIG. 3 is a diagram showing a cache coherence switch (SW) according to an embodiment of the present invention, FIG. 4 is a diagram showing state information according to an embodiment of the present invention, FIG. 5 is a flowchart showing a switching method (500) according to an embodiment of the present invention, and FIG. 6 is a diagram showing a server system (100) operating with a switching method (500) according to an embodiment of the present invention.
[0062] Referring to FIGS. 3 to 6, a switch cache (SC) of a cache coherency switch (SW) according to an embodiment of the present invention includes data, addresses, and state information corresponding to cache lines. For example, the state information may indicate a modified state (M) of a cache line, an exclusive state (E), a shared state (S), an invalid state (I), a valid state (V), a temporary state depending on whether a cache hit occurs, or a temporary state depending on whether a cache hit occurs for a leaf switch. At this time, the temporary state depending on whether a cache hit occurs may include a temporary state (T) in the case of a cache hit and a temporary state (IT) in the case of a cache miss, and the temporary state depending on whether a cache hit occurs for a leaf switch may include a temporary state (TL) in the case of a cache hit for a leaf switch and a temporary state (ITL) in the case of a cache miss for a leaf switch.
[0063] Hereinafter, for the convenience of explanation, the temporary state (T) in case of a cache hit, the temporary state (IT) in case of a cache miss, the temporary state (TL) in case of a cache hit for a leaf switch, and the temporary state (ITL) in case of a cache miss for a leaf switch may be referred to as the first temporary state (T), the second temporary state (IT), the third temporary state (TL), and the fourth temporary state (ITL), respectively. In addition, unless otherwise explained, the temporary state or the temporary state (T) may refer to the temporary state (T) in case of a cache hit. A more detailed description of the state information will be provided later.
[0064] For convenience of explanation, an example is presented in which a switch module (SM) according to an embodiment of the present invention is connected with three cache coherence switches (SW) in a tree-like topology. The three cache coherence switches (SW) are referred to as a first switch (SW1), a second switch (SW2), and a third switch (SW3). Even if the number of cache coherence switches (SW) included in the switch module (SM) is greater, each cache coherence switch (SW) can operate identically to one of the first switch (SW1), the second switch (SW2), and the third switch (SW3) described below. The first switch (SW1) can be connected to the first device (DEV1) and the second device (DEV2), the second switch (SW2) can be connected to the home agent (HA), and the third switch (SW3) can be connected to the third device (DEV3).
[0065] The first switch (SW1) can receive a read request from the first device (DEV1) (S510). The read request is one of the memory requests. The first switch (SW1) can determine whether data corresponding to the read request exists in the included switch cache (SC) (S520). That is, it can be confirmed whether a switch cache hit occurs in the first switch (SW1). If the data corresponding to the read request exists in the switch cache (SC) of the first switch (SW1), the first switch (SW1) can transfer the data of the switch cache (SC) to the first device (DEV1) and forward the read request to another switch of the switch module (SM) to maintain cache consistency (S530). That is, the read request is forwarded to the second switch (SW2) located in the upper node of the first switch (SW1).
[0066] At this time, the first switch (SW1) can generate a host response to the read request and transmit data to the first device (DEV1) together with the host response. In addition, the operation of the first switch (SW1) transmitting data to the first device (DEV1) and the operation of transmitting the read request to the second switch (SW2) can be performed simultaneously or can precede or follow each other with a slight time difference.
[0067] The second switch (SW2) transmits the read request received from the first switch (SW1) to the home agent (HA) (S540). The home agent (HA) responds to the read request by generating a host response and a cache coherence message and transmitting them to the second switch (SW2) (S550). That is, the host response and cache coherence message are transmitted to the second switch (SW2) of the switch module (SM), and the second switch (SW2) can forward them to the first switch (SW1). The switch (SW1) can perform cache coherence management by changing state information in response to the cache coherence message (S560).
[0068] A home agent (HA) according to an embodiment of the present invention can generate a Snoop message as a cache coherence message. The Snoop message includes a Snoop Data (SnpData) message and a Snoop Invalidate (SnpInv) message. The SnpData message can change the state of a cache line to a shared state (S), and the SnpInv message can change the state of a cache line to an invalidated state (I). A cache coherence message according to an embodiment of the present invention can also be generated by a device (DEV). For example, a second device (DEV2) can transmit a SnpMiss message or the like to the home agent (HA) through a first switch (SW1) and a second switch (SW2). A further description of the SnpMiss message will be provided below.
[0069] The first switch (SW1), which has received the host response and cache coherence message generated by the home agent (HA) through the second switch (SW2), may not forward the host response along with the data to the first device (DEV1), and therefore may not forward it to the first device (DEV1). On the other hand, the first switch (SW1) responds to the cache coherence message and changes the state information to maintain cache coherence in the server system (100) (S560). At this time, the cache coherence may mean coherence of the switch cache (SC) of the first switch (SW1) to the third switch (SW3) together with the device (DEV) and the host processor (HP).
[0070] As described above, the first switch (SW1) includes a cache coherency manager (CC) that performs cache coherency management operations. The cache coherency manager (CC) may include at least one of a switch agent (SA), an eviction monitor (EM), and a topology manager (TM).
[0071] FIG. 7a and FIG. 7b are diagrams for explaining the operation of a switch agent (SA) according to an embodiment of the present invention.
[0072] First, referring to FIGS. 3, 4, 6, and 7a, when data "5" is shared in the cache lines of the second device (DEV2) and the third device (DEV3), the first switch (SW1) can receive a read request for the corresponding address from the first device (DEV1) in the initial state, i.e., the invalid state (I) (↘). Since the first switch (SW1) is in the valid state (V) and the corresponding data exists in its switch cache (SC), a cache hit occurs in the first switch (SW1). Accordingly, the first switch (SW1) transfers the corresponding data to the first device (DEV1), and the first device (DEV1) changes to the shared state (S).
[0073] The first switch (SW1) transmits a read request to the home agent (HA) (♪), but before that, a third device (DEV3) may request that data "10" be written to the corresponding address (♭). Accordingly, the home agent (HA) may transmit a cache coherence message to invalidate the second device (DEV2) sharing the corresponding address via the first switch (SW1) before the write operation (♩). At this time, the cache coherence message may be a SnpInv message. The same example is described below, but is not limited thereto.
[0074] However, at that point, the home agent (HA) may not know that the first device (DEV1) is also in a shared state (S) for the cache line, so the SnpInv message may not be delivered to the first device (DEV1). After that, the home agent (HA) may receive a read request from the first device (DEV1) and process it as if the first device (DEV1) and the third device (DEV3) share the address. Therefore, since the data recognized by the first device (DEV1) and the third device (DEV3) for the address are different, "5" and "10," cache coherence may be damaged.
[0075] Next, referring to FIGS. 3, 4, 6 and 7b, the switch agent (SA) according to an embodiment of the present invention can change the state information of a cache line to a temporary state (T) in order to prevent such cache coherency damage. The temporary state (T) is a state defined for a period of time after a cache hit occurs in the first switch (SW1), the first switch (SW1) transfers data to the first device (DEV1) and forwards a read request to the second switch (SW2), and until a host response generated from the home agent (HA) is transferred to the first switch (SW1). Accordingly, the first switch (SW1) can be changed to a valid state (V) only after the temporary state (T) ends.
[0076] The switch agent (SA) can record the device to which the first switch (SW1) transferred data of the switch cache (SC), i.e., the first device (DEV1), in the list of outstanding devices, i.e., the list of managed devices for the corresponding cache line, along with the setting of the temporary state (T). In this case, even if there is a write request for the corresponding address from the third device (DEV3) as shown in Fig. 7a in the temporary state (T), the first switch (SW1) can refer to the list of outstanding devices when the SnpInv message is received and transfer the SnpInv message not only to the second device (DEV2) but also to the first device (DEV1). Accordingly, the first device (DEV1) can also have an invalid state (I), so that cache coherence can be achieved.
[0077] FIG. 8a and FIG. 8b are drawings for explaining the operation of a removal monitor (EM) according to an embodiment of the present invention.
[0078] First, referring to FIGS. 3, 4, 6, and 8A, similarly to FIG. 7A, when data "5" is shared in the cache lines of the second device (DEV2) and the third device (DEV3), the first switch (SW1) can receive a read request for the corresponding address from the first device (DEV1) in an invalid state (I). Since the first switch (SW1) is in a valid state (V) and the corresponding data exists in its switch cache (SC), a cache hit occurs in the first switch (SW1). Accordingly, the first switch (SW1) transfers the corresponding data to the first device (DEV1), and the first device (DEV1) also changes the state information so that it shares the cache line.
[0079] The first switch (SW1) transmits a read request to the home agent (HA), but before that, a request to remove data at the corresponding address may be generated from the second device (DEV2). In this case, the home agent (HA) releases the shared state (S) of the cache line for the second device (DEV2) and changes it to the initial state (invalid state (I)). After that, a request to write data "10" to the corresponding address may be received from the third device (DEV3).
[0080] However, since there is no device sharing a cache line with the third device (DEV3) from the home agent (HA)'s perspective, the home agent (HA) does not deliver the SnpInv message to the first switch (SW1). In other words, the SnpInv message is not delivered to the second device (DEV2) or the first device (DEV1). Therefore, even though the temporary state (T) is set, the data recognized by the first device (DEV1) and the third device (DEV3) for the address are different, "5" and "10", which may damage cache consistency.
[0081] Next, referring to FIGS. 3, 4, 6, and 8b, the eviction monitor (EM) according to an embodiment of the present invention can set a sharer count and adjust the sharer count in response to a cache coherence message to solve the cache coherence problem of FIG. 8. At this time, the cache coherence message may be an eviction message. The same example is described below. The sharer count represents the number of devices (DEV) connected to the first switch (SW1) that are in a shared state (S). For example, among the first device (DEV1) and the second device (DEV2) connected to the first switch (SW1), only the second device has a cache line in a shared state (S), so the eviction monitor (EM) can set the sharer count to "1". The sharer count set to "1" can be stored in the switch cache (SC) of the first switch (SW1).
[0082] Additionally, the eviction monitor (EM) can decrease the value of the sharer count by one whenever an eviction message is received in the temporary state (T). And when the sharer count becomes an arbitrary value (e.g., "0") in the temporary state (T), since there is no device (DEV) owning the corresponding cache line at that point, the eviction monitor (EM) can forward a SnpInv message to the device on the list of outstanding devices for the corresponding cache line, i.e., the first device (DEV1). This can prevent cache coherency violations from occurring.
[0083] This can be equally applied to the problem of cache coherence violation due to the operation of so-called 'silent evictions', which voluntarily invalidate cache lines that are no longer in use without notifying the home agent (HA) of the eviction result. The eviction monitor (EM) according to an embodiment of the present invention can track the SnpMiss message, which is one of the snoop messages transmitted from the connected device (DEV). When the device (DEV) receives a SnpData message or a SnpInv message for a cache line that has been silently evicted from the home agent (HA), it can return a SnpMiss message to the home agent (HA). The eviction monitor (EM) decrements the share count each time it receives a SnpMiss message, and when the share count becomes "0", it can transmit a SnpInv message to the device (DEV) recorded in the outstanding device list.
[0084] FIGS. 9A to 9C are diagrams for explaining the operation of a topology manager (TM) according to an embodiment of the present invention.
[0085] First, referring to FIGS. 3, 4, 6 and 9a, even though a cache hit occurs in the first switch (SW1) in response to a read request from the first device (DEV1) and data in the switch cache (SC) of the first switch (SW1) is transferred to the first device (DEV1), the second switch (SW2) can also transfer data in its switch cache (SC) to the first device (DEV1) via the first switch (SW1). This is because a cache hit may also occur in the second switch (SW2) since the first switch (SW1) forwards the read request from the first device (DEV1) to the second switch (SW2).
[0086] As described above, the first switch (SW1) and the second switch (SW2) can transmit data from their switch caches (SC) to the first device (DEV1) along with the host response. Therefore, the device (DEV) can perform communication without any separate changes according to the preset protocol, regardless of the operation or structure of the switch module (SM). The host response generated by the home agent (HA) is also sequentially transmitted to the first device (DEV1) through the second switch (SW2) and the first switch (SW1).
[0087] However, if the first device (DEV1) receives multiple host responses in this way, a functional error may occur. In addition, if a leaf switch such as the first switch (SW1) cannot recognize whether the transmitted host response was generated by another switch or by the home agent (HA), a situation may occur in which the second switch (SW2) responds to the generated host response and changes the temporary state (T) to the valid state (V).
[0088] Next, referring to FIGS. 3, 4, 6, and 9b, as described above, a leaf switch such as the first switch (SW1) can omit the operation of forwarding a host response generated by a home agent (HA) to the first device (DEV1). Furthermore, the same applies to a host response generated by a root switch such as an internal switch or a second switch (SW2).
[0089] In order to block such duplicate host responses, the topology manager (TM) according to an embodiment of the present invention can set a third temporary state (TL). For example, when a cache hit occurs in the first switch (SW1), which is a leaf switch, the valid state (V) is changed to the third temporary state (TL), and additionally transmitted host responses can be blocked in the section set to that state. However, as in FIG. 9a, since the third temporary state (TL) is terminated and changed to the valid state (V) by the host response generated by the second switch (SW2), not the host response generated by the home agent (HA), the first switch (SW1) may not guarantee cache consistency.
[0090] Next, referring to FIGS. 3, 4, 6 and 9c, the topology manager (TM) according to an embodiment of the present invention can generate a local message having a meaning that is valid only within the switch module (SM). That is, since the response generated by the second switch (SW2) is generated in the form of a local message, the first switch (SW1) can distinguish it from the host response by the home agent (HA). As a result, the first switch (SW1) can maintain the third temporary state (TL) when receiving the local message, and can terminate the third temporary state (TL) only when the host response by the home agent (HA) is received.
[0091] Figure 9c illustrates a "Local Go" local message. A "GO" (Global Observation) response in standards such as CXL, which is transmitted only after the Home Agent (HA) has secured ownership of the requested cache line, also corresponds to the aforementioned host response. In other words, "Local Go" can be a local message corresponding to a host response.
[0092] A local message can be generated by setting the attribute value of the reserved bit of a cache coherence message. Accordingly, the effects of the server system (100), switch module (SM), and switching method (500) according to an embodiment of the present invention can be achieved without changing the interconnect protocol or device (DEV) that defines the cache coherence message.
[0093] In this way, according to the server system (100), switch module (SM), and switching method (500) according to an embodiment of the present invention, even when the memory request has not been delivered to the home agent (HA), the memory response is delivered to the device (DEV) by the cache coherency switch (SW) including the switch cache (SC), thereby increasing the opportunity for data reuse and enabling a quick response to the request. In addition, since the cache coherency management is distributed by the switch module (SM), the overhead thereof is reduced, so that performance can be maintained even when more devices are connected. For example, as shown in the above-described FIGS. 7b and 8b, cache coherency management can be performed by explicitly calling invalidation on the device (DEV) to which data cached in the switch cache (SC) by the switch module (SM) is delivered.
[0094] Furthermore, since no change in design or control of the host processor (HP) and device (DEV) is caused regardless of the structure and operation of the switch module according to the embodiment of the present invention, compatibility and expandability can be guaranteed, and since there is no need to expand or add a cache within the device (DEV) to increase the cache hit rate, the device size can be maintained.
[0095] FIG. 10 is a diagram illustrating a cache coherence switch (SW) according to an embodiment of the present invention.
[0096] Referring to FIGS. 3 and 10, a cache coherency switch (SW) according to an embodiment of the present invention may include a packet decoder (PD), a switch cache (SC), a switch agent (SA), an eviction monitor (EM), a topology manager (TM), and a fabric manager (FM).
[0097] The packet decoder (PD) forwards the cache coherence message received by the cache coherence switch (SW) to the corresponding logic depending on the type. For example, the packet decoder (PD) can forward the SnpInv message to the switch agent (SA), and the SnpMiss message or the removal message to the removal monitor (EM). In addition, the packet decoder (PD) can forward local messages such as Local-GO to the topology manager (TM), and other read requests, data, etc. to the switch cache (SC).
[0098] The operations of the switch agent (SA), the removal monitor (EM), and the topology manager (TM) may be as described above. In addition, the switch agent (SA), the removal monitor (EM), and the topology manager (TM) may be included in the cache coherency management unit (CC), as illustrated in FIG. 3.
[0099] A switch cache (SC) includes a cache array (CA) and a state controller (CT). The cache array (CA) can store data, addresses (Addr), and state information (State) of a cache line, as in FIG. 3. Furthermore, in the case of FIG. 10, the cache array (CA) can further store a list of devices being managed by the corresponding cache coherence switch (SW), i.e., a list of outstanding devices (o / s dev), and the number of devices sharing the corresponding cache line, i.e., a sharer count (Sharer cnt). The list of outstanding devices and the sharer count can be as described above.
[0100] The state controller (CT) can control the state of a cache line. For example, the state controller (CT) can set the cache line to a valid state (V) and an invalid state (I), as well as the first temporary state (T) to the fourth temporary state (ITL) described above. The conditions under which the state controller (CT) sets the cache line to a certain state may be as described above. A more detailed explanation is provided below.
[0101] FIG. 11 is a table for explaining the state transition operation of a cache coherence switch (SW) according to an embodiment of the present invention.
[0102] Referring to FIGS. 6, 10, and 11, at a first point in time, a cache line corresponding to any cache coherence switch (SW) may be in one of a valid state (V), an invalid state, a first temporary state (T), a fourth temporary state (ITL), and the like. In each state, events such as a read request and a removal message may occur, and depending on the type of each state and event, the state information of the cache line may be changed as a result of the operation of the server system (100), the switch module (SM), or the cache coherence switch (SW) according to an embodiment of the present invention.
[0103] First, in the valid state (V), i.e., when a cache hit occurs in the cache coherence switch (SW) for a read request in the state where the cache line consistency is maintained, the first switch (SW1) or the third switch (SW3), which is a leaf switch, outputs data and a host response, the second switch (SW2), which is an internal switch, outputs data and a local message, and the read request can be forwarded to the home agent (HA). In this case, the internal switch can change to the first temporary state (T), and the leaf switch can change to the third temporary state (TL) (see FIGS. 7b and 8b). When a remove message or a SnpMiss message is received at the cache coherence switch (SW) in the valid state (V), the share count stored in the switch cache (SC) of the cache coherence switch (SW) is decremented. When the share count becomes "0", the device (DEV) connected to the cache coherence switch (SW) changes to the invalid state (I) (see FIG. 8b). When a SnpInv message or write request is received in the valid state (V), it can be changed to the invalid state (I).
[0104] When a read request is received in the invalid state (I), a cache line is reserved and the read request can be forwarded to the home agent (HA). In this case, the leaf switch, the first switch (SW1) or the third switch (SW3), can be changed to the fourth interim state (ITL), and the internal switch, the second switch (SW2), can be changed to the second interim state (IT).
[0105] When a read request or a write request is received in the first temporary state (T), the read or write operation corresponding to the request may be held until a host response is received (see FIG. 7b). When a remove message or a SnpMiss message is received in the first temporary state (T), the sharer count is decremented (see FIG. 8b), and if the sharer count is "0", a related notification may be delivered to the remove monitor (EM). When a SnpInv message is received in the first temporary state (T), a related notification may be delivered to the switch agent (SA). When data and a host response are delivered in the first temporary state (T), the data and the host response may be delivered to the device (DEV), and the sharer count may be incremented. And the corresponding cache line may be changed to a valid state. When data and local messages are delivered between cache coherence switches (SW) in the first temporary state (T), delivery of data and local messages to the device (DEV) may be blocked. Except for the case where data and host responses are transmitted in the first temporary state (T), no state change may occur due to an event occurring in the first temporary state (T). As described above, the first temporary state (T) may be changed only after receiving a host response generated by the home agent (HA).
[0106] In the case of the first switch (SW1) or the third switch (SW3), which is a leaf switch in the third temporary state (TL), i.e., the temporary state in which a cache hit of the switch cache (SC) occurs, if data and a host response are received, no additional forwarding operation is performed because the data and the host response have already been forwarded to the corresponding device (DEV). However, the corresponding device (DEV) is added to the sharer count, and the state of the cache line may be changed to a valid state (V). If other events occur in the third temporary state (TL), the case may be the same as the example of the first temporary state (T).
[0107] In the second interim state (IT), i.e., the interim state in which a cache miss occurs in the switch cache (SC), when a read request is received, the read operation corresponding to the request may be held until a host response is received. In the second interim state (IT), when data and a host response or a local message are received, the data and the host response or the local message may be forwarded to another cache coherence switch (SW) or device (DEV). When a host response is received, the share count of the cache coherence switch (SW) is incremented, and the cache line may be converted to a valid state (V). On the other hand, when a local message is received, the cache line may be converted to a first interim state (T).
[0108] In the fourth interim state (ITL), that is, the interim state in which a cache miss of the switch cache (SC) occurs, in the case of the first switch (SW1) or the third switch (SW3), which is a leaf switch, when data and a local message are received, the local message can be converted into a host response and the data and the host response can be transmitted to the corresponding device (DEV). Then, the first switch (SW1) or the third switch (SW3) can be converted to the third interim state. If other events occur in the fourth interim state (ITL), the case may be the same as the example of the second interim state (IT).
[0109] Referring again to FIGS. 6 and 10, the cache coherency switch (SW) according to an embodiment of the present invention may further include a fabric manager (FM). The fabric manager (FM) may connect a plurality of cache coherency switches (SW) into one of a first topology and a second topology in response to a topology selection signal. For example, the first topology may be the tree topology of FIG. 1, and the second topology may be the all-to-all topology of FIG. 2. However, the present invention is not limited thereto, and the fabric manager (FM) according to an embodiment of the present invention may select one of various topologies to connect a plurality of cache coherency switches (SW). For example, the second topology may be a star topology or a mesh topology.
[0110] To this end, a fabric manager (FM) according to an embodiment of the present invention may include a control signal interface, a reconfigurable routing table, and a reconfigurable port. The control signal interface provides an interface with a user (e.g., a system operator). For example, the control signal interface may receive a topology selection signal. The control signal interface may reconfigure the connection path values of a plurality of cache coherence switches (SWs) in response to the topology selection signal. At this time, the new routing table may be received through the control signal interface. The reconfigurable port may reconfigure the number of cache coherence switches (SWs) connected among the plurality of cache coherence switches (SWs), i.e., the bandwidth per port, in response to the topology selection signal.
[0111] That is, in order to change the topology between cache coherence switches (SWs) in a server system (100) according to an embodiment of the present invention, only a simple operation may be required of the system operator. First, a topology may be selected, and the physical connection between the switches may be adjusted using reconfigurable ports (bandwidth per port) and external cables to form the target topology. Then, a new routing table may be provided to the cache coherence switches (SWs) via a control signal interface so that each cache coherence switch (SW) knows its position within the selected topology.
[0112] At this time, two rules can be adopted to establish a unique path for cache coherence messages between the device (DEV) and the home agent (HA). First, a reconfigurable routing table can enforce a unique path for a specific cache coherence message, allowing all traffic between the device (DEV) and the home agent (HA) to be monitored. Second, a NINE (Non-Inclusive and Non-Exclusive) cache policy can be applied. The NINE policy allows the cache coherence switch (SW) to be placed anywhere along the path in a given topology, because the cache entries in each cache coherence switch (SW) can have different cache hierarchies determined by the reconfigurable routing table.
[0113] As described above, trade-offs may exist between various topologies, such as the number of cache coherence switches (SWs) forming a path, i.e., hop count, and average bandwidth. Therefore, the optimal topology for the workload processed by the server system (100) may vary. According to the server system (100), switch module (SM), and switching method (500) according to an embodiment of the present invention, the flexibility of the structure and operation can be improved by operating with a topology for cache coherence switches (SWs) optimized for the workload in a simple manner.
[0114] FIG. 12 is a diagram showing the overall switching architecture in a server system (100) according to an embodiment of the present invention, and FIG. 13 is a diagram showing a scheduling algorithm of a cache coherence switch (SW) according to an embodiment of the present invention.
[0115] Referring to FIGS. 6, 12, and 13, the server system (100) according to an embodiment of the present invention may include hardware and software configurations to implement the aforementioned structure and operation. Since the hardware configuration is as described above, a detailed description thereof will be omitted.
[0116] Furthermore, the software configuration of the server system (100) according to an embodiment of the present invention can maximize the opportunity for data reuse between user requests through simple and efficient scheduling. That is, the scheduling can maximize the utilization of cache locality so as to increase the cache hit rate in the cache coherence switch (SW). In addition, the software configuration of the server system (100) according to an embodiment of the present invention can perform load balancing for user requests from devices (DEV).
[0117] To this end, the software configuration of the server system (100) according to the embodiment of the present invention can perform the following scheduling, including a global request queue (GQ), a switch scheduler (SH), and a device request queue (DQ).
[0118] First, an exclusive range of a global address space can be statically allocated to each device (DEV) offline (Algorithm 1, Line 1). For example, the first device (DEV1) can be assigned addresses 0 to 99, the second device (DEV2) can be assigned addresses 100 to 199, and the third device (DEV3) can be assigned addresses 200 to 299.
[0119] At runtime, the request decoder (RD) of the switch scheduler (SH) can decode a set of requests received through the global request queue (GQ) into individual user requests. That is, a set of requests can contain multiple user requests.
[0120] The device scorer (RS) of the switch scheduler (SH) can check the corresponding address for each user request and assign a score to each device (DEV) (Algorithm 1, Lines 4-5). At this time, a high score can be assigned to the device (DEV) having the most overlapping address range with the address corresponding to the user request. For the example described above, if the address corresponding to one user request is "50", the highest score can be set to the first device (DEV1). Then, the second device (DEV2) that is relatively close to the address corresponding to the user request can be assigned the next highest score, and the third device (DEV3) can be assigned the lowest score. Then, the device (DEV) with the highest score can be selected as the candidate device (Algorithm 1, Line 6).
[0121] Next, in order to utilize the locality of the switch cache (SC) included in the cache coherency switch (SW), the group scorer (GS) of the switch scheduler (SH) adds a group score that reflects the relationship with the cache coherency switch (SW), and can be repeatedly assigned to all devices under the cache coherency switch (SW) (Algorithm 1, Lines 7-8). For example, in an example where the first device (DEV1) is set as a candidate device, the highest group score can be assigned to the device (DEV) that shares the first switch (SW1) to which the first device (DEV1) is directly connected.
[0122] In this way, the scoring operation of the switch scheduler (SH) according to the embodiment of the present invention can be set as simply as possible, thereby minimizing the overhead resulting therefrom.
[0123] Finally, the load balancer (LB) of the switch scheduler (SH) can determine which device (DEV) to send to based on the score and load of each device (DEV) (Algorithm 1, Lines 9-10). At this time, the load of the candidate device can be checked. If the candidate device is overloaded, i.e., the number of acceptable user requests has already been allocated to the candidate device, the load balancer (LB) can change the device with the next highest score to the candidate device.
[0124] Through the above process, a user request can be assigned to a selected candidate device (Algorithm 1, Lines 11-12). This can be accomplished by forwarding the user request to the device request queue (DQ) corresponding to the candidate device.
[0125] As such, the server system (100), switch module (SM), and switching method (500) according to an embodiment of the present invention can achieve both maximum locality utilization and load balancing by prioritizing data reuse opportunities between user requests until load distribution is permitted. Therefore, the server system (100), switch module (SM), and switching method (500) according to an embodiment of the present invention can perform switching operations more quickly and efficiently even for large-capacity data applications.
[0126] While representative embodiments of the present invention have been described in detail above, those skilled in the art will appreciate that various modifications to the above-described embodiments are possible without departing from the scope of the present invention. Therefore, the scope of the present invention should not be limited to the described embodiments, but should be defined not only by the claims set forth below but also by equivalents thereof.
Claims
1. Includes a plurality of cache-coherent switches interconnected in a first topology; Each of the above cache coherency switches is: A switch cache containing data, address, and status information; and A switch module including a cache coherence management unit that manages cache coherence of the above switch cache.
2. In claim 1, Each of the above cache coherency switches is: A switch module that outputs data together with a memory response to a memory request received from the outside when data corresponding to the memory request exists in the switch cache, and transmits the memory request to another cache coherence switch among a plurality of cache coherence switches or to the outside.
3. In claim 2, The above cache coherence management unit, A switch module that receives a cache-coherent message and updates the above state information.
4. In claim 1, The above cache coherence management unit, A switch module including a switch agent that changes the state information to a temporary state when data of the switch cache is output to the outside.
5. In claim 1, The above cache coherence management unit, A switch module comprising an eviction monitor that adjusts a sharer count in response to a cache coherency message received from an external source.
6. In claim 1, The above cache coherence management unit, A switch module comprising a topology manager that generates a local message specified between the plurality of cache coherence switches connected in the first topology.
7. In claim 6, The above local message is, A switch module created by setting the attribute value of the reserved bit of a cache coherence message received from the outside.
8. In claim 1, The above cache coherency switch is, A switch module further comprising a fabric manager, in response to a topology selection signal, for connecting a plurality of said cache coherency switches into a second topology different from said first topology.
9. In claim 8, The above fabric manager, A control signal interface for receiving the above topology selection signal; A reconfigurable routing table that reconfigures connection path values of a plurality of cache coherence switches in response to the topology selection signal; and A switch module including a reconfigurable port that reconfigures the number of cache coherence switches connected among a plurality of cache coherence switches in response to the topology selection signal.
10. In claim 1, The above status information is, A switch module representing a modified state, an exclusive state, a shared state, an invalid state, a valid state, a temporary state depending on whether a cache hit occurred, or a temporary state depending on whether a cache hit occurred for a leaf switch among a plurality of cache coherence switches.
11. In claim 1, A number of the above cache coherency switches are: A switch module that is a Compute Express Link (CXL) switch or a Cache Coherent Interconnect for Accelerator (CCIX) switch.
12. Multiple devices; host processor; and A server system comprising a switch module that connects a plurality of devices and the host processor or connects between the plurality of devices using an interconnect protocol with controlled cache coherency, and transmits data corresponding to a memory request received from a first device among the plurality of devices to the first device and then transmits the memory request to the host processor.
13. In claim 12, The above switch module, A server system comprising a plurality of cache coherence switches interconnected in a first topology, each having a switch cache and a cache coherence management unit.
14. In claim 13, The above cache coherence management unit, A switch agent that changes the state information to a temporary state when data of the switch cache is transmitted to the first device; In response to the above cache coherency message, an eviction monitor that adjusts the sharer count; and A server system comprising at least one of a topology manager that generates a local message defined between the plurality of cache coherence switches connected in the first topology by setting an attribute value of a reservation bit of the cache coherence message.
15. In claim 14, Each of the above cache coherency switches is: A server system further comprising a fabric manager, in response to a topology selection signal, interconnecting the first topology with a second topology different from the first topology.
16. A step in which a first switch of a switch module receives a read request from a first device among a plurality of devices connected by an interconnect protocol in which cache coherency is controlled; A step of determining whether data corresponding to the read request exists in the switch cache included in the first switch; When a cache hit occurs in the first switch, the first switch transfers data of the switch cache to the first device and forwards the read request to another switch of the switch module; A step in which a second switch of the switch module to which the read request is forwarded forwards the read request to a home agent of a host processor connected via the interconnect protocol; The home agent generates a host response and a cache coherence message in response to the read request and transmits the same to the second switch; and A switching method comprising: a step of changing state information so that cache coherence is maintained in response to the cache coherence message transmitted through the second switch by the first switch.
17. In claim 16, The first switch skips forwarding the host response generated by the home agent to the first device when a cache hit occurs in the first switch; and A switching method further comprising: a step of converting a local message received from the second switch into the host response and transmitting the same to the first device when a cache miss occurs in the first switch.
18. In claim 16, A step in which the first switch sets a temporary state from the time it forwards the read request until it receives the host response from the home agent; and A switching method further comprising: when an invalid message is received in the temporary state, the first switch transmits the invalid message to the first device.
19. In claim 16, A step of setting the first switch to a temporary state from the time it forwards the read request until the host response is received from the home agent; When a removal message is received in the above temporary state, the first switch decrements the share count; and A switching method further comprising: a step of transmitting an invalid message to the first device when the share count becomes the first value by the first switch; 20. In claim 16, A switching method further comprising: a step of changing the interconnection method in response to a topology selection signal by the first switch and the second switch.
Citation Information
Patent Citations
Storage chip, electronic equipment and storage system
CN115309670A
High performance interconnect coherence protocol
KR1020170007865A
Apparatus for forming film on substrate in inkjet pringting equipment and method thereof
KR1020230063981A
Chlorine Salinity Measurement Device That Automatically Switches From Chlorine Measurement To Salinity Measurement
KR1020240152149A
Composable infrastructure enabled by heterogeneous architecture, delivered by CXL based cached switch soc
US20230017583A1
Cited By
PCIe cache consistency system implemented on switch
CN121187982A
