Chip-to-chip interconnection with hierarchical communication architecture

Abstract

A system includes a first integrated circuit package including a first set of one or more artificial intelligence processing units and a first chip-to-chip interconnect communication unit; and a second integrated circuit package including a second set of one or more artificial intelligence processing units and a second chip-to-chip interconnect communication unit. The system also includes an interconnect between the first integrated circuit package and the second integrated circuit package, where the first chip-to-chip interconnect communication unit and the second chip-to-chip interconnect communication unit use a hierarchical communication architecture that supports credit-based data flow control and retransmission data flow control, ethernet-based communications are managed via the interconnect.

Description

Technical Field

[0001] The present disclosure relates to chip-to-chip interconnects, and more particularly to chip-to-chip interconnects utilizing a layered communication architecture. Background Art

[0002] A data center is a facility that houses computer systems and components such as computing systems, data storage systems, and other associated components. Modern applications, such as training of artificial intelligence (AI) models, drive the need for high network capabilities, storage capabilities, and computing capabilities in data centers. A large number of processing units are often connected for large-scale computing tasks. However, these processing units are often connected in a way that involves cumbersome communication protocols and / or does not scale well with the growth of network speeds. Therefore, it would be beneficial to develop technologies for flexible and scalable chip-to-chip interconnects.

[0003] The present disclosure seeks to at least partially address any or all of the above-mentioned shortcomings and disadvantages. Summary of the invention

[0004] According to a first aspect of the present disclosure, a system is provided, comprising: a first integrated circuit package, the first integrated circuit package comprising a first group of one or more artificial intelligence processing units and a first chip-to-chip interconnect communication unit; a second integrated circuit package, the second integrated circuit package comprising a second group of one or more artificial intelligence processing units and a second chip-to-chip interconnect communication unit; and an interconnection between the first integrated circuit package and the second integrated circuit package, wherein the first chip-to-chip interconnect communication unit and the second chip-to-chip interconnect communication unit use a layered communication architecture that supports credit-based data flow control and retransmission data flow control to manage Ethernet-based communications via the interconnection.

[0005] In some embodiments, the first group of one or more artificial intelligence processing units and the second group of one or more artificial intelligence processing units can be configured to process an artificial intelligence workload that includes training one or more neural networks.

[0006] In some embodiments, the first chip-to-chip interconnect communication unit and the second chip-to-chip interconnect communication unit may include electronic circuitry logic configured to support a physical layer network communication protocol.

[0007] In some embodiments, the physical layer network communication protocol may include an Ethernet communication protocol.

[0008] In some embodiments, the first chip-to-chip interconnect communication unit and the second chip-to-chip interconnect communication unit may include electronic circuitry logic configured to support a plurality of communication protocols in addition to the physical layer network communication protocol.

[0009] In some embodiments, the first chip-to-chip interconnect communication unit and the second chip-to-chip interconnect communication unit may include a hardware buffer configured to store communication data.

[0010] In some embodiments, the interconnect may include a physical interconnect between the first integrated circuit package and the second integrated circuit package.

[0011] In some embodiments, the physical interconnection may include a wire interconnection.

[0012] In some embodiments, the layered communication architecture may include an application layer.

[0013] In some embodiments, the layered communication architecture may further include a physical network layer.

[0014] In some embodiments, the layered communication architecture may further include a reliability layer.

[0015] In some embodiments, the layered communication architecture may further include a message distribution and reassembly layer.

[0016] In some embodiments, the application layer may be configured to access high bandwidth memory.

[0017] In some embodiments, the credit-based data flow control may be configured to communicate a credit limit between the first integrated circuit package and the second integrated circuit package based on a receive buffer size of the first integrated circuit package and a receive buffer size of the second integrated circuit package.

[0018] In some embodiments, the credit-based data flow control may be configured to transmit and store a count of data packets exchanged between the first integrated circuit package and the second integrated circuit package.

[0019] In some embodiments, the retransmission data flow control may be configured to acknowledge receipt of an error-free data packet and to update a counter that records a consecutive number of error-free data packets received.

[0020] In some embodiments, the retransmission data flow control may be configured to communicate receipt of an erroneous data packet and cause the erroneous data packet to be retransmitted.

[0021] According to a second aspect of the present disclosure, a method is provided, the method comprising: configuring a first integrated circuit package, the first integrated circuit package comprising a first group of one or more artificial intelligence processing units and a first chip-to-chip interconnect communication unit; configuring a second integrated circuit package, the second integrated circuit package comprising a second group of one or more artificial intelligence processing units and a second chip-to-chip interconnect communication unit; and establishing an interconnection between the first integrated circuit package and the second integrated circuit package, wherein the first chip-to-chip interconnect communication unit and the second chip-to-chip interconnect communication unit use a layered communication architecture that supports credit-based data flow control and retransmission data flow control to manage Ethernet-based communication via the interconnection.

[0022] According to a third aspect of the present disclosure, a system is provided, comprising: a first integrated circuit package, the first integrated circuit package comprising a first group of one or more artificial intelligence processing units and a first chip-to-chip interconnect communication unit; a second integrated circuit package, the second integrated circuit package comprising a second group of one or more artificial intelligence processing units and a switch communication unit, wherein the switch communication unit is configured to provide an intermediate connection point between the first integrated circuit package and at least a third integrated circuit package; and an interconnection between the first integrated circuit package and the second integrated circuit package, wherein the first chip-to-chip interconnect communication unit and the switch communication unit use a layered communication architecture that supports credit-based data flow control and retransmission data flow control to manage Ethernet-based communications via the interconnection.

[0023] In some embodiments, the third integrated circuit package may include a third set of one or more artificial intelligence processing units and a second chip-to-chip interconnect communication unit.

[0024] It will be understood that any feature described herein as being suitable for incorporation into one or more aspects or embodiments of the present disclosure is intended to be generalizable to any and all aspects and embodiments of the present disclosure. Other aspects of the present disclosure may be understood by those skilled in the art based on the specification, claims, and accompanying drawings of the present disclosure. The foregoing general description and the following detailed description are exemplary and illustrative only and are not limitations on the claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Various embodiments of the invention are disclosed in the following detailed description and accompanying drawings.

[0026] Figure 1 is a block diagram illustrating a system of interconnected chips according to one or more embodiments of the present disclosure.

[0027] Figure 2 is a block diagram illustrating a chip-to-chip interconnect communication unit according to one or more embodiments of the present disclosure.

[0028] Figure 3 is a block diagram illustrating a layered communication architecture according to one or more embodiments of the present disclosure.

[0029] Figure 4 is a schematic diagram showing a framework of chip-to-chip data flow according to one or more embodiments of the present disclosure.

[0030] Figure 5 is a schematic diagram showing an example of credit-based data flow control according to one or more embodiments of the present disclosure.

[0031] Figure 6 is a schematic diagram showing an example of confirmation of a data flow according to one or more embodiments of the present disclosure.

[0032] Figure 7 is a diagram illustrating an example of non-acknowledgment of a data stream according to one or more embodiments of the present disclosure.

[0033] Figure 8 is a diagram illustrating an example of a data flow from the perspective of sending a request according to one or more embodiments of the present disclosure.

[0034] Fig. 9 is a diagram illustrating an example of data flow from the perspective of receiving a request according to one or more embodiments of the present disclosure.

[0035] Fig.10 is a flow chart illustrating a process for interconnecting chips according to one or more embodiments of the present disclosure.

[0036] Fig.11 is a block diagram illustrating a system of chips connected via point-to-point links according to one or more embodiments of the present disclosure.

[0037] Fig. 12A and Fig. 12B is a block diagram illustrating a system of chips connected via a unit having a switch according to one or more embodiments of the present disclosure.

[0038] Fig.13 is a block diagram illustrating an integrated circuit package including a hardware switch according to one or more embodiments of the present disclosure. DETAILED DESCRIPTION

[0039] The present invention can be implemented in many ways, including being implemented as: a process; an apparatus; a system; a composition of matter; a computer program product implemented on a computer-readable storage medium; and / or a processor, for example, a processor configured to execute instructions stored on a memory coupled to a processor and / or provided by a memory coupled to a processor. In this specification, these embodiments or any other form that the present invention may take may be referred to as technology. In general, the order of the steps of the disclosed process can be changed within the scope of the present invention. Unless otherwise stated, components such as processors or memories described as being configured to perform a task may be implemented as general components that are temporarily configured to perform the task at a given time, or as specific components manufactured to perform the task. As used herein, the term "processor" refers to one or more devices, circuits, and / or processing cores configured to process data such as computer program instructions.

[0040] The following are the appended drawings which illustrate the principles of the present invention. Figure 1 A detailed description of one or more embodiments of the invention is provided. The invention is described in conjunction with these embodiments, but the invention is not limited to any embodiment. The scope of the invention is limited only by the claims, and the invention encompasses many alternatives, modifications, and equivalents. Many specific details are set forth in the following description in order to provide a thorough understanding of the invention. These details are provided for illustrative purposes, and the invention can be practiced according to the claims without some or all of these specific details. For the sake of clarity, technical materials related to the invention known in the technical field are not described in detail so as not to unnecessarily obscure the invention.

[0041] A system includes: a first integrated circuit package, the first integrated circuit package including a first set of one or more artificial intelligence processing units and a first chip-to-chip interconnect communication unit; and a second integrated circuit package, the second integrated circuit package including a second set of one or more artificial intelligence processing units and a second chip-to-chip interconnect communication unit. The system also includes an interconnect between the first integrated circuit package and the second integrated circuit package, wherein the first chip-to-chip interconnect communication unit and the second chip-to-chip interconnect communication unit use a layered communication architecture that supports credit-based data flow control and retransmission data flow control to manage Ethernet-based communications via the interconnect.

[0042] In various embodiments, a protocol is defined to connect two computing devices. This connection can be described as a chip-to-chip interconnect. As used herein, a chip includes a semiconductor material cut from a larger wafer material on which transistors can be etched. A chip may also be referred to as a computer chip, a microchip, an integrated circuit (IC), a silicon chip, etc. The advantages of the technology disclosed herein include scalability and a more efficient communication protocol. The problem to be solved is how to achieve fast and flexible chip-to-chip communication. The problem is that chip-to-chip communication often relies on a chip-specific communication protocol that is neither flexible nor scalable. As further described in detail herein, a solution includes utilizing an existing physical layer protocol (e.g., Ethernet) that is widely used and / or implemented in existing devices for chip-to-chip communication. In order to utilize the existing physical layer protocol, in various embodiments, a layered communication architecture is deployed. In various embodiments, the layered communication architecture supports credit-based flow control and packet retransmission techniques.

[0043] The technology disclosed herein solves the problems associated with the growth of artificial intelligence (AI) clusters. In AI clusters, there are typically many types of interfaces. For example, an Internet protocol can be used, which can determine the network speed; but another interface can be used for chip-to-chip communication, which will determine the chip-to-chip speed. Different proprietary chip-to-chip protocols can be used simultaneously. Using the technology disclosed herein, a standard physical protocol can be applied to chip-to-chip communication to allow improved scalability, and additional application protocol layers and reliability protocol layers can be combined to support the standard physical protocol. Using these layers, since the standard physical protocol is implemented by all devices, it is possible to scale to any speed, meaning that device-to-device communication is possible without specialized switching hardware. As further described in detail herein, the technical advantages of this approach include: 1) partitioning responsibilities for each layer to provide: a layered architecture with flexible application layer interfaces that can be fine-tuned for any application, an interconnect that can be divided into multiple parallel slices to meet the required interface bandwidth (efficient load balancing and congestion control can be used to spread packets among multiple slices), and a scalable architecture that can potentially be extended to a single-hop network through standard switches; 2) full-duplex high-speed and low-latency links that can scale with network speed growth; 3) point-to-point resilient and lossless connections; 4) support for credit-based flow control to manage traffic congestion and receive overflow; 5) and lower overhead and footprint.

[0044] Figure 11 is a block diagram illustrating an embodiment of a system of interconnected chips. In the illustrated example, system 100 includes integrated circuit package 102, interconnect 108, and integrated circuit package 110. Integrated circuit package 102 includes one or more processing units 104 and a chip-to-chip interconnect communication unit 106. Integrated circuit package 110 includes one or more processing units 112 and a chip-to-chip interconnect communication unit 114.

[0045] In the example shown, integrated circuit packages 102 and 110 are independent devices. In some embodiments, integrated circuit packages 102 and 110 are independent AI accelerators. AI accelerators refer to high-performance computing devices specifically designed for efficient processing of AI workloads (e.g., neural network training). One or more processing units 104 of integrated circuit package 102 and one or more processing units 112 of integrated circuit package 110 may each include one or more processors of various types. Examples of processors include application-specific integrated circuits (ASICs), graphics processing units (GPUs), central processing units (CPUs), field-programmable gate arrays (FPGAs), multi-core scalar processors, spatial accelerators, or other types of components configured to perform logical operations and arithmetic operations on data as specified in instructions. In various embodiments, one or more processing units 104 and one or more processing units 112 process AI workloads. In other words, in various embodiments, one or more processing units 104 and one or more processing units 112 are artificial intelligence processing units.

[0046] In the example shown, the integrated circuit package 102 and the integrated circuit package 110 are directly connected via the interconnect 108. Therefore, in the example shown, the interconnect 108 is a chip-to-chip interconnect. In other words, the interconnect 108 includes a physical path for data exchange between the integrated circuit package 102 and the integrated circuit package 110. Regarding the physical path, in some embodiments, the interconnect 108 includes a traditional wire interconnect. It is also possible to adopt other signal transmission modes, such as interconnections through optical interconnect technology or wireless interconnect technology. In various embodiments, as further described in detail herein, a layered architecture is used to connect the integrated circuit package 102 and the integrated circuit package 110. The interface for the integrated circuit package 102 and / or the integrated circuit package 110 refers to a logical state connection between the integrated circuit package 102 and the integrated circuit package 110 that follows a common protocol. In the example shown, the integrated circuit packages 102 and 110 are connected in a direct connection topology. It is also possible that the integrated circuit packages 102 and 110 are connected through a switch topology. For example, integrated circuit packages 102 and 110 can be linked via a single-hop connection in which integrated circuit package 102 communicates with integrated circuit package 110 through another device (e.g., a hardware switch). In this scenario, the communication interface described herein will also be implemented on an intermediate device (e.g., a hardware switch). The example shown shows a point-to-point connection between two devices. This example is illustrative and not limiting. Using the techniques disclosed herein, any number of devices can be connected in any of a variety of ways (e.g., point-to-multipoint via a star topology, a mesh topology, a bus topology, or other topologies).

[0047] In the example shown, chip-to-chip interconnect communication units 106 and 114 implement communication interfaces for integrated circuit packages 102 and 110, respectively. In other words, the chip-to-chip interconnect communication unit of one device handles the transmission of data (e.g., created by a processing unit) to another device via interconnect 108. In the example shown, each chip-to-chip interconnect communication unit is shown as being separate from the associated one or more processing units. It is also possible that the chip-to-chip interconnect communication unit is integrated with one or more processing units (e.g., residing on the same integrated circuit). In some embodiments, the chip-to-chip interconnect communication unit is included in a hardware component (e.g., a network interface controller) that is specifically designed to connect different computers. As described in further detail below, in some embodiments, the chip-to-chip interconnect communication unit includes Ethernet communication logic, additional protocol logic, and hardware buffers.

[0048] Figure 22 is a block diagram showing an embodiment of a chip-to-chip interconnect communication unit. In some embodiments, the chip-to-chip interconnect communication unit 200 is Figure 1 The chip-to-chip interconnect communication unit 106 and / or Figure 1 The chip-to-chip interconnect communication unit 114 in.

[0049] In the example shown, the chip-to-chip interconnect communication unit 200 includes Ethernet communication logic 202, additional protocol logic 204 (computation logic), and hardware buffers 206. In some embodiments, a network interface controller (NIC) (also known as a network interface card, network adapter, or local area network adapter) is included in the chip-to-chip interconnect communication unit 200 and includes the Ethernet communication logic 202. The Ethernet communication logic 202 includes a processor configured to use a specific physical layer (e.g., Figure 1 The interconnect 108 and / or Figure 3 The Ethernet communication logic 202 may be included in a dedicated Ethernet integrated circuit.

[0050] In various embodiments, the additional protocol logic 204 includes electronic circuitry configured to communicate using various communication protocols in a layered communication architecture. The additional protocol logic 204 supports protocols other than the physical layer communications supported by the Ethernet communication logic 202. In some embodiments, the additional protocol logic 204 is included in an application specific integrated circuit, for example, included on an ASIC. It is also possible that the additional protocol logic 204 is integrated on the same chip as the Ethernet communication logic 202. In some embodiments, the additional protocol logic 204 includes electronic circuitry configured to perform Figure 3 Application layer 302, Figure 3 The message distribution and reassembly layer 304 and Figure 3 Processing of reliability layer 306 in.

[0051] The hardware buffer 206 is configured to store data across a layered communication architecture (e.g., Figure 3The hardware buffer 206 may include different buffers allocated for different types of data (e.g., a receive buffer for data packets received from a device, a transmit buffer for data packets to be sent to a device, etc.). In some embodiments, the hardware buffer 206 includes a memory implemented using digital logic gates. In the example shown, the hardware buffer 206 is shown as being separate from the Ethernet communication logic 202 and the additional protocol logic 204. It is also possible that the hardware buffer 206 is included in the Ethernet communication logic 202 or the additional protocol logic 204.

[0052] Figure 3 is a block diagram illustrating an embodiment of a layered communication architecture. In some embodiments, at least a portion of the architecture 300 is composed of Figure 1 The chip-to-chip interconnect communication unit 106, Figure 1 The chip-to-chip interconnect communication unit 114 and / or Figure 2 The chip-to-chip interconnect communication unit 200 supports.

[0053] In the example shown, architecture 300 includes an application layer 302, a message distribution and reassembly layer 304, a reliability layer 306, and a physical network layer 308. In the example shown, application layer 302 and message distribution and reassembly layer 304 communicate with high bandwidth memory 310. Layering of communication protocols as shown in architecture 300 has the advantage of allowing flexibility and more lightweight protocols. It is possible to change the features of each layer independently or to add additional features to each layer.

[0054] In various embodiments, the application layer 302 is connected to the communication library, and the application layer manages work requests from software applications. In various embodiments, from a workload perspective, an abstraction is provided to the application through a send buffer and a receive buffer interface. The send buffer and the receive buffer hold data between two communication partners and manage data reliability and credit flow. The send buffer is also used to hold unacknowledged data packets and thus manage any data loss recovery. The receive buffer can be a data holding buffer for pipeline purposes. In some embodiments, work requests from applications can be managed by a work request manager and interfaced with a direct memory access (DMA) engine. The DMA engine can move data between a high-bandwidth memory (HBM) and the send / receive buffers. This will be discussed in detail herein (e.g., see Figure 8 and Fig. 9 ) further details the management of work requests, including the management of sending and receiving data streams. Application layer 302 also manages connections between peers. In some embodiments, the logic of application layer 302 resides in Figure 2 In the additional protocol logic 204.

[0055] With respect to sending data, in various embodiments, the message distribution and reassembly layer 304 is responsible for segmenting and reassembling message packets based on work requests from the application layer 302. As part of message segmentation, the message distribution and reassembly layer 304 determines a load balancing scheme across the various reliability tiers and distributes the packets accordingly. In some embodiments, on a device (e.g., Figure 1 Switching from one port to another within an integrated circuit package 102 or 110 in an application program memory (API) is used for load balancing. Switching logic (electronic circuitry) can be used to distribute data packets to multiple data paths. In various embodiments, the message distribution and reassembly layer 304 maintains a unique, monotonically increasing message sequence number (MSN) and attaches it as part of the data packet header. After all messages have been reliably delivered to the receiver, the message distribution and reassembly layer 304 creates appropriate message completion entries for the application layer 302. With respect to received data, the message distribution and reassembly layer 304 maintains address lookups for incoming data packets. In various embodiments, since data packets can be received in an out-of-order manner, reassembly of data packets occurs directly in the high bandwidth memory 310 without the need for additional state. In various embodiments, address lookups and reassembly are based on the MSN carried by each data packet header. In some embodiments, the logic of the message distribution and reassembly layer 304 resides in Figure 2 In the additional protocol logic 204.

[0056] In various embodiments, the reliability layer 306 is maintained per network path, and the network path can be configured as a logical / physical port of an endpoint device. In various embodiments, each data packet passing through the reliability layer 306 is sent and received in order, and the reliability layer 306 maintains a monotonically increasing sequence as follows: the sequence indicates the order of transmission, thereby indicating the order of packet reception. In various embodiments, the reliability layer 306 maintains end-to-end credit, acknowledgment of packet data, and retransmission when a packet is dropped. In this document (e.g., see Figure 5 ) further describes credit-based data flows in detail. Also herein (e.g., see Figure 6 and Figure 7 ) further describes packet confirmation and retransmission in detail. In some embodiments, the logic of reliability layer 306 resides in Figure 2 In the additional protocol logic 204.

[0057] In various embodiments, the physical network layer 308 is an interface to another device. In some embodiments, the logic of the physical network layer 308 resides in Figure 2 In some embodiments, the physical network layer 308 includes a media access control (MAC) protocol and a physical coding sublayer (PCS) protocol. In some embodiments, the physical network layer 308 includes a serializer / deserializer (Serdes) functional block that serializes / deserializes digital data. In various embodiments, (e.g., in Figure 2 The Serdes function block implemented in the digital logic within the Ethernet communication logic 202 includes a parallel-to-serial data conversion function and a serial-to-parallel data conversion function.

[0058] Figure 4 4 is a schematic diagram showing a framework for chip-to-chip data flow. Framework 400 is a device (e.g., Figure 1 Integrated circuit package 102 or Figure 1 From the perspective of the integrated circuit package 110).

[0059] In the framework 400, with respect to the transmit (TX) direction 402, data from a retransmit buffer 404 (or another data storage location) passes through a MAC 406 and a Serdes 408 functional block. With respect to the receive (RX) direction 410, data is received via the Serdes 408 and MAC 406 functional blocks and placed in an RX buffer 412. In some embodiments, the MAC 406 and Serdes 408 are included in Figure 3 In some embodiments, the retransmit buffer 404 and the RX buffer 412 are included in the physical network layer 308. Figure 2 In the hardware buffer 206 in.

[0060] In various embodiments, various features are deployed and implemented for reliable and lossless traffic between link partners, for example, to prevent data corruption and packet loss. In some embodiments, MAC 406 utilizes a cyclic redundancy check (CRC) or other error detection code. The communication partner's MAC (MAC 414) can check the CRC and flag an error if the CRC does not match, and the receiver transmission logic can discard received packets with bad CRCs.

[0061] In various embodiments, a credit-based flow control mechanism is used to maintain overflow at the receiver. For example, the RX buffer 412 can store received data and periodically exchange available space in the receive buffer with the transmitter. The transmitter can check the available credits at the receiver before sending a data packet. In this way, the two communication partners are aware of each other's buffer availability and can maintain traffic flow. If there are not enough credits left in the receive buffer, throttling of the transmitter can be performed. In this document (e.g., see Figure 5 ) further describes credit-based flow control in detail.

[0062] In various embodiments, for reliable and lossless transmission, a mechanism is utilized for reporting losses and errors to the sender and for performing retransmission of discarded / lost data packets by the sender. In various embodiments, a packet sequence number (PSN), a retransmission buffer (e.g., retransmission buffer 404), and an acknowledgement (ACK) / no acknowledgement (NAK) flow control framework are utilized to manage the retransmission of lost / discarded data packets. In the example shown, the retransmission buffer 404 stores data packets being sent to the communication partner. The data packet can be deactivated after receiving an ACK and retransmitted after receiving a NAK. The sender can attach a sequence number to each data packet and keep a record of the sequence numbers of the data packets received without any errors. The receiver can periodically update the sequence number of the last good data packet received. In this document (e.g., see Figure 6 and Figure 7 ) further describes in detail the retransmission using ACK signals and NAK signals.

[0063] Figure 5 is a schematic diagram showing an example of credit-based data flow control. In some embodiments, Figure 5 The credit management logic shown is included in Figure 3 In the reliability layer 306.

[0064] In the example shown, a transmitter 502 sends a data packet to a receiver 506 via a link 504. The transmitter 502 (TX 502) and the receiver 506 (RX 506) are communication partners. In some embodiments, the transmitter 502 and the receiver 506 are Figure 1 The integrated circuit package 102 and Figure 1 In some embodiments, link 504 is Figure 1 The interconnect 108 in.

[0065] In various embodiments, credit-based flow control uses a credit count. A credit can be a block of data of a certain size (e.g., a credit can be a block of data of 64 bytes). In various embodiments, a maximum number of credits is supported (e.g., 2048 credits). With these example values, a total of 128 kilobytes (kB) of data is supported (2048×64 bytes=128 kB). In various embodiments, the TX 502 (transmitter) and the RX 506 (receiver) maintain an absolute credit count. In various embodiments, the TX 502 maintains a Total Blocks Sent (TBS) counter for recording the total blocks sent via the link 504 after initialization, updates it for each packet sent via the link 504, and periodically sends the TBS value to the RX 506. In addition, in various embodiments, the TX 502 receives a Credit Limit (CL) from the RX 506 and periodically stores the CL locally. In various embodiments, if the TBS+packet size is less than or equal to the CL, the TX 502 allows the packet to be sent. In various embodiments, RX 506 receives data in a receive buffer and maintains an Absolute Blocks Received (ABR) counter for recording the total blocks received via link 504, updates it for each packet received, and overwrites the ABR with the TBS received from TX 502. In some embodiments, the receive buffer is included in Figure 2 The hardware buffer 206 and / or Figure 4 In various embodiments, the RX 506 maintains a CL, which is calculated by adding the ABR and the available buffer space in the receive buffer. As indicated above, the CL is periodically sent by the RX 506 to the TX 502.

[0066] In various embodiments, upon initialization, TX 502 and RX 506 update available receive credits based on the size of the receive buffer. In various embodiments, upon initialization, TX 502 sets TBS and CL to 0 and waits for a CL update from RX 506. Additionally, in various embodiments, RX 506 sets ABR to 0 and updates CL based on the size of the receive buffer. In various embodiments, RX 506 sends the updated CL, which TX 502 receives and uses to overwrite its local copy of CL. When TX 502 receives a request to send a packet over link 504, TX 502 sends the packet if TBS + packet size is less than or equal to CL. Packets are sent at Figure 5508. After sending the packet, TBS is updated to TBS=TBS+packet size. The sent packet is received by RX 506 and stored in the local receive buffer of RX 506. RX 506 also updates the ABR by the packet size (of the received packet) and updates CL=ABR+remaining space in the receive buffer. Flow control updates are sent periodically. RX 506 periodically sends the latest CL to TX 502, which overwrites its local copy of CL with the received CL (action 510). In addition, TX 502 periodically sends the latest TBS to RX 506, and RX 506 overwrites its ABR with the received TBS (action 512). Using Figure 5 The credit management framework shown can manage and throttle (reduce) communication traffic when necessary. In some embodiments, the credit management framework is built on top of the Serdes and MAC Ethernet protocols (e.g., built on top of the MAC layer). The advantage of using existing Ethernet protocols is flexibility and versatility, because most existing devices have implemented the Ethernet protocol.

[0067] Figure 6 is a schematic diagram showing an example of confirmation of a data stream. In some embodiments, Figure 6 The logic for confirming the data flow is included in Figure 3 In various embodiments, packet sequence numbers (PSNs) and ACK / NAK flow control are used to convey the status of packets received by the communication partners.

[0068] In the example shown, communication between a sending side 602 and a receiving side 604 is shown. In some embodiments, the sending side 602 and the receiving side 604 correspond to Figure 1 The integrated circuit package 102 and Figure 1 In some embodiments, data is transmitted via the integrated circuit package 110 in the embodiment of the present invention. Figure 11 is transmitted between the send side 602 and the receive side 604. In the example shown, at step 606, the send side 602 sends a packet encapsulating the "Latest Sent PSN + 1" and increments the "Latest Sent PSN" count. Then, at step 608, the receive side 604 receives the packet and, if it is a good packet, causes the receive side logic to increment the "Latest Received PSN" by 1 and also increment the good packet count (based on the last Acknowledged PSN) by 1. At step 610, the receive side logic determines whether the good packet count is equal to N (which can be a specified value), and if the good packet count is equal to N, sends an ACK. In some embodiments, the ACK is sent as a Fibre Channel (FC) packet. In other words, after N good packets are received, the ACK is sent. At step 612, the send side 602 receives the ACK and extracts the PSN to overwrite the "Last Acknowledged PSN" with the PSN extracted from the ACK.

[0069] In the above series of steps, the sending side 602 maintains two PSNs: the last confirmation PSN that keeps a record of the last received ACK (good data packet); and the latest sent PSN. The receiving side 604 also maintains two PSNs: the last confirmation PSN of the last ACK (good data packet) sent back to the sending side 602; and the latest received PSN that keeps a record of the latest received data packet. In various embodiments, the sending side 602 also maintains a timer that is restarted each time a valid ACK is received; and the receiving side 604 maintains a count of consecutive good data packets received after the last ACK was sent. Figure 6 The steps shown accurately record the count of good packets sent from the sender 602 to the receiver 604 (acknowledgement of the data flow). In the case of errors (non-acknowledgement), the NAK process (e.g., see Figure 7 ).

[0070] Figure 7 is a schematic diagram showing an example of non-acknowledgement of a data stream. In some embodiments, Figure 7 The logic for non-acknowledgement of data flow is included in the Figure 3 In the example shown, communication between the sending side 702 and the receiving side 704 is shown. In some embodiments, the sending side 702 is Figure 6 The sending side 602 in, and the receiving side 704 is Figure 6 The receiving side 604 in.

[0071] In the example shown, at step 706, the receiving side 704 receives a bad packet (e.g., a packet length error with a CRC error, a PSN count error, a packet length error where the packet length is greater than the available space in the receiving buffer, etc.). The receiving side 704 discards the bad packet. Since the bad packet is received, the receiving side 704 cannot utilize the received packet. Figure 6 ACK process in. Instead, the latest received PSN is not incremented, and a NAK is sent to the sending side 702. The NAK encapsulates the non-incrementing "latest received PSN". At this point, the "last acknowledged PSN" is the "latest received PSN". Then, in step 708, the sending side 702 receives the NAK and extracts the PSN. The logic on the sending side 702 then begins a retransmission, where the latest sent PSN is updated to the PSN extracted from the NAK incremented by 1. The sending side 702 also overwrites the last acknowledged PSN with the PSN extracted from the NAK. Therefore, in the case of a bad packet (non-acknowledgement), the receiving side 704 instructs the sending side 702 how to roll back the packet sequence number to the last good packet count.

[0072] Figure 8 is a schematic diagram showing an example of a data flow from the perspective of sending a request. Figure 8 The data flow shown is composed of Figure 3 The work request manager of the application layer 302 in is managed. In the example shown, the work request manager maintains a queue 802 of outstanding requests. In various embodiments, these requests are processed in sequence due to the serialization of the chip-to-chip reliability layer. The size of the queue can be a parameterizable number N. From the perspective of sending a request, a published work request (e.g., a published work request 804) may include a data structure that includes a message identification, a message start sequence number, a source address, a message length, a message opcode destination queue pair number, and a metadata field. In the example shown, completed requests are placed in a completion queue 806. Once a request is completed, a completion update can be published back to the requesting application. In various embodiments, for a given request to be marked as completed, the last data packet of the message needs to be reliably placed in the buffer of the reliability layer of the receiver.

[0073] In the example shown, the DMA engine 808 reads data from the high bandwidth memory and segments it into maximum transport unit (MTU) sized payloads. In some embodiments, the high bandwidth memory is Figure 3As part of the payload segmentation, an additional header may be added to each segment, and the message sequence number field in the header may be incremented. This field may be used by the receive work request manager described below (e.g., see Fig. 9 ) is used. The message segmentation process load balances the packet distribution across different send buffers. This can be done based on a configured policy, such as deficit round robin or simple round robin arbitration. In order to support possible backpressure, data can be scheduled for a specific send buffer only if the space required for the request (MTU granularity) can be absorbed. The last segment of a message can have a separate flag and a completion event can be generated once the segment has been acknowledged from all available reliability layers.

[0074] Fig. 9 is a schematic diagram showing an example of a data flow from the perspective of receiving a request. Fig. 9 The data flow shown is composed of Figure 3 The work request manager of the application layer 302 in is managed. In various embodiments, when data is successfully received into the receive buffer (for example, the data passes the CRC check and the PSN check), the fields of the packet header are parsed. In some embodiments, the fields of the packet header that are processed include the message identification, the message sequence number, the destination queue pair number, the message operation code, and the identifier of the last packet of the message. The work request manager can use these fields to determine the destination address of the message segment. After the DMA engine 902 determines the destination address, the data segment is moved to the high bandwidth memory. In some embodiments, the high bandwidth memory is Figure 3 The high bandwidth memory 310 in the work queue manager. After the last packet of the message is successfully received, the work queue manager can generate a completion. In some embodiments, from the perspective of receiving the request, the software will publish the following information to the work request queue: message identification, message start sequence number, destination base address, expected message length, and metadata to be returned as part of the completion update. The work request can be maintained in a linear lookup table that is indexed based on the queue pair number and message sequence number extracted from the header.

[0075] Fig.10 is a flow chart illustrating an embodiment of a process for interconnecting chips. In some embodiments, the interconnected chips are Figure 1 The integrated circuit package 102 and Figure 1 The integrated circuit package 110 in FIG.

[0076] At 1002, a first integrated circuit package is configured, the first integrated circuit package including a first set of one or more artificial intelligence processing units and a first chip-to-chip interconnect communication unit. In some embodiments, the first integrated circuit package is Figure 1 In some embodiments, the first group of one or more artificial intelligence processing units is Figure 1 In some embodiments, the first chip-to-chip interconnect communication unit is Figure 1 In some embodiments, configuring the first integrated circuit package includes deploying a first set of one or more artificial intelligence processing units and a first chip-to-chip interconnect communication unit in the first integrated circuit package.

[0077] At 1004, a second integrated circuit package is configured, the second integrated circuit package including a second set of one or more artificial intelligence processing units and a second chip-to-chip interconnect communication unit. In some embodiments, the second integrated circuit package is Figure 1 In some embodiments, the second group of one or more artificial intelligence processing units is Figure 1 In some embodiments, the second chip-to-chip interconnect communication unit is Figure 1 The chip-to-chip interconnect communication unit 114 in the second integrated circuit package. In some embodiments, configuring the second integrated circuit package includes deploying a second set of one or more artificial intelligence processing units and a second chip-to-chip interconnect communication unit in the second integrated circuit package.

[0078] At 1006, an interconnection is established between the first integrated circuit package and the second integrated circuit package, wherein the first chip-to-chip interconnect communication unit and the second chip-to-chip interconnect communication unit manage Ethernet-based communications via the interconnection using a layered communication architecture that supports credit-based data flow control and retransmission data flow control. In some embodiments, the interconnection established is Figure 1 In some embodiments, establishing the interconnection includes physically connecting the first integrated circuit package with the second integrated circuit package (eg, via a wire interconnection).

[0079] Fig.11 1 is a block diagram illustrating an embodiment of a system of chips connected via point-to-point links. In the illustrated example, system 1100 includes integrated circuit packages 1102, 1104, 1106, and 1108. In some embodiments, each of these integrated circuit packages is Figure 1 The system 1100 shows a topology of point-to-point links between integrated circuit packages (eg, Figure 1 However, system 1100 includes four integrated circuit packages instead of two integrated circuit packages. This significantly increases the number of links in the point-to-point topology. Instead of requiring a single link ( Figure 1 Instead of interconnect 108 in the system 1100, six links (interconnects 1110, 1112, 1114, 1116, 1118, and 1120) are required to ensure point-to-point connections between each integrated circuit package. In some embodiments, each interconnect in the system 1100 is connected to Figure 1 The interconnect 108 is the same type of interconnect. Fig.11 It shows that the point-to-point topology (similar to Figure 1 The point-to-point topology shown) is still possible with more integrated circuit packages in the system, but the connection scheme is more complex.

[0080] Fig. 12A and Fig. 12B is a block diagram showing an embodiment of a system of chips connected via a unit having a switch. Fig. 12A In the embodiment, system 1200 includes integrated circuit package 1202 having a switch and integrated circuit packages 1204, 1206, and 1208. In some embodiments, each of integrated circuit packages 1204, 1206, and 1208 is Figure 1 The integrated circuit package 102 and / or 110 in Fig.11 The integrated circuit package with a switch 1202 is different from the integrated circuit packages 1204, 1206 and 1208 in that the integrated circuit package with a switch 1202 includes a hardware switch (see Fig.13 In other words, in various embodiments, the integrated circuit package 1202 with a switch includes switching and data routing functions in addition to computing (e.g., AI accelerator) functions. Fig.11 Compared to the interconnection scheme of system 1100 in FIG. 1 , the interconnection scheme of system 1200 may be simplified by utilizing switching and data routing functions. Fig.11 Both system 1100 and system 1200 include four integrated circuit packages (eg, four computing units), except that system 1200 requires fewer connections.

[0081] In the example shown, integrated circuit packages 1204, 1206, 1208 are connected to integrated circuit package 1202 with a switch (which acts as a hub) through interconnects 1210, 1212, and 1214, respectively. In some embodiments, each interconnect in system 1200 is connected to Figure 1 The interconnect 108 and / or Fig.11 The interconnects in system 1100 are the same type of interconnects. Figure 3 The layered communication architecture shown (including Figure 5 The credit-based data flow control shown in Figure 6 and Figure 7 1202) can also be supported by system 1200. For example, interconnects 1210, 1212, and 1214 maintain point-to-point credits between integrated circuit packages 1202 with switches and their directly connected peers. Credits between integrated circuit packages without switches can also be maintained through indirect links. For example, integrated circuit packages 1204 and 1206 are indirectly connected through integrated circuit package 1202 with switches. This connection is shown as conceptual link 1216. Conceptual links 1218 and 1220 are similar representations of indirect connections through integrated circuit package 1202 with switches. Credits and other data flows can still be exchanged. This exchange via a hub unit (in this case, integrated circuit package 1202 with switches) is called end-to-end (e.g., end-to-end credits) rather than point-to-point. In order to support end-to-end communications, in various embodiments, the switching hardware of integrated circuit package 1202 with switches maintains the address spaces of various components of system 1200 to route communications between the components.

[0082] Thus, using a switched topology, fewer physical connections (in this case, three physical connections for connecting four components in system 1200 instead of four physical connections) are used. Fig.11 It is possible to connect computing units (e.g., AI accelerators) to each other using six physical connections used to connect the same number of components in system 1100. Fig. 12B A more complex switching topology (with more components) is shown in system 1250. Fig. 12B In the example shown, end-to-end conceptual links are not drawn to clarify the example. Only direct physical links are drawn. System 1250 includes six computing components, not Fig. 12A 1200. These computing components are integrated circuit packages 1252 and 1258 with switches and integrated circuit packages 1254, 1256, 1260, and 1262. In some embodiments, each of integrated circuit packages 1254, 1256, 1260, and 1262 is Figure 1 The integrated circuit package 102 and / or 110 in Fig.11 The integrated circuit package of the system 1100 or Fig. 12A In some embodiments, the integrated circuit package 1252 and / or 1258 with the switch is Fig. 12A 1202 with a switch in the integrated circuit package 1200. Five direct physical links (interconnects 1264, 1266, 1268, 1270, and 1272) are shown in the system 1250. In some embodiments, each interconnect in the system 1250 is connected to Figure 1 The interconnect 108 and / or Fig.11 The interconnection of the system 1100 or Fig. 12A The same type of interconnects are used for the interconnects of system 1200. System 1250 is similar to Fig. 12A The difference of system 1200 in FIG. 1 is that some end-to-end connections need to be routed through two hub units. For example, the end-to-end connection between integrated circuit packages 1254 and 1262 needs to be routed through integrated circuit package 1252 with a switch and integrated circuit package 1258 with a switch. Including more components with switches can reduce the number of required direct links. Various other topologies with any number of components with or without switches are also possible. The examples shown are illustrative and non-restrictive.

[0083] Fig.13 1 is a block diagram illustrating an embodiment of an integrated circuit package including a hardware switch. In some embodiments, the integrated circuit package 1300 having a switch is Fig. 12A The integrated circuit package 1202 having a switch and / or Fig. 12B 1252 and / or 1258 of the integrated circuit package with switch. In the example shown, the integrated circuit package with switch 1300 includes one or more processing units 1302 and a switch communication unit 1304. In some embodiments, the one or more processing units 1302 are Figure 1 In various embodiments, the switch communication unit 1304 is configured to connect multiple devices in a network. In some embodiments, the switch communication unit 1304 includes an Ethernet switch (also called a network switch). In some embodiments, the Ethernet switch maintains an address table of unique identifiers corresponding to different devices in the network. The unique identifier can be a hardware address (e.g., a MAC address). In various embodiments, the switch communication unit 1304 also includes additional protocol logic, which includes an electronic circuit system configured to communicate using various communication protocols of the layered communication architecture. In some embodiments, the additional protocol logic of the switch communication unit 1304 includes Figure 2In various embodiments, the switch communication unit 1304 also includes a hardware buffer space configured to store data processed during various stages of receiving and transmitting across the various layers of the layered communication architecture. In some embodiments, the hardware buffer space includes Figure 2 The hardware buffer 206 in.

[0084] Although the foregoing embodiments have been described in some detail for purposes of clarity of understanding, the invention is not limited to the details provided. There are many alternative ways to implement the invention. The disclosed embodiments are illustrative rather than restrictive.

Claims

1. A system comprising: a first integrated circuit package, the first integrated circuit package comprising a first set of one or more artificial intelligence processing units and a first chip-to-chip interconnect communication unit; a second integrated circuit package including a second set of one or more artificial intelligence processing units and a second chip-to-chip interconnect communication unit; and The interconnection between the first integrated circuit package and the second integrated circuit package, wherein the first chip-to-chip interconnect communication unit and the second chip-to-chip interconnect communication unit use a layered communication architecture supporting credit-based data flow control and retransmission data flow control to manage Ethernet-based communication via the interconnection.

2. The system according to claim 1, wherein: The first group of one or more artificial intelligence processing units and the second group of one or more artificial intelligence processing units are configured to process an artificial intelligence workload, and the artificial intelligence workload includes training one or more neural networks.

3. The system according to claim 1 or 2, wherein: The first chip-to-chip interconnect communication unit and the second chip-to-chip interconnect communication unit include electronic circuit system logic configured to support a physical layer network communication protocol; and preferably, wherein the physical layer network communication protocol includes an Ethernet communication protocol.

4. A system according to any one of the preceding claims, wherein: The first chip-to-chip interconnect communication unit and the second chip-to-chip interconnect communication unit include electronic circuitry logic configured to support a plurality of communication protocols in addition to a physical layer network communication protocol.

5. A system according to any one of the preceding claims, wherein: The first chip-to-chip interconnect communication unit and the second chip-to-chip interconnect communication unit include hardware buffers configured to store communication data.

6. A system according to any one of the preceding claims, wherein: The interconnection comprises a physical interconnection between the first integrated circuit package and the second integrated circuit package; and preferably, wherein the physical interconnection comprises a wire interconnection.

7. A system according to any one of the preceding claims, wherein: The layered communication architecture comprises an application layer; and preferably, wherein the application layer is configured to access high bandwidth memory.

8. A system according to any one of the preceding claims, wherein: The layered communication architecture further comprises: Physical network layer; and / or Reliability layer; and / or Message distribution and reassembly layer.

9. A system according to any one of the preceding claims, wherein: The credit-based data flow control is configured to communicate a credit limit between the first integrated circuit package and the second integrated circuit package based on a receive buffer size of the first integrated circuit package and a receive buffer size of the second integrated circuit package.

10. A system according to any one of the preceding claims, wherein: The credit-based data flow control is configured to transmit and store a count of data packets exchanged between the first integrated circuit package and the second integrated circuit package.

11. A system according to any one of the preceding claims, wherein: The retransmission data flow control is configured to acknowledge receipt of an error-free data packet and to update a counter that records a consecutive number of error-free data packets received.

12. A system according to any one of the preceding claims, wherein: The retransmission data flow control is configured to communicate receipt of an erroneous data packet and cause retransmission of the erroneous data packet.

13. A method comprising: configuring a first integrated circuit package, the first integrated circuit package comprising a first set of one or more artificial intelligence processing units and a first chip-to-chip interconnect communication unit; configuring a second integrated circuit package, the second integrated circuit package comprising a second set of one or more artificial intelligence processing units and a second chip-to-chip interconnect communication unit; as well as An interconnection is established between the first integrated circuit package and the second integrated circuit package, wherein the first chip-to-chip interconnect communication unit and the second chip-to-chip interconnect communication unit manage Ethernet-based communication via the interconnection using a layered communication architecture supporting credit-based data flow control and retransmission data flow control.

14. A system comprising: a first integrated circuit package, the first integrated circuit package comprising a first set of one or more artificial intelligence processing units and a first chip-to-chip interconnect communication unit; a second integrated circuit package, the second integrated circuit package comprising a second set of one or more artificial intelligence processing units and a switch communication unit, wherein the switch communication unit is configured to provide an intermediate connection point between the first integrated circuit package and at least a third integrated circuit package; and An interconnection between the first integrated circuit package and the second integrated circuit package, wherein the first chip-to-chip interconnect communication unit and the switch communication unit manage Ethernet-based communication via the interconnection using a layered communication architecture supporting credit-based data flow control and retransmission data flow control.

15. The system of claim 14, wherein: The third integrated circuit package includes a third group of one or more artificial intelligence processing units and a second chip-to-chip interconnect communication unit.

Images