Ethernet packet processing in programmable integrated circuits

Abstract

Relates to Ethernet packet processing in programmable integrated circuits. A programmable integrated circuit (PIC) includes: an internal network-on-chip (iNOC) including iNOC rows and iNOC columns; an Ethernet controller coupled to the iNOC, a plurality of network access points (NAPs) in the iNOC, the plurality of NAPs including a first NAP to parse a header of a packet and a second NAP including a decryption engine, the first NAP including logic to: receive, by the first NAP, a segment of the packet; checking the segment to determine whether the packet needs to be decrypted; and in response to detecting that the packet needs to be decrypted, sending the segment of the packet to the second NAP via the iNOC.

Description

[0001] This application is a divisional application of the Chinese patent application filed on October 26, 2022, with application number 202180031170.4 and invention name “Ethernet Packet Processing in Programmable Integrated Circuits”. The international filing date of the parent application is February 17, 2021, the international application number is PCT / US2021 / 018382, and the priority date is February 27, 2020.

[0002] Related Applications

[0003] This application claims the benefit of priority to U.S. Provisional Patent Application Serial No. 62 / 982,223, filed on February 27, 2020, the contents of which are incorporated herein by reference in their entirety. Technical Field

[0004] The subject matter disclosed herein relates generally to methods, systems, and machine-readable storage media for providing on-chip networking within a field programmable gate array (FPGA). Background Art

[0005] FPGAs are increasingly used in applications that require high-speed data communications, such as machine learning, networking acceleration, and storage acceleration. FPGAs can include high-frequency and high-throughput interfaces, but due to the lower operating frequency of the user logic in the FPGA core, the internal functions of the FPGA cannot handle high throughput.

[0006] FPGAs are commonly used in networking applications, including packet switches and routers, deep packet inspection, compression, and encryption. Modern FPGAs support very high throughput Ethernet networking interfaces, such as 400Gbps Ethernet, to support these interfaces and data rates.

[0007] For traditional FPGAs, the networking IP cores supporting high-speed interfaces are located at the boundary of the FPGA core, and FPGA developers must develop logic that provides paths from the FPGA boundary to the functional units within the FPGA. The logic supporting networking applications is usually crowded around the network interface, causing serious congestion problems and making it difficult to meet timing requirements. Summary of the invention

[0008] According to one aspect of the present invention, a programmable integrated circuit (PIC) is provided, comprising: an internal network on chip (iNOC), comprising iNOC rows and iNOC columns; an Ethernet controller coupled to the iNOC, a plurality of network access points (NAPs) in the iNOC, the plurality of NAPs comprising a first NAP for parsing a header of a packet and a second NAP comprising a decryption engine, the first NAP comprising logic for: receiving a segment of the packet by the first NAP; checking the segment to determine whether the packet needs to be decrypted; and in response to detecting that the packet needs to be decrypted, sending the segment of the packet to the second NAP via the iNOC.

[0009] According to one aspect of the present invention, a method is provided, comprising: receiving a packet requiring decryption by an Ethernet controller at a programmable integrated circuit (PIC), the PIC further comprising an internal network on chip (iNOC) coupled to the Ethernet controller and a plurality of network access points (NAPs) in the iNOC, the iNOC comprising an iNOC row and an iNOC column, the plurality of NAPs comprising a first NAP for parsing a header of the packet and a second NAP comprising a decryption engine; and transmitting a segment of the packet to a plurality of NAPs including the first NAP and the second NAP via an interface of the iNOC and the Ethernet controller; checking a first segment by the first NAP to determine whether the packet requires decryption; and sending the first segment of the packet to the second NAP via the iNOC in response to detecting that decryption of the packet is required. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The figures in the drawings depict only example embodiments of the present disclosure and are not to be considered limiting of its scope.

[0011] Figure 1 The architecture of an FPGA with on-chip networking capabilities is shown according to some example embodiments.

[0012] Figure 2 An Ethernet connection to user logic is shown according to some example embodiments.

[0013] Figure 3 Broadcasting and multicasting within an internal network-on-chip (iNOC) are shown according to some example embodiments.

[0014] Figure 4 The problem of wasting bandwidth when Ethernet packets do not fill network-on-chip (NOC) packets is shown according to some example embodiments.

[0015] Figure 5Processing of an Ethernet packet at multiple network access points (NAPs) is shown according to some example embodiments.

[0016] Figure 6 How to use smaller segments to reduce wasted network bandwidth according to some example embodiments is shown.

[0017] Figure 7 Distribution of packets to multiple NAPs is shown according to some example embodiments.

[0018] Figure 8 The processing of Ethernet packets when an Ethernet controller operates in packet mode is shown according to some example embodiments.

[0019] Fig. 9 Buffering of Ethernet packets for packet mode is shown according to some example embodiments.

[0020] Fig.10 Buffering of packets transmitted in an outbound direction is shown according to some example embodiments.

[0021] Fig.11 The use of a configurable buffer level indicator for an outbound buffer is shown according to some example embodiments.

[0022] Fig.12 The use of a cut-through mode for forwarding a packet before the packet has fully arrived is shown according to some example embodiments.

[0023] Fig.13 is an architecture for supporting multiple Ethernet modes according to some example implementations.

[0024] Fig.14 400G Ethernet packet modes are shown according to some example embodiments.

[0025] Fig.15 The simultaneous use of two 200G Ethernet interfaces in packet mode is shown according to some example embodiments.

[0026] Fig.16 The simultaneous use of one 200G Ethernet interface and one 100G interface in packet mode is shown according to some example embodiments.

[0027] Fig.17 A 400G Ethernet mode in a Quad Segment Interface (QSI) mode is shown according to some example embodiments.

[0028] Fig.18 The simultaneous use of two 200G Ethernet interfaces in QSI mode according to some example embodiments is shown.

[0029] Fig.19 Egress processing of 400G Ethernet in packet mode is shown according to some example embodiments.

[0030] Fig. 20 The versatility of a NOC for processing Ethernet packets is illustrated according to some example embodiments.

[0031] Fig.21 The use of an end-of-packet marker for saving NOC bandwidth is shown according to some example embodiments.

[0032] Fig. 22 Efficient routing of packets using vertical and horizontal communications according to some example embodiments is shown.

[0033] Fig.23 Multifunctional routing of packets using multiple paths is shown according to some example embodiments.

[0034] Fig.24 is a flow chart of a method for processing Ethernet packets at an FPGA according to some example embodiments.

[0035] Fig.25 is a flow chart of a method for distributing Ethernet packets at an FPGA according to some example implementations.

[0036] Fig.26 is a block diagram illustrating an example of a machine on or by which one or more of the example process embodiments described herein may be implemented or controlled. DETAILED DESCRIPTION

[0037] Example methods, systems, and computer programs relate to processing Ethernet packets at an FPGA. The examples represent possible variations only. Unless otherwise expressly stated, components and functions are optional and may be combined or subdivided, and the order of operations may vary or operations may be combined or subdivided. In the following description, for purposes of illustration, many specific details are set forth to provide a thorough understanding of the example embodiments. However, it will be apparent to those skilled in the art that the subject matter may be practiced without these specific details.

[0038] To support fast network interfaces, such as 400Gbps Ethernet interfaces (also known as 400G Ethernet), the data paths inside the FPGA would have to be, for example, 1024 bits wide and operate at frequencies exceeding 720MHz. These performance goals are almost impossible to achieve in traditional FPGAs.

[0039] In one aspect, a method and system for a private network inside an FPGA core is provided to efficiently load balance Ethernet traffic between multiple access points in the FPGA architecture and deliver or receive Ethernet data from anywhere within the FPGA core without requiring the FPGA designer to implement data movement registers in the FPGA logic. In addition, the FPGA network interface rearranges packet data when providing it to the FPGA logic so that a typical data path of a common clock speed, such as a 256-bit wide data path operating at 506 MHz, can be utilized at the core.

[0040] The embodiments presented herein are presented for Ethernet networks of different speeds, such as 400G, 200G, and 100G, but the same principles can be applied to other data processing network interfaces and other networking speeds. For example, as technology improves and faster Ethernet modes become available, the same principles can be used to implement these faster modes.

[0041] Improved network processing capabilities at the FPGA include at least the following capabilities: sending Ethernet services to any row along the column of the internal FPGA network; supporting multiple network access point (NAP) arrangements for versatile development of FPGA designs; intra-column quad-segment interface (QSI) to reduce transmission frequency; packet mode to deliver Ethernet packets to a single NAP to reduce transmission frequency and simplify FPGA design; intra-column packet interleaving to reduce buffering requirements at each node; buffer fill level reporting; pass-through mode to start forwarding packets before a complete packet has been received; supporting a mix of 400G, 200G and 100G Ethernet services; multi-hop packet forwarding within the internal FPGA network; and packet compression using multicast technology.

[0042] One general aspect includes a programmable integrated circuit including: an internal network on chip (iNOC) including iNOC rows and iNOC columns; clusters coupled to the iNOC, each cluster including a network access point (NAP) and programmable logic; and an Ethernet controller coupled to the iNOC. The Ethernet controller can be configured to operate in a packet mode in which each complete inbound Ethernet packet is sent from the Ethernet controller via the iNOC to one of the NAPs, wherein two or more NAPs can be configured to receive the complete inbound Ethernet packet from the Ethernet controller.

[0043] One general aspect includes a programmable integrated circuit comprising: an iNOC including iNOC rows and iNOC columns; a plurality of clusters coupled to the iNOC, each cluster including a vertical network access point (NAP) for iNOC column communications, a horizontal NAP for iNOC row communications, an active signal, and programmable logic, wherein the vertical NAPs connect to the horizontal NAPs when the active signal is activated; and an Ethernet controller coupled to the iNOC, the Ethernet controller being configurable to send Ethernet packet segments to the vertical NAPs.

[0044] Figure 1 The architecture of an FPGA 102 with on-chip networking capabilities is shown according to some example embodiments. The FPGA 102 includes an external NOC (eNOC) 104 and an internal NOC (iNOC) 120, which provide high throughput and easy-to-use access between peripheral devices and an FPGA core 122. The combination of the iNOC 120 and the eNOC 104 is referred to herein as a NOC. The FPGA core 122 is surrounded by the eNOC 104 and includes clusters 118, which are programmable logic units that can be configured by a user.

[0045] In a typical FPGA, it can be difficult to move data quickly because the FPGA logic runs slower than a typical dedicated processor. NOC implements a network hierarchy that allows fast data movement and high throughput inside the FPGA.

[0046] There are multiple external interfaces, such as Ethernet controller 114, PCIe controller 116, and GDDR6 controller 112, that exchange data with the outside of FPGA 102. In traditional FPGAs, the external interfaces are directly connected to clusters 118 and the ability to transfer data is limited by the speed of clusters 118 and the network bandwidth available to send data to clusters 118. In addition, data exchange may require the configuration of a large number of clusters 118. For example, if the PCIe controller 116 wants to talk to the DDR-4 / 5 controller 124, the user must design the FPGA to deliver (carry) read and write requests in both directions throughout the architecture, which will require the use of a large amount of user logic that may not be used for other purposes. In addition, when compared to the speed supported by the external interface, the transfer will be slow.

[0047] In some example embodiments, the eNOC 104 and iNOC 120 support read / write transactions between master devices and slave devices. Master / slave is a communication model in which one device or process can control one or more other devices. In FPGA 102, the master device can initiate transactions, i.e., send requests, and the slave device waits for any master device to send requests. The eNOC 104 also supports pass-through and store-and-forward communications.

[0048] The master device includes at least the PCIe controller 116, the user-implemented master device in the FPGA core 122, and the FPGA control unit (FCU) (allowing bitstream commands to read and write peripheral control and status (CSR) interfaces). The slave device includes at least the GDDR6 controller 112, the DDR-4 / 5 controller 124, the user-implemented slave device in the FPGA core 122, the PCIe controller 116, the FCU (allowing the master device in the subsystem such as PCIe to configure the FPGA 102), and the CSR interface of the cluster 118 (e.g., including phase-locked loops (PLLs), inputs / outputs, top-level clocks / resets).

[0049] In some example implementations, the iNOC 120 is implemented in the FPGA core 122 as regularly spaced elements, and the iNOC 120 includes a plurality of columns and a plurality of rows. Figure 1 The example shown in includes seven columns and eight rows, but other implementations may use different numbers of columns and rows (eg, ranging from 1 to 50 or more, although other values ​​are possible).

[0050] In some example embodiments, each row includes fast data paths (eg, 512 Gbps, but other speeds are possible) in the eastbound and westbound directions, and each column includes fast data paths in the northbound and southbound directions.

[0051] iNOC 120 and eNOC 104 use NOC to transfer packets and provide a packet transfer interface to the module. In some example embodiments, the packet interface supports Advanced eXtensible Interface (AXI) read and write transactions. AXI is part of the Advanced Microcontroller Bus Architecture (AMBA), an open standard on-chip interconnect specification for connecting and managing functional blocks in system-on-chip (SoC) designs.

[0052] User logic in cluster 118 may issue AXI read or write transactions to a local network access point (NAP) in cluster 118. Based on routing logic, iNOC 120 delivers the transaction to eNOC 104 at the east or west boundary of FPGA core 122. In some example implementations, iNOC 120 transmits transactions issued by user FPGA logic to the east or west side, and transactions destined for FPGA user logic arrive from the north or south side, but other business models may also be supported.

[0053] In one embodiment with 512Gbps transmission on iNOC 120, each iNOC row can support 409Gbps ​​of write data throughput assuming 4-cycle bursts, since write commands are sent as the first cycle of each transaction. Larger size bursts will result in higher throughput, while smaller size bursts will result in lower throughput. Transactions on each row or column share the available bandwidth with other transactions on the same row or column.

[0054] Each column of the iNOC 120 operates similarly to a row, except that packet data is delivered in either a northbound or southbound direction. In some example implementations, the horizontal component of the iNOC 120 enables user logic in the cluster 118 to issue read / write transactions (e.g., AXI transactions), and the vertical component of the iNOC 120 enables IP components outside of the FPGA core 122 to issue transactions to the cluster 118. In other words, the FPGA master 106 is connected to the rows of the iNOC 120, and the FPGA slave 108 is connected to the columns of the iNOC 120.

[0055] iNOC 120 supports multiple service types, including transactional data and packet data streams. Transactional data includes read and write commands, data, and responses. Command transfers are typically single cycles, and data transfers are typically short, with 4-cycle transactions being common.

[0056] Traffic types include Cache Coherent Interconnect (CCIX) data for accelerators, packet data, and raw user data. In some example implementations, the iNOC 120 delivers CCIX data from the PCIe controller 116 when operating in CCIX mode. This is provided to the user as a data stream in the CSX protocol format, which is actually a segmented interface that the user unpacks.

[0057] The packet data may be bundled into longer data streams. In some example embodiments, the iNOC 120 imposes no upper limit on the packet size. In addition, with respect to raw user data transmission, the iNOC 120 transmits flits between NAPs in the same column or row, and the flits are combined to form packets.

[0058] In some example implementations, each row of the iNOC 120 provides a master device 106 to the eNOC 104 west of the FPGA core 122, and another master device 106 to the eNOC 104 east of the FPGA core 122. Each column of the iNOC 120 provides a slave device 108 to the iNOC 120 north of the FPGA core 122, and another slave device 108 to the iNOC 120 south of the FPGA core 122. This enables user logic to read or write to any external IP or CSR interface, and enables any external IP with a master interface to access any NAP with attached user logic.

[0059] In some example embodiments, the eNOC 104 is a ring around the FPGA cores 122 that delivers transactions from outside the eNOC 104 to the clusters 118, and from the clusters 118 to outside the eNOC 104, and between clusters 118. In some example embodiments, the eNOC 104 provides address decoding, transaction command and response routing, message width adaptation, frequency adaptation (e.g., clock domain crossing), burst adaptation, and protocol conversion.

[0060] The eNOC 104 delivers read and write transactions. In some example implementations, the eNOC 104 does not deliver Ethernet packets, SerDes data, or CCIX data, and does not deliver interrupt information. However, other implementations may transmit one or more of Ethernet packets, SerDes data, CCIX data, and deliver interrupt information.

[0061] The combination of the iNOC 120 and the eNOC 104 enables any access point in the FPGA 102 to have access to any interface IP slave interface including any of the GDDR6 controller 112 , DDR-4 / 5 controller 124 , and PCIe controller 116 interfaces.

[0062] In some example implementations, each memory interface presents a slave device to the NOC (eg, with a 256b data path width) and accepts read and write transactions. In addition, the PCIe master device 106 and the slave device 108 are directly connected to the eNOC 104 .

[0063] A NOC node (or simply referred to herein as a node) is a component of a network that connects to and communicates with other nodes. Each node includes a router and a NAP for each service direction. The NAP has a connection to local user logic in the cluster 118, and the router delivers packets (e.g., micro slices) to other routers in other nodes. Therefore, the user logic accesses the NOC by communicating with the NAP that interfaces with the router.

[0064] The routers make routing policies for microslices traveling on the network, and the NAPs "slow down" the transmission of data from the high-speed NOC network to the lower speed of the user logic in the cluster 118. The NAPs also format the data into the corresponding protocol, such as Ethernet or AXI. EW (East-West) routers are routers in the east-west direction, i.e., routers in the rows of the iNOC 120. NS (North-South) routers are routers in the north-south direction, i.e., routers in the columns of the iNOC 120.

[0065] CXS or CXS interface refers to the CCIX stream interface. Graphics Double Data Rate Memory (GDDR) or GDDR6 is a memory unit (inferred to be JEDEC Standard 6.0 unless otherwise specified). Double Data Rate Memory (DDR) or DDR4 is a memory unit (inferred to be JEDEC Standard 4.0 unless otherwise specified). A WAW microchip is a special kind of microchip that encapsulates the write data channel and the write address channel in the same microchip, allowing for more efficient data transfer. The Ethernet Interface Unit (EIU) is an interface that sends Ethernet data to the iNOC column. Additionally, the CCIX Interface Unit (CXIU) is an interface that sends CXS data to the iNOC column.

[0066] Each node has two sets of router and NAP combinations: EW routers and NAPs, and NS routers and NAPs. In addition, the routers at each node are independent of other routers and drive incoming microslices to the next node (next router) or drive incoming microslices to the corresponding local NAP. Each router decides whether to consume the microslice in the node or forward the microslice to the next node based on the destination ID field in the microslice. In some example embodiments, the microslice has 12 bits of transmission information and a 291-bit payload, but other formats are possible. The microslice payload is used to carry various types of data, and in the case of AXI transactions, the 291-bit payload carries AXI signals. In non-AXI microslices, microslices are configured to encapsulate raw data, Ethernet packets, or CXS streams. Multiple microslice formats are supported for different types of payloads.

[0067] In the case of AXI transactions, the EW router is attached to the EW NAP, which acts as an AXI slave to the user logic. The NS router is connected to the NS NAP, which acts as an AXI master on the user logic. Therefore, the EW direction generates requests and receives responses, while the NS direction receives commands and generates responses.

[0068] In case of raw data transfer, an EW NAP generates raw data flits which are received by another EW NAP. The transfer occurs through an EW iNOC. Similarly, a NS NAP generates raw data flits and is received by another NS NAP.

[0069] In the case of CXS microslices, NS NAP receives and generates CXS microslices, where each CXS microslice encapsulates a CXS flow and delivers it through the NS iNOC infrastructure to the CXIU, which interfaces with PCIe or CCIX.

[0070] In the case of Ethernet microslices, the NS NAP receives and generates Ethernet packets. Each Ethernet microslice fully or partially encapsulates Ethernet packets. The iNOC infrastructure delivers the data to the EIU, which handles the interface with the Ethernet MAC. In this way, the iNOC 120 supports both memory-mapped data transfer infrastructure and pure data stream transfer infrastructure.

[0071] Thus, the NOC provides a high-speed network running around the outside of the user logic using very fast links (e.g., 2 GHz) that are thousands of bits wide, resulting in a much lower latency and much higher throughput path to that memory without the user having to design any intermediate clusters 118 for communication. Thus, the FPGA may not be programmed and the PCIe controller 116 is able to talk to the DDR-4 / 5 controller 124 without any configuration. This allows for better use of the user logic for the user's programmable functions.

[0072] In some example embodiments, the NOC is a transaction network that supports both reads and writes. Reads can be requested from any address, and writes can be performed to any address.

[0073] To benefit from NOC functionality, users instantiate NAPs in their designs. To access the various IPs visible through the NOC, users configure the NAPs. In addition, to use the NOC to communicate between clusters 118 on FPGA 102, users instantiate NAP blocks at each node and use standard interfaces, such as AXI or packet interfaces, or even the user's own interfaces. In summary, the NOC delivers multiple types of services simultaneously.

[0074] For example, a user instantiates a 256-bit NAP in their design to gain access to the iNOC 120. Assuming that the user's design can meet the 600 MHz frequency, a single NAP provides 153.6 Gbps of read throughput and 153.6 Gbps of write throughput.

[0075] Figure 2 An Ethernet connection to user logic is shown according to some example embodiments. In some example embodiments, the Ethernet controller 114 supports throughputs up to 400 Gbps, but may also support other speeds, such as 200G and 100G.

[0076] In some example embodiments, Ethernet controller 114 is directly connected to one or more ranks of iNOC 120, which enables Ethernet controller 114 to issue packets directly to one of the iNOC ranks without having to go through eNOC 104. In other example embodiments, Ethernet packets may be transmitted through eNOC 104 and may be transmitted to any rank.

[0077] exist Figure 2In the example shown, an Ethernet packet is sent from interface 204 to router 206 at the top of the iNOC column, where router 206 supports the transmission of Ethernet packets. The iNOC column delivers the packet to the correct row, where it is delivered to router 208 configured for packet transmission. Router 208 then transmits the packet to user logic 202 via the local NAP. Reverse data transmission follows the same path in the opposite direction.

[0078] In some example embodiments, for Ethernet packet transmission, 288 bits are transmitted in each microchip. The 288 bits of data deliver the data to a single router 208 designated in the microchip. To interface with user logic in Ethernet mode, the router 208 reuses the AXI data signals as well as the 32-bit additional signals. The Ethernet data is provided to the user logic on the same 288-bit interface used by the CXS and the original data transmission. In addition, the Ethernet packets can be encapsulated in the microchip with additional information.

[0079] In some example embodiments, the NAP is capable of converting Ethernet formats and protocols to AXI formats and protocols. For example, the NAP may receive an Ethernet packet, convert the data to AXI, and store the converted data in a memory location. Thus, the iNOC column supports transmission of two different types of protocols, such as Ethernet and AXI.

[0080] Figure 3 Broadcast and multicast within the iNOC 120 are shown according to some example embodiments. In addition to supporting transactions, the iNOC columns deliver Ethernet data. Each iNOC Ethernet Interface Unit (EIU) (part of the node router) (e.g., 304 to 311) accepts one or more Ethernet traffic flows from an Ethernet MAC. The EIU tags the Ethernet data with the correct destination D based on the device configuration (e.g., the user maps the physical MAC to the destination node in the column), interleaves the data with data from other MACs, and then sends the packet down the column.

[0081] The EIU with a local address equal to the destination field in the iNOC column microchip captures the microchip, copies the microchip into its local buffer. The AXI encapsulation / decapsulation is bypassed, and the received microchip is provided to the user logic. The packet start, packet end, and packet end modulo indications, as well as other Ethernet control signals are delivered as part of the microchip payload. For outbound Ethernet packets (from user logic to Ethernet controller 114), the process is reversed: the user logic pushes the data into the local EIU, which appends the "North Terminal" destination ID and source ID at the top of the column and interleaves the data with other northbound microchips. The source ID indicates which Ethernet MAC destination.

[0082] In some example embodiments, when the Ethernet controller 114 operates at 400Gbps, the Ethernet controller 114 provides a 324-bit wide interface running at about 800MHz to the EIU. By delivering data in a four-segment interface or as packets streamed over time, the amount of data fits within the throughput envelope of a single iNOC column. However, the amount of data exceeds the amount of data that a single EIU can provide to the FPGA logic. To support 400Gbps, multi-NAP data provision is used. For example, four EIUs work together, each providing / receiving 256 bits of data to provide a combined data path of 324 bits running at about 500MHz to the client. In another example embodiment, four Ethernet frames are buffered separately and sent separately to separate EIUs, each of which is streamed in a 256-bit data path at about 500MHz.

[0083] In some example embodiments, to support multi-NAP data provisioning, the EIU takes the 324-bit interface from the 400Gbps MAC core and splits it into four parallel streams of 256 bits each. Each of these streams operates in the 2GHz domain, has a unique destination ID attached, and is interleaved with the other streams to be sent along the column where they are picked up by the corresponding EIU. In other embodiments, to support slower fabric speeds, Ethernet packets provided in this manner are quad-segmented so that any destination / source NAP 304 to 311 gets the beginning or end of the packet. This mode is called QSI mode, which adds complexity but allows the design to run at slower speeds.

[0084] When the Ethernet controller 114 operates at 200 Gbps, the Ethernet controller 114 provides a 324-bit wide interface running at approximately 400 MHz (or, running at approximately 800 MHz, where more than half of the cycles are idle). When converted to a QSI interface, this amount of data fits within the throughput envelope of a single iNOC column. However, this amount of data exceeds the amount of data that a single EIU can provide to the FPGA logic. Depending on the configuration, the 200 Gbps stream is provided as a QSI to four NAPs, or as a packet streaming interface to four NAPs or two NAPs.

[0085] For Ethernet operating at 30Gbps or less, the Ethernet controller 114 provides a 256-bit interface to the EIU. The EIU crosses the data to the 2GHz iNOC clock domain and into a small switching network for transmission to one of two destination columns. Additionally, each Quad MAC has a 30G interface that buffers packets.

[0086] As in other networks, communications in the NOC use addresses to identify nodes in the network. Therefore, different nodes in the NOC (such as Ethernet, PCIe, DDR, NAP) are assigned addresses, and the NOC is responsible for address decoding to determine where the request is going and sending the request to the right location.

[0087] In some example embodiments, each router includes an address translation table that allows the base address (also called the local address) of each endpoint (router / NAP or IP core) to be independent of its physical location. Thus, when a router receives a packet with one of the router's local addresses as the destination address, the router delivers the packet (or flits) to the local NAP.

[0088] The address translation table translates the destination address into the physical address of the elements in the NOC. Although the physical addresses are unique, the same destination address may be mapped to more than one node (e.g., NAP) that uses the destination address as a local address. Therefore, the address space of the NOC is fully configurable because the address translation table of the NAP is also configurable.

[0089] The address translation table in each NAP allows the destination location of each node or IP core to be looked up by address for each transaction issue. The address translation table allows the memory mapping viewed by each NAP to be independent of the memory mapping viewed by other NAPs. For example, this enables one accelerator (user logic) to view its private memory at a given base address, and enables a different accelerator to view its memory at the same base address, while the system has actually allocated memory independently for each accelerator.

[0090] The address translation table is also used to provide security so that if some portion of the user logic coupled to a given NAP cannot access some other NAP, IP core, FCU, or some other address space, then for that given NAP, the disabled portion of the address space will be marked as inaccessible in the address translation table.

[0091] The flexibility of the configurable address translation table enables multicast and broadcast addressing schemes, since each NAP supports multiple local addresses simultaneously. For example, broadcasting in a column can be implemented by assigning the same local address as one of the NAP addresses to each NAP in the column, so that if a microflake arrives at a NAP with a broadcast address, the NAP will deliver the microflake locally.

[0092] In existing FPGA broadcast implementations, a broadcast packet must be sent individually to each recipient of the broadcast, which means sending two or more packets instead of just one packet.

[0093] Similarly, a broadcast address may be assigned to all NAPs at the top of a column connected to the eNOC 104. The packet may then be broadcast in all columns. Furthermore, the broadcast may be implemented across a row of iNOCs 120 or across multiple rows of iNOCs 120.

[0094] Multicast is the same as broadcast, but only a subset of NAPs share a given local address. Figure 3 In the example shown, multiple NAPs in the same column are assigned the same local address 4, namely NAPs 304, 306, 307, and 310. Note that these NAPs may have additional addresses that are different from other NAPs.

[0095] When a packet 302 with destination address 4 is sent along the iNOC train, NAPs 304, 306, 307 and 310 will determine that the packet 302 is addressed to one of their local addresses and deliver the packet 302 to the local NAP. Thus, one data packet is used to multicast to multiple nodes on the network.

[0096] Note that broadcast and multicast can be implemented for multiple transport mechanisms, such as Ethernet and AXI. Furthermore, because the network address table is reconfigurable, as described above, broadcast and multicast addresses can be added, modified, or deleted while the FPGA is operating.

[0097] One example of use is in applications related to machine learning. In machine learning, matrix multiplication is often used, and weights are used to identify optimization functions for training and inference used in machine learning. The weights used for matrix multiplication can be used in multiple user logic areas; therefore, using multicast is a good way to transmit weight parameters to multiple user logic areas.

[0098] Figure 4 The problem of wasting bandwidth when Ethernet packets do not fill NOC packets according to some example embodiments is shown. When Ethernet packets are delivered on a wide interface, the beginning of the Ethernet packet is usually aligned with the bus data path. Otherwise, the packet processing logic becomes more complex in order to be able to analyze the packet (e.g., parse) at any location.

[0099] For example, if the FPGA has a 128-byte (1024-bit) wide interface, the start of the Ethernet packet is aligned with the 128-byte interface to send the Ethernet packet in 1024-bit segments.

[0100] However, if the Ethernet packets do not have a size that is a multiple of 128 bits, there will be a waste since the next packet will start being transmitted in the next cycle.

[0101] In the example shown, three packets are transmitted. In the worst case, the length of an Ethernet packet is 129 bytes, which means that a 128-byte NOC packet (also referred to herein as a segment) is sent, followed by a second NOC segment of only one byte. The remaining 127 bytes in the second segment are wasted.

[0102] This means that in this worst case scenario, almost half of the bandwidth on the NOC is wasted. To accommodate the wasted bandwidth, the FPGA needs to operate at a higher frequency (e.g., 732MHz), almost twice the frequency of the ideal scenario with no waste.

[0103] Figure 5 The processing of one Ethernet packet at multiple network access points (NAPs) is shown according to some example embodiments. In QSI mode in the inbound direction, the Ethernet controller 114 receives incoming data at a specific frequency in a path of a specific width and rearranges the incoming data into a different data path width and a different frequency for transmission to the iNOC column.

[0104] In the example shown, Ethernet data is delivered to four NAPSs, such as NAPs 304, 306, 308, and 310. Such data delivered to each NAP is referred to as a segment or iNOC packet to distinguish it from the incoming Ethernet packets delivered at the Ethernet controller 114. The Ethernet packet is divided into segments and sent directly to the NAPSs 304, 306, 308, and 310 via the iNOC train, bypassing the eNOC. The NAP 304 then passes the segment to the user logic 202 which will process the segment.

[0105] Since the Ethernet packet is sent to four different NAPs, the user logic 202 in the different NAPs must coordinate the processing of the Ethernet packet because the fragments of the Ethernet packet end up in four different places. Each NAP provides a small FIFO buffer and clock domain crossing because the iNOC 120 runs at 2 GHz and the architecture runs slower, such as 500 MHz.

[0106] Note that 400G Ethernet traffic (eg, 4 bits wide at 100 GHz) may be delivered in one column of iNOC 120 operating at 2 GHz. Note that the values ​​of frequency are approximate and other similar values ​​may be used.

[0107] To make it easier to process packets when changing clock speeds, packets are aligned on clock boundaries, as shown above. Figure 4 Otherwise, the user logic 202 would have to be much more complex to be able to handle the start of an Ethernet packet anywhere within a NOC packet.

[0108] In some example embodiments, a hardwired signal SOP (Start of Packet) is provided to the NAP (separate from the iNOC bus) to identify the segment that carries the start of an Ethernet packet.

[0109] Figure 6 How to use smaller segments to reduce wasted bandwidth according to some example embodiments. Figure 4 As shown in FIG. 1 , each portion of the Ethernet packet is divided into four segments, and each segment is delivered to a NAP in the same column, as shown in FIG. Figure 5 shown.

[0110] In QSI mode, Ethernet packets can be aligned with any of the four segments, so the user logic is configured to process the start of an Ethernet packet at any of the four NAPs. Therefore, the user core logic for processing Ethernet packets becomes more complex, but the overall throughput is improved.

[0111] Notice, Figure 6 The embodiments shown in are examples and do not describe every possible embodiment. Other embodiments may utilize different numbers of segments, such as 2, 3, 5, 8, 16, etc. Figure 6 The embodiments shown in should therefore not be construed as exclusive or limiting, but rather illustrative.

[0112] exist Figure 6 In the example shown, since Ethernet packets do not have to be multiples of the segment size, there may still be waste in a segment. However, the maximum amount of waste has been reduced from 127 bytes to 31 bytes (only 1 byte of data and 31 bytes of waste in the segment). Because there is less waste in QSI mode, the clock at the core can run slower, for example, at 500 MHz instead of 700 MHz.

[0113] return Figure 5 , processing of Ethernet packets is typically done by a parser that examines the Ethernet heather and determines processing, such as sending it to storage for later retrieval, handing it over to a TCP / IP engine, sending it to a video stream handler, sending it to a decryption engine, etc.

[0114] However, since the beginning of an Ethernet packet in QSI mode can arrive at any of the four NAPs, this means that four different parsers are required in the user logic 202. Therefore, network utilization has been improved, but processor logic has increased. In addition, by distributing the processing of Ethernet packets to multiple points, there is also a relief in congestion because not all packets have to go to the same parser.

[0115] In the outbound direction, the opposite segment distribution occurs, i.e., four NAPs coordinate to send Ethernet packets out of the FPGA. The NAP used for the outbound direction can be the same NAP as the NAP used for the inbound direction, or can be a different NAP in the same iNOC column or in a different iNOC column. If different NAPs are used for the inbound and outbound directions, congestion is further reduced because of the separate data paths.

[0116] Figure 7 The distribution of NOC packets to multiple NAPs is shown according to some example embodiments. In some example embodiments, the NAPs receiving one Ethernet packet may be located in different iNOC columns or any combination thereof. For example, two NAPs may be in one iNOC column and two other NAPs may be in a second column, two NAPs may be in one iNOC column and two other NAPs may be in two other columns, one NAP may be in one column and three other NAPs may be in a second column, and so on.

[0117] The designer of the FPGA has the flexibility to determine how to process Ethernet packets based on the application the FPGA is being used for and to build the logic to process Ethernet packets efficiently.

[0118] exist Figure 7 In the example shown, each Ethernet packet is sent to NAPs in four different columns: the first Ethernet packet is sent to NAPs 701 to 704, the second Ethernet packet is sent to NAPs 705 to 708, the third Ethernet packet is sent to NAPs 709 to 712, and the fourth Ethernet packet is sent to NAPs 713 to 716. By distributing the processing of Ethernet packets, the potential bandwidth for processing Ethernet packets is increased while keeping the operating clock of the FPGA low.

[0119] Figure 8 The processing of Ethernet packets when the Ethernet controller operates in packet mode according to some example embodiments is shown. The Ethernet controller 114 may be configured to send Ethernet packets in so-called “packet mode.” In packet mode, Ethernet packets are still segmented as in QSI mode, but all segments of an Ethernet packet are delivered to the same NAP.

[0120] In the example shown, the complete Ethernet packet 802 is delivered to NAP 304 , and other Ethernet packets are similarly delivered to other NAPs 306 , 308 , and 310 .

[0121] In this example, Ethernet packet 802 includes five segments that are sent in different cycles, which may be consecutive or non-consecutive. Ethernet controller 114 buffers these segments and can group the segments to different NAPs to send through the iNOC column during the same cycle. For example, during one cycle, Ethernet controller 114 sends four segments to corresponding NAPs 304, 306, 308, and 310. In the next cycle, Ethernet controller 114 sends segments to NAPs 304, 306, and 310 because the Ethernet packet for NAP 308 fits in one segment, and so on.

[0122] Note that other configurations may include fewer or more NAPs for processing Ethernet packets. For example, in one configuration, all Ethernet packets are sent to NAP 304.

[0123] Since the complete Ethernet packet is delivered to one NAP, packet processing is simplified. For example, if all incoming traffic is sent to NAP 304, only one Ethernet parser is needed in the user logic to determine where to send the Ethernet packet for processing.

[0124] Furthermore, note that the beginnings of Ethernet packets do not have to be aligned to be sent via one iNOC train. For example, in a given cycle, Ethernet controller 114 may send the first segment of a packet to one NAP, the third segment of another packet to another NAP, and so on.

[0125] Another benefit of packet mode is that because the user logic receives the entire packet, the user logic does not have to coordinate with other cores that receive other segments of the Ethernet packet. For example, if two NAPs are configured to receive Ethernet packets, each NAP may have its own parser that is independent of the other, making packet processing easier. In addition, each NAP can be associated with the same TCP / IP engine or a different TCP / IP engine; that is, each parser can forward packets to the same TCP / IP engine, or there can be separate and independent TCP / IP engines.

[0126] In QSI mode, segments are sent in the order they are received, so segments (and Ethernet packets) are delivered in the order they arrive. However, in packet mode, there is buffering by the Ethernet controller 114. Some packets may be small and some packets may be large, and because of the buffering, this may cause Ethernet packets to be delivered to the NAP out of order (e.g., a small Ethernet packet may arrive before a previously received large Ethernet packet).

[0127] In order to handle Ethernet packets delivered out of order, a sequence number is added to each Ethernet packet. In this way, user logic can process the Ethernet packets in the correct order based on the sequence number.

[0128] Fig. 9 Buffering for Ethernet packets in packet mode according to some example embodiments is shown. The maximum size of a standard Ethernet packet is 1522 bytes, but some new data center implementations support larger sizes, such as 9000 bytes. Therefore, the FPGA is able to handle these large Ethernet packets. Therefore, the FPGA Ethernet controller 114 can efficiently process not only small Ethernet packets, but also large Ethernet packets.

[0129] As discussed above, in some implementations, Ethernet packets arrive using four times the bandwidth of a single NAP, which is why fragments of Ethernet packets are sent to four different NAPs. However, Ethernet traffic often alternates between small Ethernet packets, medium Ethernet packets, and large Ethernet packets. Buffering can handle all packet sizes without affecting traffic throughput, for example, by avoiding bottlenecks or deadlocks.

[0130] The Ethernet controller 114 includes four buffers 902 through 905 that are associated with respective NAPs in packet mode inbound operations.When an Ethernet packet arrives at the Ethernet controller 114, the Ethernet controller 114 determines the destination NAP for the packet and distributes 908 the Ethernet packet to the corresponding buffers.

[0131] The Ethernet controller 114 arbitrates 906 the transmission of segments in the buffers 902 to 905 to the NAPs. For example, in one cycle, one segment is selected from each buffer 902 to 905 and then transmitted in parallel to the four NAP destinations.

[0132] Fig.10 Buffering of packets transmitted in the outbound direction is shown according to some example embodiments. In the outbound direction (e.g., transmit packet mode), buffers 1002 to 1005 are used to buffer segments sent from the NAP. The Ethernet controller 114 then forms Ethernet packets for transmission outside the FPGA.

[0133] One of the challenges of buffering is that the NAP does not have much room for buffering before sending to the Ethernet controller 114; therefore, most of the buffering occurs at the Ethernet controller 114. However, large Ethernet packets may interfere with the transmission of small Ethernet packets because large packets require a large number of cycles. This phenomenon is called head-of-line blocking.

[0134] If a stop signal is sent from the Ethernet controller 114 to the NAP, the in-flight segment still needs to reach the Ethernet controller 114. Because of the routing trip delay, 30 to 40 cycles of data may need to be buffered in the outbound direction for each NAP. However, since there are up to 64 NAPs, the amount of buffering is not practical for the FPGA core because too much buffering would be required.

[0135] In some example embodiments, fill level signals 1006 to 1009 enable predictive flow control. Fill level signals, such as fill level signals 1006 to 1009, may be used for each of buffers 1002 to 1005. The fill level signal is a hardware signal that is not sent through the iNOC, but as a separate signal available to the FPGA core. If the buffer is full to a predetermined level, the fill level signal is set to a first value (e.g., a voltage value greater than 0, a logical value of 1), and if the buffer is not full to a predetermined level, the fill level signal is set to a second value (e.g., a voltage level of 0, a logical value of 0). For example, the predetermined level may be 75%, or a value in the range of from 50% to 99%. The use of voltage signals is useful because logic does not have to address different clock domains in the presence of different clock domains. For example, if a message is transmitted to indicate that a buffer has reached a threshold, the Ethernet controller 114 will have to establish a communication protocol with the NAP that will handle different clock domains. This will take longer and use more resources.

[0136] In some example embodiments, the predetermined level may be configurable by a developer, such that the developer may configure a threshold for the Ethernet controller 114 to set the fill level signal to 1, for example, 80% full.

[0137] Thus, the NAP can check the corresponding fill level signal, and when the fill level signal is activated, the NAP stops sending to the Ethernet controller 114 until the fill level signal is deactivated. This allows the in-flight segment to arrive at the Ethernet controller 114 without finding a full buffer. This flow control ensures that Ethernet data is not dropped.

[0138] Fig.11 The use of configurable buffer level indicators for outbound buffers according to some example embodiments is shown. In some example embodiments, instead of providing one fill level for each buffer, multiple signals may be provided for different fill levels. For example, in one embodiment, four different fill levels (e.g., 50%, 66%, 90%, 95%) are provided.

[0139] Each of the fill level signals is a hardwired signal provided to the FPGA core as described above. In some example embodiments, each of the fill level thresholds may be configurable by the designer to better handle traffic based on the application (e.g., video streaming or file downloading). The fill levels are provided to the core and user logic may read these values ​​as needed.

[0140] Fig.12 The use of a cut-through mode for forwarding a packet before the packet has completely arrived is shown according to some example embodiments. Typically, network nodes use a store-and-forward mode in which packets are stored in a buffer as they are received; when a complete packet has been received, the packet is transmitted to the next network node.

[0141] However, the time to store and forward packets increases the latency of the network, that is, how long it takes for a packet to get from source to destination.

[0142] In some applications, latency is critical, such as for streaming video or audio. To improve the latency of processing outbound Ethernet packets at the FPGA, a pass-through mode is provided in which Ethernet packets can begin to be transmitted outside the Ethernet controller 114 before the Ethernet controller 114 receives the complete packet from the FPGA core.

[0143] The cut-through threshold is a percentage value associated with a portion of the Ethernet packets that the Ethernet controller has received (e.g., 66%), so that once the percentage value is received, the Ethernet controller can start forwarding Ethernet packets to the outside. It is assumed that the user believes that the portion of the Ethernet packet that has not yet been received will have arrived at the Ethernet controller 114 before it must be sent out. Of course, the danger is that if there is a problem, the data will not be available, but this is controlled by the user logic. The benefit is reduced latency, which is very valuable for certain streaming applications.

[0144] The pass-through threshold is configurable for each buffer, so that user logic can apply different thresholds for different applications. As with the fill level signal, in some example embodiments, the pass-through signal is provided as a hardwired signal with two different voltage levels: one for the case where the threshold has not been reached, and the other for the case where the threshold has been reached (e.g., positive voltage and ground, negative voltage and ground, positive voltage and negative voltage).

[0145] Fig.13The FPGA is configured to support different Ethernet modes, such as 400G, 200G, and 100G. In some example embodiments, there are three Ethernet media access controllers (MACs): Ethernet MAC 1301 for implementing 400G or 200G, Ethernet MAC 1302 for 200G, and Ethernet MAC 1303 for 100G.

[0146] The buffer 1306 is shared by the different Ethernet MACs 1301 to 1303 so that less memory is used than if each of the Ethernet MACs had its own buffer. In one embodiment, 36 kilobytes of memory are reserved for the buffer 1306, but other values ​​are possible. Note that the buffering is suitable for both QSI mode and packet mode. Therefore, the buffering scheme is very flexible because it can be suitable for multiple Ethernet MACs in both QSI mode and packet mode, which makes the FPGA very flexible in terms of Ethernet packet processing.

[0147] Each of the Ethernet MACs 1301 to 1303 is connected to a buffer 1306 via configurable wiring depending on the operating mode. The configurable wiring is referred to herein as a switching network including demultiplexers 1310 and 1312 and multiplexers 1314 and 1316. Multiplexer 1316 is connected to one or more iNOC columns. In one embodiment, the output of multiplexer 1316 is a 256-bit wide stream at 2GHz. The Ethernet controller (e.g., 114) will select data from one of the buffers 1306 by activating the corresponding input of multiplexer 1316 and send the data from the buffer to the column in iNOC 120.

[0148] Depending on the Ethernet operating mode, the connection on the switched network is opened or closed, as shown below. Figures 14 to 18 Described in more detail.

[0149] In some example implementations, there are five different Ethernet operating modes:

[0150] – One 400G or 200G interface in QSI mode;

[0151] – One 400G or 200G interface in packet mode;

[0152] – Two 200G interfaces in QSI mode;

[0153] – Two 200G interfaces in packet mode;

[0154] – One 200G interface and one 100G interface, both in packet mode.

[0155] In some example embodiments, a select register is used to configure the selected mode. For example, the modes identified above are configured with respective values ​​0, 1, 2, 3, and 4 in the select register, but other values ​​are possible. Although embodiments are given for the inbound direction, a similar structure is provided for outgoing communications.

[0156] Pointer control 1308 includes multiple pointers to manage buffer 1306 and data stored in buffer 1306. In one embodiment, the buffer is a circular FIFO (first in, first out) buffer and includes a queue head pointer and a queue tail pointer, also referred to as a read pointer and a write pointer. Pointer control 1308 operates differently based on the operating mode, as described below. For example, in some cases, each buffer 1306 is divided into two sub-buffers, resulting in 16 actual buffers instead of 8 actual buffers. The memory structure of buffer 1306 is the same, but pointer control 1308 changes so that each half buffer has its own pointer.

[0157] Fig.14 400G Ethernet packet mode is shown according to some example embodiments. Note that this configuration can also operate as a 200G interface instead of a 400G interface. The 400G MAC 1301 provides a 1024-bit wide stream 1402 at up to 800MHz. The 1024 bits are divided into four 256-bit segments coupled to corresponding demultiplexers 1310. For example, the four segments will include bits 0..255, 256..511, 512..767, and 768..1023, respectively, from the 1024 bits of the Ethernet packet transmitted in parallel.

[0158] In this configuration, each 256-bit segment is directed by demultiplexer 1310 to one of multiplexers 1314. The other input to multiplexer 1314 is inactive (eg, connected); therefore, the output of multiplexer 1314 is the same as the one active input.

[0159] In 400G packet mode, each buffer 1306 is logically divided into two buffers 1404 and 1405, also referred to herein as sub-buffers or half-buffers. Pointer control 1308 has a separate pointer for each of the sixteen buffers.

[0160] In packet mode, data is interleaved so that each Ethernet packet goes to one of the NAPs. Fig.14In the example, each buffer is associated with a NAP from 1 to 4, and each NAP is associated with 4 buffers. When an Ethernet packet arrives, the four 256-bit segments of each cycle are sent in parallel to the four buffers of the same NAP. For example, the Ethernet packet segment for NAP 1 will go to the buffer labeled 1, the segment for NAP 2 will go to buffer 2, the segment for NAP 3 will go to buffer 3, and the segment for NAP 4 will go to buffer 4. In order to fill the buffers efficiently, the fill level of the buffers is checked to select the least full or empty buffer to provide load balancing.

[0161] The path out of Ethernet MAC 1301 is the same for NAP 1 and NAP 3, but the pointer control 1308 will add the data to the top buffer or the bottom buffer depending on the receiving NAP.

[0162] On the way out of the buffer towards the iNOC 120 , the pointer control 1308 selects data from one of the buffers, the top buffer or the bottom buffer, and the Ethernet controller 114 activates the corresponding input of the multiplexer 1316 .

[0163] Fig.15 1 shows the simultaneous use of two 200G Ethernet interfaces in packet mode according to some example embodiments. The Ethernet MAC 1301 (referred to as 200GA) is set to operate at 200G rather than Fig.14 400G operation shown. Ethernet MAC 1302 (referred to as 200GB) provides a second 200G interface. Each 200G interface provides a 1024-bit wide stream at up to 400MHz (e.g., 300MHz).

[0164] The 200GB A segment streams pass through demultiplexers 1310 which connect their outputs to the odd buffers (1st, 3rd, 5th and 7th buffers). The streams from the 200GB are connected to multiplexers 1314 of the even buffers.

[0165] As with the 400G mode, the buffer is split into two buffers 1502 and 1504. Four NAPs are associated with Ethernet packets referred to as A1, A2, B1, and B2. A1 and A2 are associated with flows from 200G A, while B1 and B2 are associated with flows from 200G B.

[0166] Note that the buffers for 200G A and 200G B are independent of each other and can run on different clocks to synchronize with the corresponding MACs.

[0167] In 400G packet mode, there is one lane providing 1028 bits per cycle. For two 200G packet mode interfaces, the stream width of 1028 bits is the same, but the frequency is halved; therefore, the throughput is the same for one 400G and two 200G. On the way to the iNOC 120 column, the operation is the same as the reference Fig.14 The operations described are the same.

[0168] Fig.16 The simultaneous use of one 200G Ethernet interface and one 100G interface in packet mode according to some example embodiments is shown. In this mode, the Ethernet MAC 1302 for the 200G interface and the Ethernet MAC 1303 for the 100G interface are active.

[0169] Ethernet MAC 1302 (referred to as 200G B) provides 1024-bit wide streams at less than 400 MHz (e.g., 300 MHz), while Ethernet MAC 1303 (referred to as 100G A) provides 256-bit wide streams at 500 MHz or less.

[0170] As previously discussed, 200G B Ethernet sends four segments per cycle to the buffer in parallel. On the other hand, 100G A sends one 256 segment per cycle, which can be directed to one of eight buffers (four buffers 1602 labeled A1 and four buffers 1604 labeled A2). Therefore, a 1024-bit Ethernet packet over a 100G interface will be sent to a buffer within four clock cycles.

[0171] The wiring from Ethernet MAC 1302 connects each segment to one of the odd multiplexers 1314, and the output of Ethernet MAC 1303 passes through a demultiplexer 1312 having four outputs, each connected to one of the odd multiplexers 1314. Thus, each multiplexer 1314 has one active input and an output connected to a corresponding buffer.

[0172] As before, there are four NAPs active in grouping mode, labeled A1, A2, B1, and B2. The path to the iNOC 120 column is through multiplexer 1316 as in the previous mode.

[0173] Fig.17 400G Ethernet mode in QSI mode according to some example embodiments is shown. In QSI mode, a 1024-bit Ethernet packet received in one cycle is divided into four 256-bit segments, and each segment is sent to a different NAP.

[0174] In 400G QSI mode, four of the eight buffers 1702 are used and the other four buffers 1704 are not used. The incoming 1024 bits are sent to the four buffers 1702 via the demultiplexers 1310 and the multiplexers 1314. One of the outputs from each demultiplexer 1310 is connected to a corresponding multiplexer 1314. Each of the multiplexers 1314 of the active buffers 1702 connects one input to the output connected to the buffer 1702.

[0175] Fig.18 The simultaneous use of two 200G Ethernet interfaces in QSI mode according to some example embodiments is shown. The cabling of two 200G in QSI mode is similar to that of the earlier reference Fig.15 The wiring described for packet mode is the same. However, in QSI mode, buffer 1306 is not split into two buffers.

[0176] Ethernet MAC 1301 (200G A) uses the odd buffer, while Ethernet MAC 1302 (200G B) uses the right buffer. Pointer control 1308 treats each buffer as a FIFO queue with a write pointer and a read pointer.

[0177] When segments from buffer 1306 are sent to the iNOC 120 column, multiplexer 1316 simultaneously selects data from four buffers (A or B) and sends the data in parallel to four different NAPs, as described above with reference to FIG. Figure 5 Note that other combinations of different Ethernet QSI modes are also possible, such as multiple 100G, 200G and 100G combinations, etc.

[0178] Fig.19 Egress processing of 400G Ethernet in packet mode is shown according to some example embodiments. In the egress direction (transmit), there are eight buffers, and each buffer is associated with one of the eight NAPs.

[0179] Fig. 20 The versatility of a NOC for processing Ethernet packets is shown according to some example embodiments. Typically, a parser 2002 is used to inspect Ethernet packets and decide how to process the Ethernet packets, such as by sending the packets to a decryption engine 2004 for decryption or to a TCP / IP engine 2006 for network communication.

[0180] In QSI mode, the start of an Ethernet packet can be in one of four locations. In the example shown, the segment is sent to one of the NAPs 304 to 307 (labeled NAP 0 to NAP3). Therefore, four parsers 2002 are required, one for each NAP 304 to 307 (only one parser 2002 is shown). This increases the complexity of the user logic.

[0181] In some example embodiments, the FPGA provides great flexibility in routing data for processing. Segments received at one NAP can be quickly sent to another NAP, which can be in the same column or row. In some cases, segments can be sent without the need for store and forwarding segments simply by using the underlying AXI transport functions of the iNOC.

[0182] This routing flexibility enables sharing of blocks, such as compression and decryption engines. For example, in QSI mode, NAP 304 receives a segment containing the beginning of an Ethernet packet. Parser 2002 examines the header and determines that the packet must be decrypted and decides to send it down the list to NAP 308 connected to user logic with decryption engine 2004.

[0183] NAP 304 informs NAP 308 that a segment on the bus needs to be delivered to the user logic. Therefore, no store and forward is required because the multicast capability on the iNOC column enables the segment to be delivered to NAP 4. Similarly, other NAPs can send Ethernet packets to be decrypted to the decryption engine 2004. Therefore, the decryption engine 2004 may be shared by multiple NAPs.

[0184] In some example embodiments, the parser 2002 detects a TCP / IP header in the Ethernet packet. Using the iNOC's AXI transport, the parser 2002 decides to forward the segment down the column to the TCP / IP engine 2006 in the same manner as described above for the decryption case, but since the TCP / IP engine 2006 only requires the header of the Ethernet packet, the parser 2002 sends the body of the Ethernet packet (also referred to as the payload) to the memory, such as the GDDR6 controller 112, via the iNOC 120 and the eNOC 104.

[0185] The user logic may determine how to process the TCP / IP packet and then pass the packet to another user logic block, such as an IP router, by passing the Ethernet header and memory information to store the payload.

[0186] Sending to memory can also be used by the decryption engine 2004. In this way, the decryption engine 2004 is not flooded with all the segments to be decrypted, but the decryption engine 2004 processes the Ethernet packets sequentially, requesting a new packet to be decrypted from the memory when the decryption engine 2004 is free.

[0187] The same principle can be used in other scenarios. For example, the parser 2002 can perform a deep inspection of the Ethernet packet, beyond the TCP / IP header into the application portion of the packet to determine whether the packet corresponds to a video stream (e.g., streaming a movie to a user). In this case, another processor can be used to stream the video, and the parser 2002 will send the Ethernet packets to the streaming video processor instead of sending them to the TCP / IP engine 2006.

[0188] Fig.21 The use of an end-of-packet marker for saving NOC bandwidth according to some example embodiments is shown. In some example embodiments, customized signaling is used to improve bandwidth utilization. This method is referred to herein as a compression method because it improves bandwidth utilization even though no actual compression of the segments is performed to reduce the number of bytes that actually carry information.

[0189] In some example embodiments, rather than using QSI to segment data, the beginning of an Ethernet packet (e.g., 2104) is always sent to one NAP (e.g., NAP 1) in parallel with data sent to the other three NAPs (e.g., NAPs 2 to 4). Figure 4 As described, this scenario means that there can be a lot of wasted bandwidth, and as a result the FPGA core must run at a high clock (724 Hz) to keep up with 400G Ethernet.

[0190] The last segment indicator, also known as the end of packet (EOP), is associated with the last segment in an Ethernet packet, and is available to all NAPs. In some example embodiments, segment 5 is multicast to NAPs 1 through 4. Thus, if the last segment arrives at NAP 1, no encoding is required: there are three empty segments going to NAPs 2 through 4 in the following cycles. Therefore, the bandwidth that would have been allocated to NAPs 2 through 4 in these cycles can be used for other communication purposes on the iNOC column.

[0191] Furthermore, a value called MOD is shared among NAPs 1 to 4. MOD is an integer indicating how many bytes of data are included in the last segment (for example, 4 bytes are used, which means that 28 bytes are not used).

[0192] Fig. 22Efficient routing of packets using vertical and horizontal communications is shown according to some example embodiments.In some example embodiments, the iNOC's horizontal NAPs 304 (for communication along rows) are collocated with the vertical NAPs 2202 (for communication along columns).

[0193] Horizontal NAP 304 and vertical NAP 2202 are collocated because they are physically close to each other so that communications from rows can be quickly routed to columns and vice versa. In some example embodiments, gate 2208 connects the line from vertical NAP 2202 to horizontal NAP 304, and gate 2208 is controlled by a signal called valid bit, so that when the valid bit is set to logic 1, the two lines are connected together. Gate 2208 can be an AND logic gate whose inputs are the valid bit and the vertical line and whose output is the horizontal line. One gate 2208 is shown, but additional gates are also arranged for the remaining lines in vertical NAP 2202, and the gates of all lines share the valid bit signal as input.

[0194] Because the horizontal and vertical NAPs are juxtaposed, sending data from columns to rows is fast and efficient, and the logic takes up a very small amount of space in the FPGA core.

[0195] In addition, in some example embodiments, the vertical NAP 2202 lines and the horizontal NAP 304 lines are staggered; that is, the horizontal lines are alternately arranged between the vertical lines in the same block. Because the lines are staggered, the redirection of data from one direction to another is very fast, and the logic takes up a very small amount of space on the FPGA.

[0196] As mentioned above Fig. 20 As discussed, the parser 2002 analyzes the Ethernet header and sends the payload of the Ethernet packet to the external memory (e.g., the GDDR6 controller 112). Therefore, a simple and fast way to send the payload to the memory is to simply redirect the Ethernet packet payload along the iNOC column to the iNOC row by setting the valid bit and passing the incoming segment to the eNOC 104 and the memory GDDR6 controller 112.

[0197] For example, the parser 2002 looks at the incoming header and decides to send the payload to memory for later processing (e.g., when the FPGA acts as a network router, the TCP / IP packet is routed to the next destination). After receiving a few bytes at the beginning of the packet (e.g., the header plus 0 bytes or more), the parser 2002 quickly determines in real time that the payload will be sent to memory.

[0198] The user logic then activates the valid bit to copy the Ethernet packet bits from the vertical NAP 2202 to the horizontal NAP 304. When data is incoming, the data is sent to memory without having to store the complete Ethernet packet.

[0199] An address generator 2210 in the user logic generates a memory address where the data is to be stored; this address is then used to route the data to the memory (eg, the GDDR6 controller 112).

[0200] iNOC 120 provides great flexibility for user logic to quickly route data from one location to another in the FPGA. The design of user logic is simpler because designers do not need to plan complex data storage and forwarding within the FPGA.

[0201] In the outbound direction, there is usually an egress block 2218 of user logic that sends the Ethernet packet to the outside of the FPGA. In some cases, the Ethernet packet payload is in memory, so the egress block 2218 has the header information 2212 and address information 2214 of the Ethernet packet, where the address information 2214 is the address in the external memory that holds all or part of the Ethernet packet. In some cases, the entire packet is stored in memory, so the header information 2212 is not needed.

[0202] As in the inbound direction, vertical NAPs 2206 and horizontal NAPs 2204 are juxtaposed and interleaved. Valid bits 2216 are used to connect the lines of horizontal NAPs 2204 to the lines of vertical NAPs 2206.

[0203] The egress block 2218 then issues a read request (e.g., an AXI command) to get the Ethernet data from memory, and the Ethernet data arrives via the iNOC rows. As the data arrives, the data is immediately copied to the iNOC columns via the vertical NAP 2206. The data then goes up through the columns to the eNOC 104 and then to the Ethernet controller 114.

[0204] If the header information 2212 is not in memory, the egress block 2218 appends the header information 2212 and then sends the header information 2212 and the payload from memory to the eNOC 104 .

[0205] In some example embodiments, the payload of the Ethernet packet is striped across multiple external memories (e.g., GDDR6 controller 112), which means that the data is stored in several memories. For example, a first block of data is stored in a first memory, a second block is stored in a second memory, and so on.

[0206] When obtaining Ethernet data, the egress block 2218 issues multiple reads for different memories. In this way, because the access to the memory is load balanced, the possibility of deadlock or bottleneck is reduced.

[0207] Fig.23 Versatile routing of packets using multiple paths is shown in accordance with some example embodiments. When an Ethernet packet arrives at NAP 2302, user logic can instruct NAP 2302 to forward westward (e.g., NAP 2304) or eastward (e.g., NAP 2306), and in a vertical direction, typically downward (e.g., NAP 2308), although NAP 2302 can also send data upward in a different cycle.

[0208] In some example embodiments, the transport in the iNOC uses AXI, which has microchip-to-microchip transmission capabilities. In this way, data can be sent from one end of the FPGA to the other. In the traditional approach, network traffic at the FPGA is processed in one location, which causes congestion in that area. However, with the current approach, the processing of network traffic can be distributed throughout the FPGA. For example, Fig. 22 The egress block 2218 is away from the NAP 304 which parses the Ethernet header.

[0209] Additionally, the encryption engine may be located at the user logic coupled to the NAP 2306, rather than having to send data down the iNOC columns, and the TCP / IP engine may be located at the user logic coupled to the NAP 2306. Additionally, other NAPs (e.g., NAP 2308) may send Ethernet packets to the decryption engine coupled to the NAP 2306 by sending the data on a different row and then up another column. For example, the NAP 2308 may send the encrypted packet via NAP 2310 which forwards the encrypted packet to the NAP 2306.

[0210] And all of these services are possible without having to store and forward packets, which greatly improves efficiency and processing speed.The cycle in which forwarding occurs can be the same cycle in which data arrives (eg, from Ethernet controller 114).

[0211] In some example embodiments, the iNOC column includes a valid signal, a ready signal, and a destination signal. The ready signal indicates when the vertical iNOC is ready to accept data. The destination signal includes the address of the NAP as the destination of the data. In each cycle, the valid signal and the destination signal are specified to route the data to its destination.

[0212] Each NAP can be configured for streaming mode, which means forwarding data to another NAP when receiving data. For example, by setting the valid signal to active, the data is redirected to another NAP in the iNOC row. The NAP can also be configured as an AXI node, which means that the data will be transmitted through the iNOC using micro slices.

[0213] In addition, the flexible network capabilities of iNOC also help simplify the processing of Ethernet packets in packet mode. In fact, packet mode can be very useful in some cases. Since all the data for an Ethernet packet goes to the same NAP, the NAP can parse the header and then determine where to send the header and payload. When additional segments of the Ethernet packet arrive at the NAP with the parser, these segments are then quickly streamed to other NAPs or storage. The encrypted packets will be sent to the decryption engine one segment at a time, so the decryption engine is not overwhelmed by too much data.

[0214] Fig.24 is a flow chart of a method 2400 for processing Ethernet packets at an FPGA according to some example embodiments. At operation 2402, a programmable integrated circuit receives a configuration for the programmable integrated circuit. The programmable integrated circuit includes: an internal network on chip (iNOC) including iNOC rows and iNOC columns; a plurality of clusters coupled to the iNOC, each cluster including a network access point (NAP) and programmable logic; and an Ethernet controller coupled to the iNOC. The Ethernet controller can be configured to operate in a packet mode in which each complete inbound Ethernet packet is sent from the Ethernet controller via the iNOC to one of the NAPs, and two or more NAPs can be configured to receive the complete inbound Ethernet packet from the Ethernet controller. The method 2400 also includes an operation 2404 for providing Ethernet communications for the plurality of clusters via the Ethernet controller.

[0215] In one example, the Ethernet controller is configured to: receive an inbound Ethernet packet using a first data path having a width of a first number of bits; segment the received inbound Ethernet packet into segments having a second number of bits; and send each segment to one NAP via the iNOC.

[0216] In one example, the Ethernet controller includes a plurality of buffers for buffering inbound Ethernet packets before sending segments of the inbound Ethernet packets to the NAP via the iNOC.

[0217] In one example, an Ethernet controller includes a pointer control module having a plurality of pointers for controlling storage of data into a plurality of buffers.

[0218] In one example, each buffer is reserved for communication with one NAP.

[0219] In one example, each buffer is divided into two sub-buffers, and each sub-buffer is reserved for communication with one NAP.

[0220] In one example, the Ethernet controller includes a plurality of outbound buffers for buffering outbound Ethernet packets received from the NAP via the iNOC.

[0221] In one example, the Ethernet controller provides a fill level signal for each outbound buffer to the plurality of clusters, the fill level signal being activated when a predetermined amount of the buffer is filled.

[0222] In one example, the Ethernet controller provides a plurality of buffer level reached signals for each outbound buffer to the plurality of clusters, each buffer level reached signal being activated when a corresponding predetermined amount of the buffer is filled.

[0223] In one example, the Ethernet controller can be configured to detect activation of a cut-through threshold level for one of the outbound buffers, wherein the Ethernet controller starts transmitting an Ethernet packet in the outbound buffer when the cut-through threshold level is activated even if a complete Ethernet packet is not available in the outbound buffer.

[0224] In one example, the Ethernet controller is a 400G Ethernet controller.

[0225] In one example, the programmable integrated circuit includes an additional Ethernet controller, wherein the Ethernet controller and the additional Ethernet controller share a buffer for inbound Ethernet packets.

[0226] Fig.25 is a flow chart of a method 2500 for distributing Ethernet packets at an FPGA according to some example implementations. The method 2500 includes an operation 2502 for receiving, at a programmable integrated circuit, a configuration for a programmable integrated circuit, the programmable integrated circuit including: an internal network on chip (iNOC) including iNOC rows and iNOC columns; a plurality of clusters coupled to the iNOC, each cluster including a vertical network access point (NAP) for iNOC column communications, a horizontal NAP for iNOC row communications, an active signal, and programmable logic, wherein the vertical NAP is connected to the horizontal NAP when the active signal is activated; and an Ethernet controller coupled to the iNOC, the Ethernet controller being configurable to send Ethernet packet segments to the vertical NAP.

[0227] The method 2500 also includes an operation 2504 for providing Ethernet communications for the plurality of clusters via an Ethernet controller.

[0228] In one example, the lines connecting the vertical NAPs to the iNOC columns are staggered with the lines connecting the horizontal NAPs to the iNOC rows.

[0229] In one example, a first cluster includes a first vertical NAP and a parser in programmable logic that analyzes a header of an inbound Ethernet packet to determine processing for the inbound Ethernet packet.

[0230] In one example, the parser sends a header of a TCP / IP Ethernet packet to the second NAP by adding the address of the second NAP as a destination of a portion of the Ethernet packet, wherein the parser sends a payload of the TCP / IP Ethernet packet to the memory via an iNOC line.

[0231] In one example, a third NAP in a third cluster is used to send an outbound Ethernet packet, wherein the third cluster retrieves a payload of the outbound Ethernet packet from a memory and sends the outbound packet to an Ethernet controller via an iNOC column.

[0232] In one example, the parser redirects the inbound Ethernet packet to a decryption engine in a fourth NAP in a fourth core via an iNOC column.

[0233] In one example, the parser redirects the inbound Ethernet packet to a decryption engine in a fifth NAP in a fifth core via an iNOC line by activating an active signal.

[0234] In one example, the Ethernet controller can be configured to operate in a quad segment interface (QSI) to: receive an inbound Ethernet packet using a first data path having a width of a first number of bits; segment the received inbound Ethernet packet into four segments having a second number of bits; and send each segment to one of four NAPs in one of the iNOC columns.

[0235] In one example, four NAPs share a decryption engine.

[0236] In one example, four NAPs share a MOD value that indicates the end of an inbound Ethernet packet.

[0237] Fig.262600, on which or by which one or more example process implementations described herein may be implemented or controlled. In alternative embodiments, machine 2600 may operate as a standalone device or may be connected (e.g., networked) to other machines. In a networked deployment, machine 2600 may operate in a server-client network environment with the capabilities of a server machine, a client machine, or both. In an example, machine 2600 may act as a peer machine in a peer-to-peer (P2P) (or other distributed) network environment. In addition, although only a single machine 2600 is shown, the term "machine" should also be deemed to include any collection of machines that individually or jointly execute a set (or multiple sets) of instructions to perform any one or more of the methods discussed herein, such as via cloud computing, software as a service (SaaS) or other computer cluster configurations.

[0238] As described herein, examples may include logic, multiple components or mechanisms, or may be operated by logic, multiple components or mechanisms. A circuit system is a collection of circuits implemented in a tangible entity including hardware (e.g., simple circuits, gates, logic). Circuit system components may be flexible over time and potential hardware variability. A circuit system includes components that can perform specific operations individually or in combination when operating. In an example, the hardware of the circuit system may be designed to perform specific operations (e.g., hardwired) invariably. In an example, the hardware of the circuit system may include variably connected physical components (e.g., execution units, transistors, simple circuits) to encode instructions for specific operations, and the variably connected physical components include a computer-readable medium that is physically modified (e.g., magnetically, electrically, by the movable placement of constant mass particles). When the physical components are connected, the basic electrical properties of the hardware components are changed (e.g., from an insulator to a conductor, or from a conductor to an insulator). Instructions enable embedded hardware (e.g., execution units or loading mechanisms) to create components of the circuit system in hardware via variable connections to perform a part of a specific operation when operating. Thus, when the device operates, the computer-readable medium is communicatively coupled to other components of the circuit system. In examples, any physical component can be used in more than one component of more than one circuit system. For example, in operation, an execution unit can be used in a first circuit in a first circuit system at one point in time and reused by a second circuit in the first circuit system or by a third circuit in the second circuit system at a different time.

[0239] The machine (e.g., computer system) 2600 may include a hardware processor 2602 (e.g., a central processing unit (CPU), a hardware processor core, or any combination thereof), a graphics processing unit (GPU) 2603, a main memory 2604, and a static memory 2606, some or all of which may communicate with each other via an interconnect (e.g., a bus) 2608. The machine 2600 may also include a display device 2610, an alphanumeric input device 2612 (e.g., a keyboard), and a user interface (UI) navigation device 2614 (e.g., a mouse). In an example, the display device 2610, the alphanumeric input device 2612, and the UI navigation device 2614 may be a touch screen display. The machine 2600 may additionally include a mass storage device (e.g., a drive unit) 2616, a signal generating device 2618 (e.g., a speaker), a network interface device 2620, and one or more sensors 2621 such as a global positioning system (GPS) sensor, a compass, an accelerometer, or other sensors. The machine 2600 may include an output controller 2628, such as a serial (e.g., Universal Serial Bus (USB)), parallel, or other wired or wireless (e.g., infrared (IR), near field communication (NFC)) connection to communicate with or control one or more peripheral devices (e.g., printer, card reader).

[0240] The mass storage device 2616 may include a machine-readable medium 2622 on which is stored one or more sets of data structures or instructions 2624 (e.g., software) that implement or are utilized by any one or more of the techniques or functions described herein. The instructions 2624 may also reside, completely or at least partially, within the main memory 2604, within the static memory 2606, within the hardware processor 2602, or within the GPU 2603 during execution by the machine 2600. In an example, one or any combination of the hardware processor 2602, the GPU 2603, the main memory 2604, the static memory 2606, or the mass storage device 2616 may constitute a machine-readable medium.

[0241] Although the machine-readable medium 2622 is illustrated as a single medium, the term “machine-readable medium” may include a single medium or multiple media (eg, a centralized or distributed database, and / or associated caches and servers) configured to store one or more instructions 2624 .

[0242] The term "machine-readable medium" may include any medium capable of storing, encoding, or carrying instructions 2624 that are executed by the machine 2600 and cause the machine 2600 to perform any one or more of the techniques of the present disclosure, or may include any medium capable of storing, encoding, or carrying data structures used by such instructions 2624 or data structures associated with such instructions 2624. Non-limiting machine-readable medium examples may include solid-state memory and optical and magnetic media. In an example, a large-capacity machine-readable medium includes a machine-readable medium 2622 having a plurality of particles, the particles having a constant (e.g., stationary) mass. Therefore, the large-capacity machine-readable medium is not a transient propagation signal. Specific examples of large-capacity machine-readable media may include: non-volatile memory, such as semiconductor memory devices (e.g., electrically programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM)) and flash memory devices; magnetic disks, such as internal hard disks and removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks.

[0243] The instructions 2624 may also be transmitted or received over the communication network 2626 via the network interface device 2620 using a transmission medium.

[0244] Throughout the specification, multiple instances can be implemented as parts, operations or structures described as single instances. Although the individual operations of one or more methods are shown and described as separate operations, one or more of the individual operations can be performed simultaneously and do not require the operations to be performed in the order shown. The structure and function presented as separate parts in the example configuration can be implemented as a combined structure or parts. Similarly, the structure and function presented as a single part can be implemented as a separate part. These and other variations, modifications, additions and improvements fall within the scope of the subject matter herein.

[0245] The present invention can also be implemented through the following embodiments.

[0246] Embodiment 1. A programmable integrated circuit comprising:

[0247] an internal network-on-chip (iNOC), which includes iNOC rows and iNOC columns;

[0248] a plurality of clusters coupled to the iNOC, each cluster comprising a network access point (NAP) and programmable logic; and

[0249] an Ethernet controller coupled to the iNOC, the Ethernet controller being configurable to operate in a packet mode in which each complete inbound Ethernet packet is sent from the Ethernet controller via the iNOC to one of the NAPs, wherein two or more NAPs are configurable to receive the complete inbound Ethernet packet from the Ethernet controller.

[0250] Embodiment 2. According to the programmable integrated circuit of embodiment 1, the Ethernet controller is configured to:

[0251] receiving the inbound Ethernet packet using a first data path having a width of a first bit number;

[0252] segmenting a received inbound Ethernet packet into segments having a second number of bits; and

[0253] Each segment is sent via the iNOC to one of the NAPs.

[0254] Embodiment 3. The programmable integrated circuit of embodiment 1, wherein the Ethernet controller includes a plurality of buffers for buffering the inbound Ethernet packets before sending the segments of the inbound Ethernet packets to the NAP via the iNOC.

[0255] Embodiment 4. A programmable integrated circuit according to embodiment 3, wherein the Ethernet controller includes a pointer control module, and the pointer control module has multiple pointers for controlling the storage of data into the multiple buffers.

[0256] Embodiment 5. The programmable integrated circuit of embodiment 3, wherein each buffer is reserved for communication with one of the NAPs.

[0257] Embodiment 6. The programmable integrated circuit of embodiment 3, wherein each buffer is divided into two sub-buffers, and each sub-buffer is reserved for communication with one of the NAPs.

[0258] Embodiment 7. The programmable integrated circuit of Embodiment 1, wherein the Ethernet controller comprises a plurality of outbound buffers for buffering outbound Ethernet packets received from the NAP via the iNOC.

[0259] Embodiment 8. A programmable integrated circuit according to Embodiment 7, wherein the Ethernet controller provides a fill level signal for each outbound buffer to the plurality of clusters, the fill level signal being activated when a predetermined amount of the outbound buffer is filled.

[0260] Embodiment 9. A programmable integrated circuit according to embodiment 7, wherein the Ethernet controller provides multiple buffer level reached signals for each outbound buffer to the multiple clusters, and each buffer level reached signal is activated when a corresponding predetermined amount of the outbound buffer is filled.

[0261] Embodiment 10. A programmable integrated circuit according to Embodiment 7, wherein the Ethernet controller is configurable to detect activation of a pass-through threshold level for one of the outbound buffers, wherein the Ethernet controller starts transmitting an outbound Ethernet packet in the outbound buffer when the pass-through threshold level is activated, even if the outbound Ethernet packet is not fully available in the outbound buffer.

[0262] Embodiment 11. A programmable integrated circuit according to embodiment 1, wherein the Ethernet controller is a 400G Ethernet controller.

[0263] Embodiment 12. The programmable integrated circuit of embodiment 1 further comprising:

[0264] A further Ethernet controller, wherein the Ethernet controller and the further Ethernet controller share a buffer for inbound Ethernet packets.

[0265] Embodiment 13. A programmable integrated circuit according to Embodiment 12, wherein the Ethernet controller and the additional Ethernet controller support 200G Ethernet.

[0266] Embodiment 14. A programmable integrated circuit according to Embodiment 12, wherein the Ethernet controller supports 200G Ethernet and the additional Ethernet controller supports 100G Ethernet.

[0267] Embodiment 15. A method comprising:

[0268] A configuration for a programmable integrated circuit is received at the programmable integrated circuit, the programmable integrated circuit comprising:

[0269] an internal network-on-chip (iNOC), which includes iNOC rows and iNOC columns;

[0270] a plurality of clusters coupled to the iNOC, each cluster comprising a network access point (NAP) and programmable logic; and

[0271] an Ethernet controller coupled to the iNOC, the Ethernet controller being configurable to operate in a packet mode in which each complete inbound Ethernet packet is sent from the Ethernet controller via the iNOC to one of the NAPs, wherein two or more NAPs are configurable to receive the complete inbound Ethernet packet from the Ethernet controller; and

[0272] Ethernet communications are provided for the plurality of clusters via the Ethernet controller.

[0273] Embodiment 16. The method according to embodiment 15 further comprises:

[0274] receiving, by the Ethernet controller, the inbound Ethernet packet using a first data path having a width of a first bit number;

[0275] segmenting, by the Ethernet controller, received inbound Ethernet packets into segments having a second number of bits; and

[0276] Each segment is sent from the Ethernet controller via the iNOC to one of the NAPs.

[0277] Embodiment 17. The method of Embodiment 15, wherein the Ethernet controller includes a plurality of buffers for buffering the inbound Ethernet packets before sending the segments of the inbound Ethernet packets to the NAP via the iNOC.

[0278] Embodiment 18. A method according to embodiment 17, wherein the Ethernet controller includes a pointer control module, and the pointer control module has multiple pointers for controlling the storage of data into the multiple buffers.

[0279] Embodiment 19. The method of embodiment 15, wherein the Ethernet controller includes a plurality of outbound buffers for buffering outbound Ethernet packets received from the NAP via the iNOC.

[0280] Embodiment 20. A method according to embodiment 19, wherein the Ethernet controller provides a fill level signal for each outbound buffer to the multiple clusters, and the fill level signal is activated when a predetermined amount of the outbound buffer is filled.

[0281] The embodiments shown herein are described in sufficient detail to enable those skilled in the art to practice the disclosed teachings. Other embodiments may be used and derived therefrom, so that structural and logical substitutions and changes may be made without departing from the scope of the present disclosure. Therefore, the specific embodiments should not be considered to have a limiting meaning, and the scope of the various embodiments is limited only by the appended claims and the full scope of equivalents to which these claims are assigned.

[0282] As used herein, the term "or" may be interpreted as inclusive or exclusive meanings. In addition, multiple instances may be provided for resources, operations, or structures described as single instances herein. In addition, the boundaries between various resources, operations, modules, engines, and data stores are arbitrary to some extent, and specific operations are shown in the context of specific illustrative configurations. Other allocations of functions are foreseen and may fall within the scope of various embodiments of the present disclosure. Typically, structures and functions presented as separate resources in the example configuration may be implemented as combined structures or resources. Similarly, structures and functions presented as single resources may be implemented as separate resources. These and other variations, modifications, additions, and improvements fall within the scope of the embodiments of the present disclosure as represented by the appended claims. Therefore, the specification and the accompanying drawings should be considered illustrative rather than restrictive.

Claims

1. A programmable integrated circuit (PIC), comprising: an internal network-on-chip (iNOC), which includes iNOC rows and iNOC columns; an Ethernet controller coupled to the iNOC, A plurality of network access points (NAPs) in the iNOC, the plurality of NAPs comprising a first NAP for parsing a header of a packet and a second NAP comprising a decryption engine, the first NAP comprising logic for: receiving, by the first NAP, segments of the packet; examining the segment to determine whether decryption of the packet is required; as well as In response to detecting a need to decrypt the packet, the segment of the packet is sent to the second NAP via the iNOC.

2. The PIC according to claim 1, wherein: Each NAP can be configured to forward segments to other NAPs in the same iNOC column by using the iNOC without storing and forwarding the segments.

3. The PIC according to claim 1, wherein: The first NAP and the second NAP are located in a same column in the iNOC or in a same row in the iNOC.

4. The PIC according to claim 1, wherein: The segments of the packet are distributed across a plurality of NAPs, wherein the first NAP is configured to receive a first segment of the packet.

5. The PIC according to claim 1, wherein: The decryption engine in the second NAP is shared by multiple NAPs.

6. The PIC according to claim 1, wherein: The logic of the first NAP further includes: The body of the packet is sent to the Ethernet controller.

7. The PIC according to claim 1, wherein: The logic of the first NAP further includes: receiving, by the first NAP, segments of a new packet; determining that the new packet is a TCP / IP (Transmission Network Protocol / Internet Protocol) packet; and The segments of the new packet are sent to a third NAP configured to process TCP / IP packets.

8. The PIC according to claim 1, wherein: The logic of the second NAP includes: storing the packets to be decrypted in the Ethernet controller; and The packets requiring decryption are processed by downloading each packet from the Ethernet controller.

9. The PIC according to claim 1, wherein: Each NAP includes at least one local address that is different from the local addresses of other NAPs.

10. The PIC according to claim 1, wherein: The NAP in the iNOC can also be configured to provide multicast and broadcast features for Advanced Extensible Interface (AXI) communications.

11. A method comprising: receiving a packet requiring decryption by an Ethernet controller at a programmable integrated circuit (PIC), the PIC further comprising an internal network on chip (iNOC) coupled to the Ethernet controller and a plurality of network access points (NAPs) in the iNOC, the iNOC comprising iNOC rows and iNOC columns, the plurality of NAPs comprising a first NAP for parsing a header of the packet and a second NAP comprising a decryption engine; as well as transmitting the segments of the packet to a plurality of NAPs including the first NAP and the second NAP via an interface of the iNOC and the Ethernet controller; examining, by the first NAP, the first segment to determine whether the packet requires decryption; as well as In response to detecting a need to decrypt the packet, a first segment of the packet is sent to the second NAP via the iNOC.

12. The method according to claim 11, wherein: Each NAP can be configured to forward segments to other NAPs in the same iNOC column by using the iNOC without storing and forwarding the segments.

13. The method according to claim 11, wherein: The first NAP and the second NAP are located in a same column in the iNOC or in a same row in the iNOC.

14. The method according to claim 13, wherein: The segments of the packet are distributed across the plurality of NAPs, wherein the first NAP is configured to receive a first segment of the packet.

15. The method according to claim 11, wherein: The decryption engine in the second NAP is shared by multiple NAPs.

16. The method according to claim 11, further comprising: The body of the packet is sent by the first NAP to the Ethernet controller.

17. The method according to claim 11, further comprising: receiving, by the first NAP, segments of a new packet; Determining that the new packet is a TCP / IP (Transmission Network Protocol / Internet Protocol) packet; as well as The segments of the new packet are sent to a third NAP configured to process the TCP / IP packet.

18. The method according to claim 11, further comprising: Storing the packets to be decrypted in the Ethernet controller; as well as The packets requiring decryption are processed by the second NAP by downloading each packet from the Ethernet controller.

19. The method according to claim 11, wherein: Each NAP includes at least one local address that is different from the local addresses of other NAPs.

20. The method according to claim 11, wherein: The NAP can also be configured to provide multicast and broadcast features for Advanced Extensible Interface (AXI) communications.

Images