Optional path ordering in packet-based networks

Abstract

A system can include a plurality of electronic devices and a plurality of hardware transceivers. The plurality of electronic devices can include a plurality of chiplets and can be coupled to each other via an interface network. The plurality of hardware transceivers can each be included in or coupled to a respective electronic device of the plurality of electronic devices. The plurality of hardware transceivers can each be configured to receive a data packet from a source device. The data packet can each include a protocol field specifying ordering information for delivery to a destination device and a path field specifying path information for routing the delivery to the destination device. The source device and the destination device can each include a chiplet. The plurality of hardware transceivers can each be further configured to transmit the received data packet to the destination device using at least the ordering information of each received data packet.

Description

[0001] Priority application

[0002] This application claims priority to U.S. Application No. 17 / 007,224, filed August 31, 2020, which is incorporated herein by reference in its entirety. Background Technology

[0003] Chiplets are an emerging technology for integrating various processing functionalities. Generally, a chiplet system consists of discrete modules (each referred to as a "chiplet") integrated on an interposer layer and, in many instances, interconnected via one or more established networks as needed to provide the required functionality to the system. The interposer layer and the contained chiplets may be packaged together for easy interconnection with other components of a larger system. Each chiplet may contain one or more individual integrated circuits or "chips" (ICs), which may be combined with discrete circuit components and coupled together to a corresponding substrate for attachment to the interposer layer. Most or all of the chiplets in the system will be individually configured to communicate via one or more established networks.

[0004] Chipsets, configured as individual modules within a system, differ from a system implemented on a single chip containing different device blocks (e.g., intellectual property (IP) blocks) on a substrate (e.g., a single die), such as a system-on-a-chip (SoC) or multiple discrete packaged devices integrated on a printed circuit board (PCB). Generally, chiplets offer better performance (e.g., lower power consumption, reduced latency, etc.) than discrete packaged devices, and offer greater manufacturing benefits than a single die chip. These manufacturing benefits may include higher yields or reduced development costs and time.

[0005] A chiplet system may comprise, for example, one or more application (or processor) chiplets and one or more support chiplets. Here, the distinction between application chiplets and support chiplets is merely a reference to possible design scenarios for chiplet systems. Thus, for example, a synthetic vision chiplet system may comprise (by example only) application chiplets for generating synthetic vision output, and support chiplets such as memory controller chiplets, sensor interface chiplets, or communication chiplets. In typical use cases, synthetic vision designers may design the application chiplets and obtain support chiplets from other sources. Therefore, design expenditures (e.g., in terms of time or complexity) are reduced by avoiding the design and manufacture of the functionality embodied in the support chiplets. Chiplets also support the tight integration of IP blocks that might otherwise be difficult, such as IP blocks manufactured using different processing technologies or with different feature sizes (or utilizing different contact technologies or pitches). Therefore, assemblies or IC assemblies of multiple ICs with different physical, electrical, or communication characteristics can be assembled in a modular manner to provide an assembly that implements the desired functionality. Chiplet systems also facilitate adaptation to the needs of different larger systems that will be incorporated into the chiplet system. In examples, ICs or other assemblies can be optimized for power, speed, or heat generation for specific functions, as may be the case with sensors. Compared to attempting integration with other devices on a single die, these ICs or other assemblies can be integrated more easily. Furthermore, by reducing the overall size of the die, the yield of small chips is often higher than that of more complex single-die devices. Attached Figure Description

[0006] This disclosure will be more fully understood from the detailed description given below and the accompanying drawings of various embodiments thereof. However, the drawings should not be construed as limiting this disclosure to the specific embodiments, but are for explanation and understanding only.

[0007] Figure 1A and 1B An example of a chiplet system according to an embodiment is shown.

[0008] Figure 2 Components of an example of a memory controller chiplet according to an embodiment are shown.

[0009] Figure 3 Examples of routing between electronic devices (e.g., chiplets in a chiplet layout using a chiplet protocol interface (CPI) network) in a device layout using an interface network according to an embodiment are shown.

[0010] Figure 4 This is a block diagram of data packets applicable to a system that allows optional path ordering in packet transmission, according to some embodiments of the present disclosure.

[0011] Figure 5Based on some embodiments of this disclosure Figure 4 A flowchart illustrating specific instances of data grouping.

[0012] Figure 6 This is a flowchart illustrating the operation of a method for optional path ordering for packet transmission according to some embodiments of the present disclosure.

[0013] Figure 7 This is a flowchart illustrating the operation of a method for selective path ordering in packet transmission using protocol field values ​​contained in the packets being transmitted, according to some embodiments of this disclosure.

[0014] Figure 8 This is a block diagram of an example computer system in which embodiments of this disclosure may be operated. Detailed Implementation

[0015] The following description Figure 1A and 1B Examples of chiplet systems and components operating within them are provided. In the context of such chiplet systems, packet transmission can be controlled using information contained in each packet, indicating one or more requirements for delivering the packet from a source endpoint (e.g., a chiplet) to a destination endpoint (e.g., another chiplet). Packets containing various types of information used in the operation of the chiplet system are transmitted within the system. Some use cases may not require data packets to be delivered in any particular order. Other use cases may require ordered delivery from the source endpoint to the destination endpoint for proper system operation. Some use cases may require ordered delivery of data packets sharing common characteristics (e.g., public addresses or public transaction identifiers). Some use cases may not require packets to be delivered via a specific path. Other use cases may require packets to be delivered via a specific path. For example, a system that forces all packets to be ordered for delivery via a predetermined path may satisfy all use cases, but may result in a degraded overall packet transmission performance in the system. For example, packets requiring ordered delivery from the source to the destination may be delivered in various orders and / or via different paths to avoid congestion. Therefore, a system is needed that supports all the use cases discussed above to provide packet transmission with high flexibility and high performance for a variety of applications, such as… Figure 1A and 1B Small chip system.

[0016] The subject of this invention uses an information control system (e.g., contained in each transmitted packet) Figure 1A and 1BIn a chiplet system, packet transmission occurs. Each transmitted packet may contain two fields specifying whether ordered delivery is required and whether a specific delivery path is needed. These two fields may include a first field containing order information (e.g., called the protocol field) and a second field containing path information (e.g., called the path field). In one example, the protocol field indicates whether out-of-order delivery is used (e.g., allowing delivery in random order), whether ordered delivery is used via a single predetermined path (e.g., designed into the system), or whether ordered delivery is used via a path specified by the path field.

[0017] Using order and path fields in data grouping provides flexible system support for multiple sorting requirements without degrading the performance of use cases that do not require sorting all data groups. Because sorting information is provided in each group, rather than setting the sorting scheme to configuration mode at system startup, groups with different sorting requirements can be transmitted simultaneously within the system.

[0018] Figure 1A and 1B An example of a chiplet system 110 according to an embodiment is shown. Figure 1A This is an illustration of a chiplet system 110 mounted on a peripheral board 105, which can be connected to a wider range of computer systems via, for example, peripheral component interconnect high-speed (PCIe). The chiplet system 110 includes a package substrate 115, an interposer 120, and four chips: an application chiplet 125, a host interface chiplet 135, a memory controller chiplet 140, and a memory device chiplet 150. Other systems may include numerous additional chipsets to provide additional functionality, as will become apparent from the discussion below. The package of the chiplet system 110 is shown with a cover or shroud 165, but other packaging techniques and structures can be used for the chiplet system. Figure 1B This is a block diagram for the purpose of clearly labeling the components in a chiplet system.

[0019] Application chip 125 is shown to include an on-chip network (NOC) 130 for supporting chiplet network 155 for inter-chiplet communication. In an example embodiment, NOC 130 may be included on application chip 125. In an example, NOC 130 may be defined in response to selected supporting chipsets (e.g., chipsets 135, 140, and 150), thereby allowing the designer to select an appropriate number of chiplet network connections or switches for NOC 130. In an example, NOC 130 may reside on a separate chiplet or even within interposer 120. In the example discussed herein, NOC 130 implements a chiplet protocol interface (CPI) network.

[0020] CPI is a packet-based network that supports virtual channels to enable flexible and high-speed interaction between chiplets. CPI bridges the chiplet-internal network to chiplet network 155. For example, the Advanced Scalable Interface (AXI) is a widely used specification for designing intra-chip communication. However, the AXI specification covers a large number of physical design options, such as the number of physical channels, signal timing, and power. Within a single chip, these options are typically selected to meet design goals such as power consumption and speed. However, to achieve flexibility in chiplet systems, adapters such as CPI are used to intersect between various AXI design options that can be implemented in various chiplets. By implementing a mapping from physical channels to virtual channels and encapsulating time-based signaling using packetization protocols, CPI bridges the intra-chiplet network 155.

[0021] CPI can utilize various physical layers to transmit packets. A physical layer may contain simple conductive connections, or it may contain drivers to increase voltage, or otherwise facilitate signal transmission over longer distances. An example of such a physical layer may include an Advanced Interface Bus (AIB), which in various instances may be implemented in intermediate layer 120. The AIB uses source-synchronous data transfer with a forwarding clock to transmit and receive data. Packets are transmitted across the AIB at Single Data Rate (SDR) or Double Data Rate (DDR) relative to the transmitted clock. The AIB supports various channel widths. When operating in SDR mode, the AIB channel width is a multiple of 20 bits (20, 40, 60…), and for DDR mode, the AIB channel width is a multiple of 40 bits (40, 80, 120…). The AIB channel width includes both transmitted and received signals. Channels can be configured with a symmetrical number of transmit (TX) and receive (RX) inputs / outputs (I / O), or with an asymmetrical number of transmitters and receivers (e.g., all transmitters or all receivers). The channel can act as either the AIB master or slave depending on which chip provides the master clock. The AIB I / O unit supports three clock modes: asynchronous (i.e., non-timed), SDR, and DDR. In various instances, the non-timed mode is used for clocking and some control signals. SDR mode can use a dedicated SDR-only I / O unit or a dual-purpose SDR / DDR I / O unit.

[0022] In this example, CPI packet protocols (e.g., point-to-point or routable) can use symmetrical receive and transmit I / O units within an AIB channel. CPI streaming protocols allow for more flexible utilization of AIB I / O units. In this example, streaming-mode AIB channels can configure I / O units as all TX, all RX, or half TX and half RX. CPI packet protocols can use AIB channels in SDR or DDR operating modes. In one example, AIB channels are configured in increments of 80 I / O units (i.e., 40 TX and 40 RX) for SDR mode and in increments of 40 I / O units for DDR mode. CPI streaming protocols can use AIB channels in either SDR or DDR operating modes. Here, in this example, AIB channels are configured in increments of 40 I / O units for both SDR and DDR modes. In this example, a unique interface identifier is assigned to each AIB channel. This identifier is used during CPI reset and initialization to determine paired AIB channels across neighboring chiplets. In this example, the interface identifier is a 20-bit value comprising a seven-bit chiplet identifier, a seven-bit column identifier, and a six-bit link identifier. The AIB physical layer uses an AIB out-of-band shift register to transmit the interface identifier. Bits 32 to 51 of the shift register are used to transmit the 20-bit interface identifier in both directions across the AIB interface.

[0023] AIB defines a stacked group of AIB channels as an AIB channel column. An AIB channel column has a certain number of AIB channels, plus auxiliary channels. The auxiliary channels contain signals used for AIB initialization. All AIB channels within a column (except for the auxiliary channels) have the same configuration (e.g., all TX, all RX, or half TX and half RX, and the same number of data I / O signals). In this example, AIB channels are numbered sequentially in ascending order, starting with the AIB channel adjacent to the AUX channel. The AIB channel adjacent to the AUX channel is defined as AIB channel zero.

[0024] AIB channels are typically configured as half TX data and half RX data, all TX data, or all RX data plus associated clock and promiscuous control. In some example implementations, the number of TX data signals relative to the RX data signals is determined at design time and cannot be configured as part of system initialization.

[0025] The CPI packet protocol (point-to-point and routable) uses symmetrical receive and transmit I / O units within the AIB channel. The CPI streaming protocol allows for more flexible use of the AIB I / O units. In some example implementations, the I / O units for the streaming mode of the AIB channel can be configured as all TX, all RX, or half TX and half RX.

[0026] Typically, the CPI interface on an individual chiplet may include serialization-deserialization (SERDES) hardware. SERDES interconnects are well-suited for scenarios requiring high-speed signaling and low signal counts. However, SERDES can introduce additional power consumption and longer latency for multiplexing and demultiplexing, error detection or correction (e.g., using block-level cyclic redundancy check (CRC)), link-level retries, or forward error correction. However, when low latency or power consumption is a primary concern for ultra-short-distance chiplet-to-chiplet interconnects, parallel interfaces that allow data transfer with minimal latency can be utilized. CPIs contain elements designed to minimize both latency and power consumption in these ultra-short-distance chiplet interconnects.

[0027] For flow control, CPI employs a credit-based technique. For example, the sender side of the application chip 125, such as the memory controller chip 140, provides credits indicating available buffers. In this example, the CPI receiver contains buffers for each virtual channel for a given transmission time unit. Therefore, if the CPI receiver supports five messages and a single virtual channel in time, the receiver has five buffers arranged into five entries (e.g., one entry per unit time). If four virtual channels are supported, the receiver has twenty buffers arranged into five entries. Each buffer holds the payload of one CPI packet.

[0028] When a sender transmits data to a receiver, the sender decrements its available credits based on the transmission. Once the receiver has used up all its credits, the sender stops sending packets to the receiver. This ensures that the receiver always has an available buffer to store transmissions.

[0029] When the receiver processes the received packet and releases the buffer, it communicates the available buffer space back to the sender. This credit value, indicating the available buffer space, can then be returned by the sender to allow the transmission of additional information.

[0030] The paper also describes a chiplet mesh network 160 that uses direct chiplet-to-chiplet technology without requiring a NOC130. The chiplet mesh network 160 can be implemented in a CPI or another chiplet-to-chiplet protocol. The chiplet mesh network 160 typically implements a chiplet pipeline, where one chiplet acts as an interface to the pipeline, while other chipslets in the pipeline interface only interface with themselves.

[0031] In addition, dedicated device interfaces can be used to interconnect chiplets, such as one or more industry-standard memory interfaces 145 (e.g., synchronous memory interfaces, such as DDR5, DDR6). This allows for the connection of a chiplet system or individual chiplets to external devices (e.g., larger systems can be connected via a desired interface (e.g., a PCIe interface)). In an example, an external interface may be implemented via a host interface chiplet 135, which, in the depicted example, provides a PCIe interface external to the chiplet system 110. Such dedicated interfaces 145 are commonly used when industry practice or standards have converged on them. The illustrated example of connecting a memory controller chiplet 140 to the DDR interface 145 of a dynamic random access memory (DRAM) device 150 exemplifies this industry practice.

[0032] Among the various possible supporting chiplets, the memory controller chiplet 140 is likely to be present in the chiplet system 110 due to the ubiquitous use of storage for computer processing and the application of advanced technologies to memory devices. Therefore, using the memory device chiplet 150 and the memory controller chiplet 140, both designed by other designers, allows chiplet system designers to obtain robust products manufactured by established companies. Typically, the memory controller chiplet 140 provides a memory device-specific interface for reading, writing, or erasing data. The memory controller chiplet 140 often provides additional features such as error detection, error correction, maintenance operations, or atomic operation execution. For some types of memory, maintenance operations are often specific to the memory device 150, such as garbage collection in NAND flash or storage-class memory, or temperature regulation (e.g., cross-temperature management) in NAND flash memory. In instances, maintenance operations may involve logic-to-physical (L2P) mapping or management to provide an indirection hierarchy between the physical and logical representations of data. In other types of memory, such as DRAM, some memory operations, such as refresh, may be controlled at certain times by the host processor or memory controller, and at other times by the DRAM memory device or logic associated with one or more DRAM devices, such as interface chips (in this example, buffers).

[0033] Atomic operations are data manipulations that can be performed, for example, by memory controller chiplet 140. In other chiplet systems, atomic operations can be performed by other chipsets. For example, an atomic operation can be specified as an "increment" in a command by application chiplet 125, the command containing a memory address and possibly an increment value. Upon receiving the command, memory controller chiplet 140 retrieves a number from the specified memory address, increments the number by the amount specified in the command, and stores the result. Upon successful completion, memory controller chiplet 140 provides application chiplet 125 with an indication that the command was successful. Atomic operations avoid data transfer across chiplet network 160, thereby reducing latency in executing such commands.

[0034] Atomic operations can be categorized into built-in atoms or programmable (e.g., custom) atoms. Built-in atoms are a finite set of operations that are implemented immutably in the hardware. Programmable atoms are small programs that can run on programmable atom units (PAUs) (e.g., custom atom units (CAUs)) of the memory controller chiplet 140. Figure 1A and 1B An example of a memory controller chip for discussing PAU is shown.

[0035] The memory device chiplet 150 can be or contains any combination of volatile memory devices or non-volatile memory. Examples of volatile memory devices include, but are not limited to, random access memory (RAM) – such as DRAM, synchronous DRAM (SDRAM), graphics DDR type 6 SDRAM (GDDR6 SDRAM), etc. Examples of non-volatile memory devices include, but are not limited to, NAND flash memory, memory-class memory (e.g., phase-change memory or memristor-based technology), ferroelectric RAM (FeRAM), etc. The illustrated example includes memory device 150 as a chiplet; however, memory device 150 may reside elsewhere, such as in different packages on board 105. For many applications, multiple memory device chipsets may be provided. In the example, these memory device chipsets may each implement one or more memory technologies. In the example, the memory chiplet may contain multiple stacked memory dies of different technologies, such as stacked or additionally communicating with one or more DRAM devices or one or more SRAM devices. The memory controller 140 can also be used to coordinate the operation between multiple memory chips in the chiplet system 110; for example, utilizing one or more memory chips in one or more tiers of cache memory and using one or more additional memory chips as main memory. The chiplet system 110 may also include multiple memory controllers 140, which can be used to provide memory control functionality for individual processors, sensors, networks, etc. For example, the chiplet architecture of the chiplet system 110 offers the advantage of allowing adaptation to different memory storage technologies and different memory interfaces through updated chiplet configurations without requiring redesign of the rest of the system architecture.

[0036] Figure 2 Components of an example of a memory controller chiplet 205 according to an embodiment are shown. The memory controller chiplet 205 includes a cache 210, a cache controller 215, an off-die memory controller 220 (e.g., for communicating with off-die memory 175), a network communication interface 225 (e.g., for interfacing with chiplet network 180 and communicating with other chiplets), and a set of atomic and merge operations 250. Members of this set may include, for example, a write merge unit 155, a dangerous unit (160), a built-in atomic unit 165, or a PAU 170. The various components are illustrated logically and they may not necessarily be implemented. For example, a built-in atomic unit 165 may include different means along the path to off-die memory. For example, a built-in atom may be in an interface means / buffer on the memory chiplet, as discussed above. In contrast, programmable atomic operation 170 may be implemented in a separate processor on the memory controller chiplet 105 (but in various instances, it may be implemented elsewhere, such as on the memory chiplet itself).

[0037] The off-die memory controller 220 is directly coupled to the off-die memory 275 (e.g., via a bus or other communication connection) to provide write operations to, and read operations from, the one or more off-die memories, such as off-die memory 275 and off-die memory 280. In the depicted example, the off-die memory controller 220 is also coupled to outputs to the atom and merge unit 250 and inputs to the cache controller 215 (e.g., a memory-side cache controller).

[0038] In the instance configuration, the cache controller 215 is directly coupled to the cache 210 and can be coupled to the network communication interface 225 for input (e.g., incoming read or write requests) and coupled to the off-die memory controller 220 as an output.

[0039] Network communication interface 225 includes packet decoder 230, network input queue 235, packet encoder 240, and network output queue 245 to support packet-based chiplet network 285, such as CPI. Chiplet network 285 provides packet routing between processors, memory controllers, mixed-thread processors, configurable processing circuitry, or communication interfaces. In this packet-based communication system, each packet typically contains destination and source addressing, as well as any data payload or instructions. In instances, depending on the configuration, chiplet network 285 may be implemented as a collection of crossbars with a folded Clos configuration, or as a mesh network providing additional connectivity.

[0040] In various instances, the chiplet network 285 may be part of an asynchronous switching architecture. Here, data packets may be routed along any of various paths, such that any selected data packet may arrive at its addressed destination at any time among multiple different times, depending on the route. Furthermore, the chiplet network 285 may be implemented at least partially as a synchronous communication network, such as a synchronous mesh communication network. Two configurations of the communication network are used, for example, according to this disclosure, upon careful consideration.

[0041] The memory controller chip 205 can receive packets having, for example, a source address, a read request, and a physical address. In response, the off-die memory controller 220 or the cache controller 215 reads data from the specified physical address (which may be in off-die memory 275 or cache 210) and assembles the response packet into a source address containing the requested data. Similarly, the memory controller chip 205 can receive packets having a source address, a write request, and a physical address. In response, the memory controller chip 205 writes data to the specified physical address (which may be in cache 210 or off-die memory 275 or 280) and assembles the response packet into a source address containing confirmation that data has been stored in memory.

[0042] Therefore, where possible, the memory controller chiplet 205 can receive read and write requests via chiplet network 285 and process the requests using cache controller 215, which interfaces with cache 210. If cache controller 215 cannot process the request, off-die memory controller 220 processes the request by communicating with off-die memory 275 or 280, atomic and merge operations 250, or both. As described above, one or more cache levels can also be implemented in off-die memory 275 or 280; and in some such instances, they can be directly accessed by cache controller 215. Data read by off-die memory controller 220 can be cached in cache 210 by cache controller 215 for later use.

[0043] Atom and merge operation 250 is coupled (as input) to receive the output of off-die memory controller 220 and provides the output to cache 210, network communication interface 225, or directly to chiplet network 285. Memory danger clear (reset) unit 260, write merge unit 265, and built-in (e.g., predetermined) atom operation unit 265 can each be implemented as a state machine with other combinational logic circuitry (such as adders, shifters, comparators, AND gates, OR gates, XOR gates, or any suitable combination thereof) or other logic circuitry. These components may also include one or more registers or buffers to store operands or other data. PAU 270 may be implemented as one or more processor cores or control circuitry, and various state machines with other combinational logic circuitry or other logic circuitry, and may also include one or more registers, buffers, or memories to store addresses, executable instructions, operands, and other data, or may be implemented as a processor.

[0044] Write merging unit 255 receives read data and request data, and merges the request data and read data to produce a single unit having the read data and the source address to be used in response or return data packets. Write merging unit 255 provides the merged data to the write port of cache 210 (or equivalently, to cache controller 215 for writing to cache 210). Optionally, write merging unit 255 provides the merged data to network communication interface 225 for encoding and preparing response or return data packets for transmission on chiplet network 285.

[0045] When requested data is used for a built-in atomic operation, the built-in atomic operation unit 265 receives the request and reads the data from the write merging unit 265 or directly from the off-chip memory controller 220. The atomic operation is performed, and using the write merging unit 255, the resulting data is written to the cache 210 or provided to the network communication interface 225 to encode and prepare response or return data packets for transmission over the chiplet network 285.

[0046] Built-in atomic operation unit 265 handles predefined atomic operations, such as fetch and increment or compare and swap. In examples, these operations perform simple read-modify-write operations on a single memory location of 32 bytes or less. Atomic memory operations are initiated from request packets transmitted via chiplet network 285. The request packet has a physical address, atomic operator type, operand size, and optionally up to 32 bytes of data. The atomic operation performs a read-modify-write operation on a cache line of cache 210, thereby filling the cache memory as necessary. The atomic operator response can be a simple complete response or a response with up to 32 bytes of data. Example atomic memory operators include fetch and AND, fetch and OR, fetch and XOR, fetch and add, fetch and subtract, fetch and increment, fetch and decrement, fetch and minimum, fetch and maximum, fetch and swap, and compare and swap. In various example embodiments, 32-bit and 64-bit operations and operations on 16 or 32 bytes of data are supported. The methods disclosed herein are also compatible with hardware that supports larger or smaller operations and more or less data.

[0047] Built-in atomic operations may also involve requests for "standard" atomic operations on the requested data, such as relatively simple single-cycle integer atoms, such as fetch and increment or compare and swap, whose throughput will be the same as conventional memory read or write operations that do not involve atomic operations. For these operations, cache controller 215 can typically preserve cache lines in cache 210 by setting a danger bit (in hardware) so that the cache line cannot be read by another process during translation. Data is obtained from off-chip memory 275 or cache 210 and provided to built-in atomic operation unit 265 to perform the requested atomic operation. After the atomic operation, in addition to providing the obtained data to data packet encoder 240 to encode outgoing data packets for transmission on chiplet network 285, built-in atomic operation unit 265 also provides the obtained data to write merging unit 255, which writes the obtained data to cache circuitry 210. After the obtained data is written to cache 210, memory danger clearing unit 260 clears any corresponding danger bits set.

[0048] The PAU 270 implements high-performance (high throughput and low latency) programmable atomic operations (also known as "custom atomic operations"), comparable to the performance of built-in atomic operations. Instead of performing multiple memory accesses, in response to an atomic operation request specifying a programmable atomic operation and a memory address, circuitry in the memory controller chiplet 205 transmits the atomic operation request to the PAU 270 and sets a danger bit in a memory danger register corresponding to the memory address used in the atomic operation to ensure that no other operation (read, write, or atomic) is performed on that memory line. The danger bit is then cleared after the atomic operation is completed. The additional, direct data path provided to the PAU 270 for performing programmable atomic operations allows for additional write operations without being limited by the bandwidth of the communication network and without increasing any congestion on the communication network.

[0049] The PAU 270 includes a multi-threaded processor, such as a RISC-VIS-based multi-threaded processor, with one or more processor cores and further featuring an extended instruction set for performing programmable atomic operations. When equipped with the extended instruction set for performing programmable atomic operations, the PAU 270 can be embodied as one or more hybrid-threaded processors. In some example implementations, the PAU 270 provides bucket-style polling instantaneous thread switching to maintain a high instruction-per-clock rate.

[0050] Programmable atomic operations can be executed by PAU 270, which relate to requests for programmable atomic operations on requested data. Users can prepare programming code to provide such programmable atomic operations. For example, a programmable atomic operation can be a relatively simple multi-loop operation, such as floating-point addition, or it can be a relatively complex multi-instruction operation, such as a Bloom filter insert. Programmable atomic operations can be the same as or different from predetermined atomic operations, as long as they are defined by the user and not the system vendor. For these operations, cache controller 215 can preserve cache lines in cache 210 by setting a danger bit (in hardware), preventing the cache lines from being read by another process during translation. Data is obtained from cache 210 or off-chip memory 275 or 280 and provided to PAU 270 to execute the requested programmable atomic operation. After the atomic operation, PAU 270 provides the obtained data to network communication interface 225 to directly encode outgoing data packets containing the obtained data for transmission on chiplet network 285. Furthermore, PAU 270 provides the obtained data to cache controller 215, which in turn writes the obtained data to cache 210. After writing the obtained data to cache 210, cache control circuit 215 clears any corresponding dangerous bits that were set.

[0051] In the selected example, the approach taken for programmable atomic operations is to provide multiple general-purpose custom atomic request types, which can be sent from a source of origin, such as a processor or other system component, to the memory controller chiplet 205 via chiplet network 285. Cache controller 215 or off-die memory controller 220 identifies the request as a custom atom and forwards the request to PAU 270. In a representative embodiment, PAU 270: (1) is a programmable processing element capable of efficiently performing user-defined atomic operations; (2) can perform load and store operations on memory, arithmetic and logical operations, and control flow decisions; and (3) utilizes a RISC-V ISA with a new set of dedicated instructions to facilitate interaction with such controllers 215, 220, thereby performing user-defined operations atomically. In a desirable example, the RISC-V ISA contains a complete instruction set supporting high-level language operators and data types. The PAU 270 can utilize a RISC-V ISA, but will typically support a more limited instruction set and a limited register file size to reduce the die size of the cell when included within the memory controller chiplet 205.

[0052] As mentioned above, before writing read data to cache 210, the memory danger clearing unit 260 clears the set danger bits of the reserved cache lines. Therefore, when the write merging unit 255 receives a request and read data, the memory danger clearing unit 260 can transmit a reset or clear signal to cache 210 to reset the set memory danger bits of the reserved cache lines. Furthermore, resetting this danger bit also releases pending read or write requests involving the specified (or reserved) cache line, thereby providing the pending read or write requests to the inbound request multiplexer for selection and processing.

[0053] Figure 3 This illustration shows an example of routing between electronic devices (e.g., chipsets in chiplet layout 300 using a chiplet protocol interface (CPI) network) in a device layout 300 according to an embodiment. Device layout 300 includes electronic devices, which in this example may include one or more electronic devices in the form of individual chipsets 310A, 310B, 310C, 310D, 310E, 310F, 310G, and 310H, interconnected via a network including nodes 330A, 330B, 330C, 330D, 330E, 330F, 330G, and 330H. Nodes 330A-330H may be housed in an NOC 340, which is a separate chiplet or part of another chiplet (e.g., Figure 1A and 1BNOC 130 (shown as part of application chiplet 125), or nodes 330A-330H, can be distributed in other structures of the chiplet system, such as in one or more of chipsets 310A-310H. Each node may contain one or more switches and / or transceivers arranged and controlled to provide the communication as described herein. Electronic devices 310A-310H each contain one of hardware transceivers 320A-320H. In one embodiment, electronic devices 310A-310H are each chiplets, and the network is a CPI network. Thus, device layout (or chiplet layout) 300 includes chipsets 310A, 310B, 310C, 310D, 310E, 310F, 310G, and 310H, which are interconnected via a CPI network including nodes 330A, 330B, 330C, 330D, 330E, 330F, 330G, and 330H. Each of the 310A-310H chips contains one of the 320A-320H hardware transceivers.

[0054] CPI packets can be transmitted between chiplets 310 using the Advanced Interface Bus (AIB). The AIB provides physical layer functionality. The physical layer uses source-synchronous data transfer with a forwarding clock to transmit and receive data. Packets are transmitted across the AIB relative to the transmitted clock in SDR or DDR. The AIB supports various channel widths. When operating in SDR mode, the AIB channel width is a multiple of 20 bits (20, 40, 60…), and for DDR mode, the AIB channel width is a multiple of 40 bits (40, 80, 120…). The AIB channel width includes both transmit and receive signals. Channels can be configured with a symmetrical number of transmit (TX) and receive (RX) inputs / outputs (I / O), or with an asymmetrical number of transmitters and receivers (e.g., all transmitters or all receivers).

[0055] The AIB adapter provides interfaces to the AIB link layer and to the AIB physical layer (PHY). The AIB adapter provides a data hierarchy register, a power-on reset sequencer, and a control signal shift register.

[0056] The AIB physical layer consists of AIB I / O units. AIB I / O units (implemented in some embodiments by hardware transceiver 320) can be input-only, output-only, or bidirectional. An AIB channel consists of a set of AIB I / O units, the number of which depends on the AIB channel configuration. A received signal on a chiplet is connected to a pair of transmitted signals on the chipsets. In some embodiments, each column includes an auxiliary (AUX) channel and data channels numbered 0 to N.

[0057] Data packets are routed within chiplet 310 via network node 330. Node 330 can determine the next node 330 to forward a received data packet to based on one or more data fields of the data packet. For example, source or destination address, source or destination port, virtual channel, or any suitable combination thereof can be hashed to select consecutive network nodes or available network paths. This method of path selection can be used to balance network traffic.

[0058] Therefore, in Figure 3 The diagram illustrates the data path from chiplet 310A to chiplet 310D. Data packets are sent from hardware transceiver 320A to network node 330A; forwarded by network node 330A to network node 330C; forwarded by network node 330C to network node 330D; and then passed from network node 330D to hardware transceiver 320D of chiplet 310D.

[0059] Figure 3 The diagram also illustrates a second data path from chiplet 310A to chiplet 310G. Data packets are sent from hardware transceiver 320A to network node 330A; forwarded by network node 330A to network node 330B; forwarded by network node 330B to network node 330D; forwarded by network node 330D to network node 330C; forwarded by network node 330C to network node 330E; forwarded by network node 330E to network node 330F; forwarded by network node 330F to network node 330H; forwarded by network node 330H to network node 330G; and then passed from network node 330G to hardware transceiver 320G of chiplet 310G. Figure 3 As can be seen, multiple paths through the network can be used for data transfer between any pair of chips.

[0060] The AIB I / O unit supports three clock modes: asynchronous (i.e., non-timed), SDR, and DDR. Non-timed mode is used for clocking and some control signals. SDR mode can use a dedicated SDR-only I / O unit or a dual-purpose SDR / DDR I / O unit.

[0061] The CPI packet protocol (point-to-point and routable) can use the AIB channel in either SDR or DDR operating mode. In some example implementations, the AIB channel will increment by 80 I / O units (i.e., 40 TX and 40 RX) for SDR mode and by 40 I / O units for DDR mode.

[0062] The CPI streaming protocol can use the AIB channel in either SDR or DDR operating mode. In some example implementations, the AIB channel increments by 40 I / O units for both modes (SDR and DDR).

[0063] A unique interface identifier is assigned to each AIB channel. This identifier is used during CPI reset and initialization to determine the paired AIB channels across neighboring chiplets. In some implementations, the interface identifier is a 20-bit value comprising a seven-bit chiplet identifier, a seven-bit column identifier, and a six-bit link identifier. The AIB physical layer uses an AIB out-of-band shift register to transmit the interface identifier. Bits 32 to 51 of the shift register are used to transmit the 20-bit interface identifier in both directions across the AIB interface.

[0064] In some implementations, AIB channels are numbered in ascending order, starting with the AIB channel adjacent to the AUX channel. The AIB channel adjacent to the AUX channel is defined as AIB channel zero.

[0065] Figure 3 As an example, eight chiplets 310 are shown connected by a network comprising eight nodes 330. The chiplet network can contain more or fewer chiplets 310 and more or fewer nodes 330, thus allowing the creation of chiplet networks of any size.

[0066] Data packets can be transmitted between electronic devices (e.g., chiplets) 310 via an interface network (e.g., a CPI network) based on paths and ordering requirements specified on a packet-by-packet basis. Each data packet can specify its own ordering requirements to allow for flexibility in ordering and routing control, such as improving data transmission performance by reducing unnecessary congestion. In various embodiments, the ordering and routing for data packet transmission are determined to maximize data transmission performance while satisfying ordering and / or routing requirements.

[0067] Figure 4 This is a block diagram of a data packet 400 suitable for a system allowing optional path ordering in packet transmission, according to some embodiments of the present disclosure. The system may include a plurality of electronic devices (e.g., electronic devices 310A-310H) coupled to each other via an interface network (e.g., including nodes 330A-330H) and a plurality of hardware transceivers (e.g., hardware transceivers 320A-320H) each contained in or coupled to said electronic devices. The plurality of hardware transceivers may each receive data packets from source devices of the plurality of electronic devices and transmit the received data packets to destination devices of the plurality of electronic devices. These data packets may each be instances of data packet 400.

[0068] Data packet 400 includes a protocol field 405, a path field 410, and one or more additional fields containing any type of data for system operation. Protocol field 405 specifies sequencing information for delivering data packet 400 to a destination device of a plurality of electronic devices. The sequencing information specifies the order in which each data packet 400 arrives at its destination device relative to other data packets 400. Path field 410 specifies path information for routing the delivery of data packet 400 to the destination device. Upon receiving each data packet 400, the hardware transceiver can transmit it to the destination device using at least the sequencing information contained in the data packet, and can transmit it to the destination device using both the sequencing information and the path information contained in the data packet when the sequencing information contained in the data packet indicates that the delivery of the data packet needs to be routed.

[0069] In one embodiment, the sorting information contained in protocol field 405 may be represented by one or more values ​​selected from the following values, which include (but are not limited to):

[0070] ● The first value (e.g., 0) indicates out-of-order delivery;

[0071] ● The second value (e.g., 1) indicates the ordered delivery via the path specified in path field 410; and

[0072] ● The third value (e.g., 2) indicates an ordered delivery through a single pre-defined (e.g., default) path.

[0073] Data packets 400 transmitted in the system may include a first packet (each having a first value in the protocol field), a second packet (each having a second value in the protocol field), and / or a third packet (each having a third value in the protocol field). Multiple hardware transceivers may each receive these data packets 400, and transmit the first packets of the received data packets to the destination device in a predetermined order, transmit the second packets of the received data packets to the destination device in a predetermined order via one or more paths in the interface network specified in the path field, and transmit the third packets of the received data packets to the destination device in a predetermined order via a single predetermined path (e.g., a default path or a path set at system startup). For ordered delivery, data packets may be transmitted in the order in which the hardware transceivers receive the data packets or in other predetermined manner as if they were received from the source device.

[0074] Protocol field 405 and path field 410 are included in every data packet 400 transmitted in the system. For example, protocol field 405 and path field 410 may be included in the header of every data packet 400. Protocol field 405 specifies whether the data packet is intended for out-of-order delivery, ordered delivery via the path specified in the path field, or ordered delivery via a single predetermined path. When the protocol field of the packet specifies ordered delivery via the path specified in the path field, path field 410 specifies the path, and path selection may be ignored when the protocol field specifies out-of-order delivery or ordered delivery via a single predetermined path. In one example, the system includes, for example,... Figure 3 The illustrated chiplet layout 300 includes multiple electronic devices, each a chiplet, and a CPI network is used to transmit data packets between the chiplets. In this example, chiplets 310A-310H each represent an instance of an electronic device among multiple electronic devices, hardware transceivers 320A-320H each represent a hardware transceiver among multiple hardware transceivers, and the CPI network represents an instance of an interface network. Data packets 400 may contain data packets transmitted in the CPI network.

[0075] Figure 5 This is a block diagram of data packet 500 adapted for transmission in a CPI network according to some embodiments of the present disclosure. When used in packet transmission in a CPI network, data packet 500 may represent an instance of data packet 400. Figure 5 As shown in the example, data packet 500 is divided into flow control units (chips), each chip consisting of 36 bits. The first chip 501 of data packet 500 (e.g., a header) contains a Credit / Path Order (CP) field 505, a Path (PATH) field 510, a Source Identifier (SID) field 515 (e.g., the first four bits of an eight-bit SID), a Destination Identifier (DID) field 520, a Sequence Continuation (SC) field 525, a Length (LEN) field 530, and a Command (CMD) field 535. The second chip 502 of data packet 500 contains additional data that may be unrelated to the routing of the packet but is used by the receiver. Here, the second micro-piece contains an address (ADDR) (e.g., a memory address), which is divided into two fields—the ADDR field has field 540, holding bits 15 through 33 of the address, and field 550 holds bits 3 through 6 of the address—a transaction identifier (TID) field 545, a "half-word" (H) field 555, and a field 560 holding the remaining portion of the SID (e.g., the last four bits). In various embodiments of this disclosure, the data packet 500 may contain any number of micro-pieces, and other micro-pieces may have similar characteristics to those in other embodiments. Figure 5 The different fields shown.

[0076] The CP field 505 can represent an instance of the protocol field 405. It is a two-bit field indicating whether out-of-order delivery is used (e.g., allowing delivery in random order), whether ordered delivery is used via a path from source to destination, or whether delivery is used via the path specified by the PATH field 510. The PATH field 510 can represent an instance of the path field 410. It is an eight-bit field (e.g., filled with eight bits of the destination device's address, such as...). Figure 5 (As shown). Table 1 lists example descriptions of the values ​​for CP field 505. These field sizes and values ​​are for illustrative purposes only and may contain any size and value necessary to accommodate sorting and path information.

[0077] CP field value describe 0 disorder 1 Path sorting based on the PATH field 2 Single path sorting 3 reserve

[0078] Table 1

[0079] Data packets 500 with the CP field 505 set to 0 can be delivered without any specific order. The PATH field 510 can be ignored. Data packets 500 can be routed regardless of the value of the PATH field 510 and can be received by the destination device in a different order than the order in which they were sent by the source device. This avoids congestion and allows for greater throughput in the CPI network.

[0080] Data packets 500 with CP field 505 set to 1 and PATH field 510 specifying a common path value are transmitted in the order they were sent by the source device (or in the order otherwise specified by the source device). PATH field 510 is used to determine this path, ensuring that all data packets 500 with the same value in this field are transmitted via the same path through the CPI network. In this example, PATH field 510 contains address bits 14:7 specifying the path. All data packets 500 with CP field 505 set to 1 and PATH field 510 set to a common address are transmitted sequentially from their source device to their destination device via the same path identified by the common address in the CPI network. This ensures correct system operation, depending on the ordered transmission of data packets through the paths specified in the packets.

[0081] Data packets 500 with CP field 505 set to 2 can be transmitted sequentially from the source device to the destination device via a single predetermined path. This single predetermined path can be considered the system default path, independent of the value of PATH field 510 (which can have any value). All data packets 500 with CP field 505 set to 2, regardless of the value of their PATH field 510, are transmitted sequentially from their source device to their destination device via the predetermined path. This sequential transmission of data packets via the predetermined path (e.g., the system default path) ensures correct system operation.

[0082] The value 3 of CP field 505 is reserved for future use (e.g., when another type of ordered or unordered delivery is required). When value 3 is not used, it can default to the description of another value. For example, when value 3 is "reserved" (not for another transmission method), data packets 500 with CP field 505 set to 1 or 3 can be transmitted in the same manner.

[0083] The use of the CP field 505 and PATH field 510 in each data packet 500 provides flexible network support for multiple sorting requirements without degrading the performance of use cases that do not require sorting all packets. Because sorting information is provided on a packet-by-packet basis, rather than setting a sorting scheme for all data packets being transmitted, data packets with different sorting requirements can be arranged in a deterministic time and space across the CPI network to maximize the performance of packet-based data transmission. For example, it allows data packets to be transmitted simultaneously over different paths.

[0084] The remaining fields of data packet 500 are illustrated as examples and, aside from identifying the source and destination devices, may not affect the order and path of transmission. The SID field 515 stores a four-bit SID. The SID uniquely identifies the source device (e.g., the source chiplet) in the network. The DID field 520 stores a 12-bit DID. The DID uniquely identifies the destination device (e.g., the destination chiplet) in the network. It is guaranteed that all data packets with the SC field 525 set are transmitted in sequence. The LEN field 530 is a five-bit field indicating the number of chips included in data packet 500. The interpretation of the LEN field 530 can be non-linear. For example, values ​​0-22 can be interpreted as 0-22 chips in data packet 500, while values ​​23-27 can be interpreted as 33-37 chips in data packet 500 (i.e., 10 more than the indicated value). Other values ​​for the LEN field 530 can be defined by the vendor, rather than by the protocol.

[0085] Commands for data packet 500 are stored in CMD field 535, which is a seven-bit field. Commands can be write commands, read commands, predefined atomic operation commands, custom atomic operation commands, read responses, acknowledgment responses, or vendor-specific commands. Additionally, commands can indicate the virtual channel of data packet 500. For example, different commands can be used for different virtual channels, or bits 1, 2, 3, or 4 of the seven-bit command field 535 can be used to indicate the virtual channel, while the remaining bits can be used to indicate the command.

[0086] The memory access command may further identify the number of bytes to be written or accessed, the memory space to be accessed (e.g., die memory 375 or instruction memory for custom atomic operations), or any suitable combination thereof. In some example embodiments, the command may instruct additional bits on a later microchip to identify the command.

[0087] The ADDR fields, when combined, specify the memory address to be accessed. However, as shown, ADDR field 540 in the header stores a portion of the address that can be used to determine the routing path based on the CP value. TID field 545 identifies the transaction associated with data packet 500. H field 555 contains additional address bits required when using half-word access. When the four bits of SID field 515 are insufficient to identify the source device (e.g., source chiplet) in the network, SID field 560 stores another four-bit SID.

[0088] Figure 6 This is a flowchart illustrating the operation of a method 600 for optional path sequencing in packet transmission according to some embodiments of the present disclosure. Method 600 can be performed for data packet transmission between a source device and a destination device in a system comprising multiple electronic devices (e.g., chiplets) via an interface network that connects the multiple electronic devices to each other via various routes. In an example, the source device and / or the destination device are each a chiplet, and the interface network comprises a CPI network as discussed in this disclosure. In an example, the multiple electronic devices include memory devices.

[0089] At 610, data packets are received from the source device. Each data packet contains sorting information destined for the destination device and path information that will route the data to the destination device via the interface network.

[0090] At 620, the received data packets are transmitted to the destination device using at least the ordering information for each packet in the received data packets. The received data packets are also transmitted to the destination device using the ordering information and path information for each packet in the received data packets, where the ordering information indicates that the delivery of packets needs to be routed. In response to the ordering information specifying ordered delivery, the received data packets are transmitted to the destination device using ordered delivery. In response to the ordering information specifying unordered delivery, the received data packets are transmitted to the destination device using unordered delivery.

[0091] Figure 7This is a flowchart illustrating the operation of a method 700 for alternative path sequencing of packet transmission using protocol field values ​​in transmitted packets, according to some embodiments of the present disclosure. Method 700 may represent another embodiment of method 600 for transmitting data packets between a source device and a destination device in a system containing multiple electronic devices (e.g., chiplets) by connecting multiple electronic devices to each other via an interface network with various routes.

[0092] At 710, data packets are received from the source device. Each data packet contains a protocol field and a path field. The protocol field specifies ordering information for transmission to the destination device, and the path field specifies path information for routing the transmission to the destination device via the interface network. An example of a data packet is data packet 400, while data packet 500 is a more specific example for a CPI network.

[0093] At 720, the value of the protocol field for each received data packet is determined. In one instance, the protocol field may have one of at least three values: a first value (e.g., 0), a second value (e.g., 1), and a third value (e.g., 2). The first value indicates out-of-order delivery. The second value indicates ordered delivery and specifies the path used to route the delivery to the destination device via the path specified in the path field. The third value indicates ordered delivery via a single predetermined path (i.e., regardless of the path field value). Ordered delivery requires that data packets be delivered to the destination device in the order in which they were received from the source device or in an order otherwise specified (e.g., the order specified by the source device).

[0094] At 730, if the protocol field of the received data packet has a first value, the received data packets are not transmitted to the destination device in a predetermined order. At 740, if the protocol field of the received data packet has a second value, the received data packets are transmitted to the destination device in a predetermined order via the path in the interface network specified in the path field. At 750, if the protocol field of the received data packet has a third value, the received data packets are transmitted to the destination device in a predetermined order via a single predetermined path.

[0095] Figure 8This is a block diagram of an instance machine 800 by which any or more of the techniques discussed herein (e.g., method 600 or method 700) can be implemented. As described herein, an instance may contain logic or multiple components or mechanisms in machine 800, or operable therefrom. A circuit system (e.g., a processing circuit system) is a collection of circuits (e.g., simple circuits, gates, logic, etc.) implemented in a tangible entity containing the hardware of machine 800. The membership of a circuit system can be flexible over time. A circuit system contains members that can perform a particular task individually or in combination during operation. In an instance, the hardware of a circuit system can be designed in an immutable manner to perform a particular operation (e.g., hardwired). In an instance, the hardware of a circuit may contain physically connected components (e.g., execution units, transistors, simple circuits, etc.) and machine-readable media that are physically modified (e.g., the magnetism, electricity, movable placement, etc. of an unchanging number of particles) to encode a particular operation. When connecting physical components, the fundamental electrical properties of the hardware components are changed, for example, from an insulator to a conductor, or vice versa. Instructions enable embedded hardware (e.g., an execution unit or a loading mechanism) to hardware-create components of a circuit system via variable connections to perform specific operations when in operation. Thus, in this example, a machine-readable media element is part of the circuit system or communicatively coupled to other components of the circuit system during device operation. In this example, any one of the physical components can be used in more than one component of more than one circuit system. For example, in operation, an execution unit may be used at one point in time in a first circuit of a first circuit system and reused by a second circuit of the first circuit system, or reused at a different time by a third circuit of a second circuit system. Further examples of these components of machine 800 are given below.

[0096] In alternative embodiments, machine 800 may be used as a standalone device or may be connected (e.g., networked) to other machines. In a networked deployment, machine 800 may operate as a server machine, a client machine, or both in a server-client network environment. In an example, machine 800 may act as a peer-to-peer (P2P) (or other distributed) network environment. Machine 800 may be a personal computer (PC), tablet PC, set-top box (STB), personal digital assistant (PDA), mobile phone, network appliance, network router, switch, or bridge, or any machine capable of executing (sequentially or otherwise) instructions specifying actions to be taken by said machine. Furthermore, although only a single machine is shown, the term "machine" should also be considered as encompassing any collection of machines, such as cloud computing, Software as a Service (SaaS), or other computer cluster configurations, that individually or jointly execute one or more sets of instructions to perform any one or more of the methodologies discussed herein.

[0097] Machine 800 (e.g., a computer system) may include a hardware processor 802 (e.g., a central processing unit (CPU), a graphics processing unit (GPU), a hardware processor core, or any combination thereof), main memory 804, static memory 806 (e.g., memory or storage device for firmware, microcode, basic input / output (BIOS), unified extensible firmware interface (UEFI), etc.), and mass storage device 808 (e.g., a hard disk drive, tape drive, flash memory, or other block device), some or all of which may communicate with each other via interconnect 830 (e.g., a bus). Machine 800 may further include a display unit 810, an alphanumeric input device 812 (e.g., a keyboard), and a user interface (UI) navigation device 814 (e.g., a mouse). In an example, the display unit 810, the input device 812, and the UI navigation device 814 may be a touchscreen display. Machine 800 may additionally include a storage device (e.g., a drive unit) 808, a signal generation device 818 (e.g., a speaker), a network interface device 820, and one or more sensors 816, such as a Global Positioning System (GPS) sensor, a compass, an accelerometer, or other sensors. Machine 800 may include an output controller 828, for example, serially (e.g., Universal Serial Bus (USB), parallelly, or other wired or wireless (e.g., infrared (IR), near field communication (NFC), etc.) connected to communicate with or control one or more peripheral devices (e.g., a printer, a card reader, etc.).

[0098] The registers of processor 802, main memory 804, static memory 806, or mass storage device 808 may be or contain machine-readable media 822, on which one or more data structures or sets of instructions 824 (e.g., software) embodying or utilized by any one or more of the techniques or functions described herein may be stored. Instructions 824 may also reside wholly or at least partially in any of the registers of processor 802, main memory 804, static memory 806, or mass storage device 808 during execution by machine 800. In an example, one or any combination of hardware processor 802, main memory 804, static memory 806, or mass storage device 808 may constitute machine-readable media 822. While machine-readable media 822 is described as a single medium, the term "machine-readable media" may include a single medium or multiple media (e.g., a centralized or distributed database, or associated caches and servers) configured to store the one or more instructions 824.

[0099] The term "machine-readable media" can include any medium capable of storing, encoding, or carrying instructions executable by machine 800 and causing machine 800 to perform any one or more of the technologies disclosed herein, or any medium capable of storing, encoding, or carrying data structures used or associated with such instructions. Examples of non-limiting machine-readable media may include solid-state memory, optical media, magnetic media, and signals (e.g., radio frequency signals, other photon-based signals, sound signals, etc.). In examples, non-transitory machine-readable media includes machine-readable media having a plurality of particles having invariant (e.g., rest) mass and therefore being composed of matter. Thus, non-transitory machine-readable media is machine-readable media that does not contain transiently propagating signals. Specific examples of non-transitory machine-readable media may include: non-volatile memory, such as semiconductor memory devices (e.g., electrically programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM)) and flash memory devices; magnetic disks, such as internal hard disks and removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks.

[0100] In an example, information stored or otherwise provided on machine-readable medium 822 may represent instructions 824, such as instructions 824 itself or a format from which instructions 824 can be derived. This format from which instructions 824 can be derived may include source code, encoded instructions (e.g., in compressed or encrypted form), packaged instructions (e.g., split into multiple packages), etc. Information representing instructions 824 on machine-readable medium 822 may be processed by a processing circuitry system into instructions to perform any of the operations discussed herein. For example, deriving instructions 824 from information (e.g., processed by a processing circuitry system) may include: compiling (e.g., from source code, object code, etc.), interpreting, loading, organizing (e.g., dynamically or statically linking), encoding, decoding, encrypting, decrypting, encapsulating, decapsulating, or otherwise manipulating information into instructions 824.

[0101] In an example, the derivation of instruction 824 may involve assembling, compiling, or decompiling information (e.g., by a processing circuitry) to create instruction 824 from some intermediate or preprocessed format provided by machine-readable media 822. When information is provided in multiple parts, the information may be combined, decapsulated, and modified to create instruction 824. For example, the information may be in multiple compressed source code data packets (or object code, or binary executable code, etc.) on one or more remote servers. The source code package may be encrypted during transmission over a network and, if necessary, decrypted, decompressed, assembled (e.g., linked), and compiled or decompiled at the local machine (e.g., into a stand-alone executable library, etc.) and executed by the local machine.

[0102] Instruction 824 may further utilize any of a plurality of transport protocols (e.g., Frame Relay, Internet Protocol (IP), Transmission Control Protocol (TCP), User Datagram Protocol (UDP), Hypertext Transfer Protocol (HTTP), etc.) to transmit or receive over communication network 826 via network interface device 820 using a transport medium. Example communication networks may include local area networks (LANs), wide area networks (WANs), packet data networks (e.g., the Internet), mobile phone networks (e.g., cellular networks), conventional telephone (POTS) networks, and wireless data networks (e.g., the Institute of Electrical and Electronics Engineers (IEEE) 802.11 series standards, referred to as… The IEEE 802.16 series of standards are called IEEE 802.15.4 series standards, point-to-point (P2P) networks, etc. In an example, network interface device 820 may include one or more physical jacks (e.g., Ethernet, coaxial, or telephone jacks) or one or more antennas to connect to communication network 826. In an example, network interface device 820 may include multiple antennas to communicate wirelessly using at least one of single-input multiple-output (SIMO), multiple-input multiple-output (MIMO), or multiple-input single-output (MISO) technologies. The term "transmission medium" should be considered as including any intangible medium capable of storing, encoding, or carrying instructions for execution by machine 800, and including digital or analog communication signals or other intangible media used to facilitate communication of this software. The transmission medium is a machine-readable medium.

[0103] In the foregoing description, some exemplary embodiments of this disclosure have been described. It will be apparent that various modifications can be made to this disclosure without departing from the broader spirit and scope of the examples set forth in the appended claims. Therefore, the description and drawings should be viewed in an illustrative rather than restrictive sense. The following is a non-exhaustive list of examples of embodiments of this disclosure. Reference is made to the accompanying drawings, which form a part thereof, and specific embodiments in which the invention may be practiced are illustrated by way of illustration. These embodiments have been described in sufficient detail to enable those skilled in the art to practice the invention, and it should be understood that embodiments may be combined or other embodiments may be utilized, and structural, logical, and electrical changes may be made without departing from the spirit and scope of the invention. References to “a,” “an,” or “various” embodiments in this disclosure are not necessarily references to the same embodiment, and such references contemplate more than one embodiment. The following detailed description provides examples, and the scope of the invention is defined by the appended claims and their legal equivalents.

[0104] In Example 1, a method is provided. The method may include receiving data packets from a source device of a plurality of electronic devices coupled to each other via an interface network. Each data packet may contain a protocol field and a path field, the protocol field specifying ordering information for delivery to a destination device of the plurality of electronic devices, and the path field specifying path information for routing the delivery to the destination device via the interface network. The plurality of electronic devices may contain a plurality of chips. The source device and the destination device may each contain a chip from the plurality of chips. The method may further include transmitting the received data packets to the destination device using at least the ordering information of each packet in the received data packets.

[0105] In Example 2, the subject of transmitting the received data packets to the destination device according to Example 1 may optionally include: transmitting a first packet of the received data packets to the destination device out of a predetermined order; transmitting a second packet of the received data packets to the destination device in a predetermined order via one or more paths in the interface network specified in the second packet; and transmitting a third packet of the received data packets to the destination device in a predetermined order via a single predetermined path. Each of the first packets has a first value in the protocol field indicating out-of-order delivery. Each of the second packets has a second value in the protocol field indicating ordered delivery and a value in the path field specifying the path used to route the delivery to the destination device. Each of the third packets has a third value in the protocol field indicating ordered delivery via the single predetermined path.

[0106] In Example 3, the subject of transmitting the received data packets to the destination device according to Example 1 may optionally include: transmitting the received data packets to the destination device using the ordered transmission in response to the sorting information specifying ordered transmission; and transmitting the received data packets to the destination device using the unordered transmission in response to the sorting information specifying unordered transmission.

[0107] In Example 4, the subject of transmitting the received data packets to the destination device according to Example 3 may optionally include: transmitting the received data packets to the destination device using the sorting information of each packet in the received data packets and the path information of each packet in the received data packets, wherein the sorting information indicates that the path information is required.

[0108] In Example 5, the subject of transmitting the received data packets to the destination device according to Example 1 may optionally include at least one of the following: transmitting a first packet of the received data packets to the destination device out of a predetermined order; transmitting a second packet of the received data packets to the destination device in a predetermined order via one or more paths in the interface network specified in the second packet; or transmitting a third packet of the received data packets to the destination device in a predetermined order via a single predetermined path. Each of the first packets has a first value in the protocol field indicating out-of-order delivery. Each of the second packets has a second value in the protocol field indicating ordered delivery and a value in the path field specifying the path used to route the delivery to the destination device. Each of the third packets has a third value in the protocol field indicating ordered delivery via the single predetermined path.

[0109] In Example 6, the data packet described according to any one or any combination of Examples 1 to 5 may optionally include commands for operating a memory device containing the plurality of electronic devices.

[0110] In Example 7, a system may include multiple electronic devices and multiple hardware transceivers. The multiple electronic devices may include multiple chips and may be coupled to each other via an interface network. The multiple hardware transceivers may each be contained in or coupled to a corresponding electronic device of the multiple electronic devices. The multiple hardware transceivers may each be configured to receive data packets from a source device. Each data packet includes a protocol field and a path field, the protocol field specifying ordering information for delivery to a destination device, and the path field specifying path information for routing the delivery to the destination device. The source device and the destination device may each include a chip. The multiple hardware transceivers may each be further configured to transmit the received data packets to the destination device using at least the ordering information of each packet in the received data packets.

[0111] In Example 8, the subject described in Example 7 can optionally be configured such that each of the plurality of hardware transceivers is configured to: transmit the received data packets to the destination device using the ordered delivery in response to the ordering information specifying ordered delivery; and transmit the received data packets to the destination device using the unordered delivery in response to the ordering information specifying unordered delivery.

[0112] In Example 9, the subject matter described according to any one or any combination of Examples 7 and 8 may optionally be configured such that each of the plurality of hardware transceivers is configured to transmit the received data packets to the destination device using the ordering information of each of the received data packets and the path information of each of the received data packets, wherein the ordering information indicates that the path information is required.

[0113] In Example 10, the subject matter described according to any one or any combination of Examples 7 to 9 can optionally be configured such that each of the plurality of hardware transceivers is configured to transmit a first packet of the received data packets to the destination device out of predetermined order. Each of the first packets has a first value in the protocol field indicating out-of-order delivery.

[0114] In Example 11, the subject matter described according to any one or any combination of Examples 7 to 10 can optionally be configured such that each of the plurality of hardware transceivers is configured to transmit a second packet of the received data packets to the destination device in a predetermined order via one or more paths in the interface network specified in the second packet. Each of the second packets has a second value in the protocol field indicating ordered delivery and a value in the path field specifying the path used to route the delivery to the destination device.

[0115] In Example 12, the subject matter described according to any one or any combination of Examples 7 to 11 can optionally be configured such that each of the plurality of hardware transceivers is configured to transmit a third packet of the received data packets to the destination device via a single predetermined path in a predetermined order. Each of the third packets has a third value in the protocol field indicating ordered delivery via the single predetermined path.

[0116] In Example 13, the subject matter according to Example 7 can optionally be configured such that each of the plurality of hardware transceivers is configured to perform at least one of the following: transmit a first packet of the received data packets to the destination device out of predetermined order; transmit a second packet of the received data packets to the destination device in predetermined order via one or more paths in the interface network specified in the second packet; or transmit a third packet of the received data packets to the destination device in predetermined order via a single predetermined path. Each of the first packets has a first value in the protocol field indicating out-of-order delivery. Each of the second packets has a second value in the protocol field indicating ordered delivery and a value in the path field specifying the path used to route the delivery to the destination device. Each of the third packets has a third value in the protocol field indicating ordered delivery via the single predetermined path.

[0117] In Example 14, the subject matter described according to any one or any combination of Examples 7 to 13 may optionally be configured to include a memory device that includes the plurality of chiplets and chiplet protocol interface network, and therefore the data packets each include a command field containing memory access commands.

[0118] In Example 15, a non-transitory machine-readable medium stores instructions that, when executed by a system, cause the system to perform operations including: receiving data packets from a source device comprising a plurality of electronic devices comprising a plurality of chiplets and coupled to each other via an interface network; and transmitting the received data packets to the destination device using at least the ordering information of each of the received data packets. Each data packet comprises a protocol field and a path field, the protocol field specifying ordering information for transmission to the destination device of the plurality of electronic devices, and the path field specifying path information for routing the transmission to the destination device via the interface network. The source device and the destination device may each comprise a chiplet from the plurality of chiplets.

[0119] In Example 16, the subject of transmitting the received data packets to the destination device according to Example 15 may optionally include transmitting a first packet of the received data packets to the destination device out of a predetermined order, transmitting a second packet of the received data packets to the destination device in a predetermined order via one or more paths in the interface network specified in the second packet, and transmitting a third packet of the received data packets to the destination device in a predetermined order via a single predetermined path. Each of the first packets has a first value in the protocol field indicating out-of-order delivery. Each of the second packets has a second value in the protocol field indicating ordered delivery and a value in the path field specifying the path used to route the delivery to the destination device. Each of the third packets has a third value in the protocol field indicating ordered delivery via the single predetermined path.

[0120] In Example 17, the subject of transmitting the received data packet to the destination device according to Example 15 may optionally include: transmitting the received data packet to the destination device using the ordered transmission in response to the sorting information specifying ordered transmission; and transmitting the received data packet to the destination device using the unordered transmission in response to the sorting information specifying unordered transmission.

[0121] In Example 18, the subject of transmitting the received data packets to the destination device according to Example 17 may optionally include: transmitting the received data packets to the destination device using the sorting information of each packet in the received data packets and the path information of each packet in the received data packets, wherein the sorting information indicates that the path information is required.

[0122] In Example 19, the subject of transmitting the received data packets to the destination device according to Example 15 may optionally include at least one of the following: transmitting a first packet of the received data packets to the destination device out of a predetermined order; transmitting a second packet of the received data packets to the destination device in a predetermined order via one or more paths in the interface network specified in the second packet; or transmitting a third packet of the received data packets to the destination device in a predetermined order via a single predetermined path. Each of the first packets has a first value in the protocol field indicating out-of-order delivery. Each of the second packets has a second value in the protocol field indicating ordered delivery and a value in the path field specifying the path used to route the delivery to the destination device. Each of the third packets has a third value in the protocol field indicating ordered delivery via the single predetermined path.

[0123] In Example 20, the data packets described according to any one or any combination of Examples 15 to 19 may optionally include commands for operating a memory device containing the plurality of electronic devices.

Claims

1. A method for packet transmission, comprising: Data packets are received from a source device comprising multiple electronic devices that are coupled to each other via an interface network. Each data packet contains a protocol field and a path field. The protocol field specifies ordering information for transmission to a destination device of the multiple electronic devices, and the path field specifies path information for routing the transmission to the destination device via the interface network. The source device and the destination device each contain a chip among the multiple chips. The first packet of the received data packets is transmitted to the destination device out of order, and each of the first packets has a first value in the protocol field indicating out-of-order delivery; and The second packet of the received data packets is transmitted to the destination device in a predetermined order through one or more paths in the interface network specified in the second packet, each of the second packets having a second value of the protocol field indicating ordered transmission and a value of the path field specifying the path for routing the transmission to the destination device.

2. The method according to claim 1, further comprising: The third packets of the received data packets are transmitted to the destination device in a predetermined order via a single predetermined path, each of the third packets having a third value of the protocol field indicating the ordered transmission via the single predetermined path.

3. The method of claim 2, wherein transmitting the second packet includes transmitting the second packet in the order in which the second packet is received; and transmitting the third packet includes transmitting the third packet in the order in which the third packet is received.

4. The method of claim 2, wherein transmitting the second packet comprises transmitting the second packet in a predetermined order at the source device receiving the second packet; and transmitting the third packet comprises transmitting the third packet in the predetermined order at the source device receiving the third packet.

5. The method of claim 1, wherein the data packet contains commands for operating a memory device comprising the plurality of electronic devices.

6. The method of claim 5, further comprising: The third packets of the received data packets are transmitted to the destination device in a predetermined order via a single predetermined path, each of the third packets having a third value of the protocol field indicating the ordered transmission via the single predetermined path.

7. The method of claim 1, further comprising determining the sorting information on a group-by-group basis.

8. The method according to claim 1, further comprising simultaneously transmitting multiple data packets of the received data packets to corresponding destination devices through different paths in the interface network.

9. A system for packet transmission, comprising: Multiple electronic devices coupled to each other via an interface network, the multiple electronic devices comprising multiple small chips; as well as A plurality of hardware transceivers, each contained in or coupled to a respective electronic device among the plurality of electronic devices, wherein each of the plurality of hardware transceivers is configured to: Data packets are received from a source device. Each data packet contains a protocol field and a path field. The protocol field specifies sorting information for transmission to a destination device. The path field specifies path information for routing the transmission to the destination device. The source device and the destination device each contain a chip among the plurality of chips. The first packet of the received data packets is transmitted to the destination device out of order, and each of the first packets has a first value in the protocol field indicating out-of-order delivery; and The second packet of the received data packets is transmitted to the destination device in a predetermined order through one or more paths in the interface network specified in the second packet, each of the second packets having a second value of the protocol field indicating ordered transmission and a value of the path field specifying the path for routing the transmission to the destination device.

10. The system of claim 9, wherein each of the plurality of hardware transceivers is further configured to transmit a third packet of the received data packets to the destination device via a single predetermined path in a predetermined order, each of the third packets having a third value of the protocol field indicating ordered transmission via the single predetermined path.

11. The system of claim 9, further comprising a memory device including the plurality of chiplets and an interface network, wherein each data packet includes a command field containing a memory access command.

12. The system of claim 11, wherein each of the plurality of hardware transceivers is configured to: The first packet is not transmitted to the destination device in a predetermined order; The second packet is transmitted to the destination device in the predetermined order via one or more paths in the interface network specified in the second packet; and The third packets of the received data packets are transmitted to the destination device in a predetermined order via a single predetermined path, each of the third packets having a third value of the protocol field indicating the ordered transmission via the single predetermined path.

13. The system of claim 11, wherein the memory device comprises a memory device chip from a variety of memory technologies.

14. The system of claim 11, wherein the memory device includes a memory device chiplet, the memory device chiplet comprising stacked memory dies of various memory technologies.

15. A non-transitory machine-readable medium storing instructions, which, when executed by a system, cause the system to perform operations including: Data packets are received from a source device of a plurality of electronic devices coupled to each other via an interface network. Each data packet contains a protocol field and a path field. The protocol field specifies ordering information for transmission to a destination device of the plurality of electronic devices. The path field specifies path information for routing the transmission to the destination device via the interface network. The plurality of electronic devices include a plurality of chips. The source device and the destination device each include a chip from the plurality of chips. The first packet of the received data packets is transmitted to the destination device out of order, and each of the first packets has a first value in the protocol field indicating out-of-order delivery; and The second packet of the received data packets is transmitted to the destination device in a predetermined order through one or more paths in the interface network specified in the second packet, each of the second packets having a second value of the protocol field indicating ordered transmission and a value of the path field specifying the path for routing the transmission to the destination device.

16. The non-transitory machine-readable medium of claim 15, wherein the operation further comprises: The third packets of the received data packets are transmitted to the destination device in a predetermined order via a single predetermined path, each of the third packets having a third value of the protocol field indicating the ordered transmission via the single predetermined path.

17. The non-transitory machine-readable medium of claim 15, wherein the data packets contain commands for operating a memory device comprising the plurality of electronic devices.

18. The non-transitory machine-readable medium of claim 17, wherein the operation further comprises transmitting a third packet of the received data packets to the destination device via a single predetermined path in a predetermined order, each of the third packets having a third value of the protocol field indicating ordered transmission via the single predetermined path.

19. The non-transitory machine-readable medium of claim 15, wherein the operation further comprises determining the sorting information on a group-by-group basis.

20. The non-transitory machine-readable medium of claim 19, wherein the operation further comprises: Multiple data packets from the received data packets are simultaneously transmitted to their respective destination devices via different paths in the interface network.

Images