Data flow driven hierarchical 2D-mesh network-on-chip structure

Abstract

The application discloses a data flow driven hierarchical 2D-Mesh network-on-chip structure, which comprises a node layer, a routing and gateway layer, a data flow mapping and control layer and a collective communication acceleration layer; the node layer is composed of a heterogeneous Tile array composed of computing / storage / communication Tiles; the routing and gateway layer is responsible for intra-layer forwarding and cross-layer convergence; the data flow mapping and control layer generates routing and bandwidth configuration according to a parallel strategy during a compiling period, and performs adaptive scheduling based on a link counter and congestion feedback during a running period; the collective communication acceleration layer internally embeds reduction and replication logic in the gateway, and maps a collective communication primitive into an optimal path and bandwidth allocation; the nodes in the chip are regularly arranged in a 2D grid in each layer, an upper layer node is formed through preset KxK grouping, and an N-layer hierarchical structure is recursively constructed; the application can realize a bandwidth gradient and a locally prioritized communication path, and improves support for collective communication.

Description

Technical Field

[0001] This invention relates to the field of on-chip network technology, and more specifically to a data flow-driven hierarchical 2D-Mesh on-chip network structure. Background Technology

[0002] LLM cloud training and inference typically employ a combination of multidimensional parallel strategies, including data parallelism (DP), tensor parallelism (TP), pipelined parallelism (PP), expert parallelism (EP / MoE), and sequence parallelism (SP). The communication of these various methods exhibits a hierarchical traffic characteristic of high local intensity and low cross-layer frequency.

[0003] The existing data center interconnect architecture UB-Mesh is based on the "hierarchical and localized" nD-FullMesh, emphasizing high bandwidth over short distances and low bandwidth over long distances. However, this architecture is designed for data center networks and rack / board-level hardware construction, relying on components such as switches and optoelectronic interconnects. It lacks on-chip 2D regular layout and wiring constraint modeling, making it difficult to directly map to a manufacturable on-chip network structure.

[0004] Among the publicly available hierarchical NoC schemes, some adopt a "lower-layer mesh, upper-layer fully interconnected or asymmetric mesh" structure, and achieve cross-domain communication through hierarchical routing or virtual routers. For example, CN109189720A uses a mesh subnet and an upper-layer fully interconnected virtual subnet; US9253085B2 introduces a hierarchical mesh to reduce hop count; and US10911261B2 adopts a global / local network hierarchy and supports broadcast / multicast. Overall, these schemes emphasize routing and topology hierarchy, but have limited support for the coordinated support of typical LLM data flows and aggregated communication primitives.

[0005] Common aggregation communication mechanisms in many-core systems include barrier, broadcast, reduce, scan / prefix-sum, and all-to-all, primarily used for synchronization, reduction, and data redistribution. Existing implementations are mostly at the software or library level (such as NCCL / MPI), lacking deep collaboration with the topology, routing, and bandwidth allocation of on-chip NoC.

[0006] In summary, the shortcomings of the existing technology are as follows:

[0007] Most existing hierarchical interconnect architectures originate from the design concepts of data centers or board-level systems. For example, UB-Mesh proposes "hierarchically localized nD-FullMesh" and emphasizes short-distance direct connections and data localization. However, it is designed for data center networks and rack / board-level hardware construction, relying on components such as switches, optoelectronic interconnects, and unified buses to organize bandwidth. It lacks on-chip 2D regular layout and wiring constraint modeling, making it difficult to directly map to a manufacturable on-chip network structure. When its concept is applied to the chip, upper-layer full interconnects or high-dimensional connections will put significant pressure on the number, length, and timing convergence of physical connections.

[0008] In the on-chip network patent, CN109189720A adopts a hierarchical structure of "a first-layer mesh virtual subnet and a second-layer fully interconnected virtual subnet," connecting each mesh subnet through full interconnection. This approach improves cross-subnet communication capabilities at the logical level, but the second-layer full interconnection does not consider on-chip layout and wiring constraints, easily leading to complex routing, uneven link lengths, and increased power consumption. Furthermore, its hierarchical structure of "virtual subnets and full interconnection" is difficult to replicate the regular 2D version of a real chip. Figure 1 One-to-one correspondence.

[0009] Solutions such as US9253085B2 and US10911261B2 focus on hierarchical routing, virtual routers, or bridging mechanisms for local / global networks, primarily addressing hierarchical addressing and route reachability issues. However, they do not establish bandwidth grading based on LLM data flows or structural constraints of "high bandwidth at lower layers and low bandwidth at upper layers." Their hierarchical organization and link configuration are still biased towards general designs, lacking native support for communication of large model sets and gateway-side aggregation / replication / reduction capabilities.

[0010] From an application workload perspective, existing NoCs are mostly geared towards general communication workloads and lack system-level adaptation to the parallel structure and data flow characteristics of LLM training and inference. Typical DP / TP / PP / EP / SP parallel modes correspond to diverse primitive combinations such as full reduction, full collection, reduction broadcast, all-to-all, and point-to-point activation streams, while traditional NoCs only provide point-to-point or simple broadcasts, lacking the ability to hierarchically orchestrate and accelerate aggregated communication at the gateway / routing level, making it difficult to fully leverage the data locality advantages of "local priority and cross-layer control".

[0011] In addition, existing solutions mostly assume homogeneous computing nodes in terms of node type and interface capabilities, lacking a unified topology planning and cross-layer scheduling strategy for heterogeneous computing / storage / communication tiles; in scale-up and scale-out scenarios, there is also a lack of a mechanism to integrate on-chip communication and off-chip interconnection into hierarchical scheduling, resulting in insufficient on-chip-off-chip collaboration efficiency and failing to meet the bandwidth and latency requirements of large-scale multi-chip / multi-package collaboration. Summary of the Invention

[0012] In view of the technical defects mentioned in the background art, the purpose of this invention is to provide a data flow-driven hierarchical 2D-Mesh on-chip network structure, which aims to at least partially solve one of the technical problems in the related art.

[0013] To achieve the above objectives, the present invention provides a data flow-driven hierarchical 2D-Mesh on-chip network structure, the structure comprising a node layer, a routing and gateway layer, a data flow mapping and control layer, and a collection communication acceleration layer;

[0014] The node layer comprises a heterogeneous tile array consisting of computing / storage / communication tiles, used to provide computing and data residency capabilities;

[0015] The routing and gateway layer is responsible for intra-layer forwarding and cross-layer aggregation, and is configured such that the upper-layer link bandwidth is lower than that of the lower layer to meet cabling and power consumption constraints.

[0016] The data flow mapping and control layer is used to generate routes and bandwidth configurations based on parallel strategies during compilation, and to perform adaptive scheduling based on link counters and congestion feedback during runtime.

[0017] The aggregated communication acceleration layer is used to embed reduction and replication logic in the gateway, mapping aggregated communication primitives to optimal paths and bandwidth allocation;

[0018] In this chip, nodes are arranged in a regular 2D mesh at each layer. Upper-layer nodes are formed by pre-set K×K grouping, and N-layer hierarchical structure is recursively constructed, where K and N are both positive integers greater than one. Local full connectivity is used in the lower-layer group to provide high bandwidth over short distances. Cross-layer communication is aggregated from the gateway to the upper-layer 2D-Mesh, thus forming a communication path with bandwidth gradient and local priority.

[0019] As a specific implementation of this application, the bottom-level nodes are arranged in a regular 2D grid to form a Tile array; each K×K Tile is grouped to form an upper-level node, and the upper-level nodes continue to form a 2D regular grid, recursively forming an N-layer structure; wherein, the node address adopts a field encoding method of hierarchical identifier, group coordinates, intra-group coordinates and node type to facilitate resolution and forwarding by intra-layer and cross-layer routers.

[0020] As a specific implementation of this application, each K×K group adopts local full connectivity to achieve low-latency, high-bandwidth data exchange; the local full connectivity is implemented using direct links or equivalent direct connectivity through small switches / logical interconnects to ensure cabling feasibility and maintain the advantage of low hop count.

[0021] As a specific implementation of this application, the routing and gateway layer is configured with gateway routers and intra-layer routers; each upper-layer node is configured with a gateway router, hereinafter referred to as the gateway, which is responsible for lower-layer traffic aggregation and upper-layer bandwidth control; the gateway includes an input queue, a congestion counter, a priority arbitrator and a cross-layer scheduler, and controls the upper-layer link bandwidth through port scheduling; it performs hierarchical and throttling on cross-layer traffic, so that the upper-layer link bandwidth is significantly less than that of the lower layer, thereby meeting cabling and power consumption constraints.

[0022] As one specific implementation of this application, the microarchitecture of the intra-layer router includes:

[0023] Each tile is configured with a router and network interface. The router uses a pipelined processing architecture with input buffers, virtual channels, arbitrators, and switching structures. The network interface is responsible for packet fragmentation, header parsing, and reassembly. Routing uses dimensional / XY routing or equivalent deterministic path selection. When the destination node is within the same group, intra-layer forwarding is completed directly. When the destination node is in a different group, it first goes to the gateway and then enters the upper-layer 2D-Mesh routing.

[0024] As a specific implementation of this application, the data flow mapping and control layer adopts a hybrid mechanism of compile-time estimation and runtime feedback in terms of control and scheduling;

[0025] During compilation, the traffic structure is estimated based on a parallel strategy, and a mapping table of link priorities, bandwidth budgets, routing priorities, and aggregated communication is generated.

[0026] During operation, bandwidth share and queue weight are adjusted based on link counters, congestion / hotspot monitoring and feedback control to achieve hotspot awareness and load balancing; lower-layer bandwidth is prioritized for local hotspot communication, and throttling / segmentation transmission strategies are adopted for cross-layer traffic to avoid upper-layer congestion; hotspot communication is prioritized to be completed locally or at lower layers, and cross-layer communication is carried out in a controlled manner.

[0027] As a specific implementation of this application, the aggregated communication acceleration layer is used to embed reduction and replication logic in the gateway, specifically including:

[0028] The gateway has a built-in CCL acceleration unit, which includes a set communication request parser, segmentation buffer, partial reduction unit, replication / distribution unit, and scheduling table / counter, and maintains session state by group_id and operation dimensions. Each set communication message carries group_id, op_code, segment_id, and epoch / seq fields on the network interface side. Among them, group_id identifies the communication group and the set of participating nodes, op_code specifies the primitive type, segment_id indicates the segment index, and epoch / seq is used for rounds and ordered delivery. The gateway uses this information to perform traffic splitting, out-of-order reordering, and ordered merging, and aligns the segments in the local cache.

[0029] Reduce operations first complete partial reduction within the group through local full connections, then the gateway aligns the segments and performs pipelined reduction. Partial reduction units can perform accumulation / comparison or other reduction operations on multiple inputs of the same segment and generate partial reduction results. When the counter confirms that the contribution of the segment is complete, the gateway sends the reduction result to the upper layer, or directly executes reduce-scatter to feed the reduction result back to the corresponding node in the lower layer according to the fragment mapping table. All-Gather and Broadcast are distributed to the lower layer by the gateway caching segments and according to the topology-aware replication tree or multicast table, supporting segmented pipelined and parallel fan-out. All-Reduce adopts a hierarchical path of local reduction, cross-layer reduction and distribution to reduce cross-layer round trips and reduce upper-layer link occupancy. All-to-All adopts segmentation and time-slice scheduling, combined with credit / queue gating and priority arbitration to avoid cross-layer conflicts and head-of-line blocking, and supports bandwidth quotas based on traffic matrix.

[0030] The on-chip many-core CCL generates primitive execution graphs and time slot tables through the CCL engine, clarifying the participating sets, segmentation order, and routing priority for each round. The gateway maps the orchestration results to the scheduling of local queues, cross-layer ports, and replication / reduction units, achieving fast convergence on set communication primitives and ensuring consistency and predictability of cross-layer and intra-layer execution. The set communication primitives include barrier synchronization, reduction, prefix scan reduction, and all-to-all communication.

[0031] As a specific implementation of this application, the communication tile serves as an on-chip / off-chip interconnection outlet, including scale-up and scale-out ports; the gateway and the communication tile work together to perform cross-layer and cross-chip communication scheduling to ensure that high-priority data streams are processed at the local level.

[0032] As a specific implementation of this application, cross-layer connections adopt shared link time-division multiplexing to reduce cabling resource consumption, or adopt local cross switches or multi-port switching structures to improve concurrency and fine-grained bandwidth control capabilities.

[0033] Intra-layer connections can add a small number of short connections or quasi-full connections to hot communication areas while retaining the 2D-Mesh skeleton.

[0034] The technical solution provided by the embodiments of the present invention improves the performance and feasibility of on-chip networks at both the structural and control levels, thereby bringing the following beneficial effects:

[0035] 1. Intra-layer local full connectivity and K×K grouping significantly shorten the physical path of common hotspot communication, reducing the average number of hops and end-to-end latency; for neighbor and intra-group communication that dominates LLM inference and training, data exchange can be completed without entering the upper layer link, improving local throughput and reducing the probability of upper layer congestion.

[0036] 2. Cross-layer aggregation is achieved through gateways and explicit bandwidth gradient settings, ensuring that the upper-layer link bandwidth is lower than that of the lower layer. This design suppresses the disorderly spread of long-distance communication and avoids runaway global interconnection area and power consumption. Compared with full upper-layer interconnection or irregular virtual subnets, the regular 2D layout and bandwidth gradation of this invention make wiring and timing convergence more controllable, improving the feasibility and yield of chip physical implementation.

[0037] 3. The data flow-driven compile-time estimation and runtime feedback mechanism can pre-allocate communication based on the DP / TP / PP / EP / SP parallel strategy, and make adaptive adjustments based on congestion counters and hotspot detection during runtime. This improves link utilization and load balancing capabilities without changing the hardware structure, and reduces global performance jitter caused by sudden hotspots.

[0038] 4. The gateway has built-in aggregation communication acceleration logic, which transforms primitives such as All-Reduce, Reduce-Scatter, All-Gather, Broadcast, and All-to-All into hierarchical paths of local reduction / replication followed by cross-layer aggregation / distribution. This significantly reduces the number of cross-layer round trips and invalid data movement, improving the execution efficiency of aggregation communication. Combined with the barrier / reduce / scan / all-to-all orchestration capabilities of the on-chip many-core CCL, it can form shorter and more stable synchronization and reduction paths in multi-core arrays, improving the throughput of training and inference while reducing synchronization overhead.

[0039] 5. Communication Tile and gateway collaboration provides a unified scheduling entry point for scale-up and scale-out scenarios, enabling on-chip, in-package and off-chip interconnects to be collaboratively optimized under the same hierarchical framework, improving the efficiency of multi-chip collaborative training and inference;

[0040] 6. The parameterized N-layer structure has good scalability and can adjust the number of layers and K value under different chip sizes, different packaging forms and different power budgets, thereby achieving a controllable balance between performance, routing and cost. This parameterized structure is also easy to evolve in tandem with subsequent process evolution (higher bandwidth memory, denser interconnects) and has continuous scalability. Attached Figure Description

[0041] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the accompanying drawings used in the description of the specific embodiments or the prior art will be briefly introduced below.

[0042] Figure 1 This is a schematic diagram of a data flow-driven hierarchical 2D-Mesh on-chip network structure provided in an embodiment of the present invention;

[0043] Figure 2 This is a schematic diagram of gateway aggregation and bandwidth gradient provided in an embodiment of the present invention;

[0044] Figure 3 This is a schematic diagram of a data flow-driven compile-time / runtime closed-loop control provided in an embodiment of the present invention;

[0045] Figure 4 This is a schematic diagram of a many-core CCL and gateway microarchitecture provided in an embodiment of the present invention;

[0046] Figure 5 This is a schematic diagram of a heterogeneous tile provided in an embodiment of the present invention. Detailed Implementation

[0047] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0048] It should be understood that, when used in this specification and the appended claims, the terms "comprising" and "including" indicate the presence of the described features, integrals, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0049] Throughout this specification, references to "an embodiment," "an example," or "an example" mean that a particular feature, structure, or characteristic described in connection with that embodiment or example is included in at least one embodiment of the invention. Therefore, the phrases "in an embodiment," "in an embodiment," "an example," or "an example" appearing in various places throughout the specification do not necessarily refer to the same embodiment or example. Furthermore, specific features, structures, or characteristics can be combined in one or more embodiments or examples in any suitable combination and / or sub-combination.

[0050] Network-on-Chip (NoC): On-chip network, used to connect multiple cores / multiple tiles within a chip via interconnect structures.

[0051] LLM (Large Language Model): Large model.

[0052] Gateway Router (GW): A cross-layer aggregation and routing node responsible for cross-layer bandwidth control and acceleration of aggregated communication.

[0053] Virtual Channel (VC): A virtual channel used to improve router throughput and avoid congestion.

[0054] Network Interface (NI): The network interface is responsible for packet fragmentation, reassembly, and protocol adaptation.

[0055] CCL (Collective Communication Library): A collection of collection communication primitives and their implementations.

[0056] DP (Data Parallel): Data parallelism, with typical communication being All-Reduce.

[0057] TP (Tensor Parallel): Tensor parallelism, with typical communication being All-Gather / Reduce-Scatter.

[0058] PP (Pipeline Parallel): Pipeline parallelism, with typical communication being point-to-point active transmission.

[0059] EP (Expert Parallel): Experts work in parallel, with typical communication being All-to-All (common in MoE).

[0060] SP (Sequence Parallel): Sequence parallelism, with typical communication being All-Gather.

[0061] MoE (Mixture of Experts): Hybrid expert model.

[0062] HBM (High Bandwidth Memory): High-bandwidth storage.

[0063] MC (Memory Controller): Memory access controller.

[0064] Mesh Network: A type of on-chip network topology.

[0065] reduce-scatter: reduction and dispersion is a core set communication primitive in distributed training.

[0066] All-Gather: Used to synchronize and aggregate data among multiple participating processes, it is a core set communication primitive in distributed training.

[0067] Broadcast: Broadcasting.

[0068] It should be noted that, unless otherwise stated, the technical terms used in this embodiment have the common meaning as understood in the relevant technical field.

[0069] The embodiments of the present invention are illustrated with relevant applications of large models, but are not intended to limit them.

[0070] Please refer to Figure 1 The present invention provides a data flow-driven hierarchical 2D-Mesh on-chip network structure, the structure including a node layer, a routing and gateway layer, a data flow mapping and control layer, and a collection communication acceleration layer;

[0071] The node layer comprises a heterogeneous tile array consisting of computing / storage / communication tiles, used to provide computing and data residency capabilities;

[0072] The routing and gateway layer is responsible for intra-layer forwarding and cross-layer aggregation, and is configured such that the upper-layer link bandwidth is lower than that of the lower layer to meet cabling and power consumption constraints.

[0073] The data flow mapping and control layer is used to generate routes and bandwidth configurations based on parallel strategies during compilation, and to perform adaptive scheduling based on link counters and congestion feedback during runtime.

[0074] The aggregated communication acceleration layer is used to embed reduction and replication logic in the gateway, mapping aggregated communication primitives to optimal paths and bandwidth allocation;

[0075] In this chip, nodes are arranged in a regular 2D mesh at each layer. Upper-layer nodes are formed by pre-set K×K grouping, and N-layer hierarchical structure is recursively constructed, where K and N are both positive integers greater than one. Local full connectivity is used in the lower-layer group to provide high bandwidth over short distances. Cross-layer communication is aggregated from the gateway to the upper-layer 2D-Mesh, thus forming a communication path with bandwidth gradient and local priority.

[0076] In this embodiment, the bottom-level nodes are arranged in a regular 2D grid to form a Tile array; each K×K Tile is grouped to form an upper-level node, and the upper-level nodes continue to form a 2D regular grid, recursively forming an N-layer structure; to support hierarchical addressing, the node address adopts a field-based encoding method of hierarchical identifier, group coordinates, intra-group coordinates and node type, so as to facilitate resolution and forwarding by intra-layer and cross-layer routers;

[0077] Reference Figure 1 Taking K as an example of 2, each K×K group uses local full connectivity to achieve low-latency, high-bandwidth data exchange. Local full connectivity is achieved through direct links or through small switches / logical interconnects to achieve equivalent direct connectivity, ensuring cabling feasibility and maintaining the advantage of low hop count. This achieves both intra-layer connectivity and local full connectivity.

[0078] Reference Figure 2 The routing and gateway layer is configured with gateway routers and intra-layer routers; each upper-layer node is configured with a gateway router, hereinafter referred to as gateway (i.e., gateway is short for gateway router), which is responsible for lower-layer traffic aggregation and upper-layer bandwidth control; the gateway includes an input queue, a congestion counter, a priority arbitrator and a cross-layer scheduler, and controls the upper-layer link bandwidth through port scheduling; it performs hierarchical and throttling on cross-layer traffic, so that the upper-layer link bandwidth is significantly less than that of the lower layer, thereby meeting cabling and power consumption constraints.

[0079] The microarchitecture of the intra-layer router includes:

[0080] Each tile is configured with a router and network interface. The router uses a pipelined processing architecture with input buffers, virtual channels, arbitrators, and switching structures. The network interface is responsible for packet fragmentation, header parsing, and reassembly. Routing uses dimensional / XY routing or equivalent deterministic path selection. When the destination node is within the same group, intra-layer forwarding is completed directly. When the destination node is in a different group, it first goes to the gateway and then enters the upper-layer 2D-Mesh routing.

[0081] In this embodiment, refer to Figure 3 The data flow mapping and control layer adopts a hybrid mechanism of compile-time estimation and runtime feedback in terms of control and scheduling.

[0082] During compilation, the traffic structure is estimated based on the computation graph / parallel strategy, and a link priority, bandwidth budget, routing priority and aggregate communication mapping table are generated; among which, the parallel strategy includes DP / TP / PP / EP / SP parallel strategy;

[0083] During operation, bandwidth share and queue weight are adjusted based on link counters, congestion / hotspot monitoring and feedback control to achieve hotspot awareness and load balancing; lower-layer bandwidth is prioritized for local hotspot communication, and throttling / segmentation transmission strategies are adopted for cross-layer traffic to avoid upper-layer congestion; hotspot communication is prioritized to be completed locally or at lower layers, and cross-layer communication is carried out in a controlled manner.

[0084] Furthermore, the aggregated communication acceleration layer is used to integrate reduction and replication logic within the gateway, referring to... Figure 4 Specifically, it includes:

[0085] The gateway has a built-in CCL acceleration unit, which includes a set communication request parser, segmentation buffer, partial reduction unit, replication / distribution unit, and scheduling table / counter, and maintains session state by group_id and operation dimensions. Each set communication message carries group_id, op_code, segment_id, and epoch / seq fields on the network interface side. Among them, group_id identifies the communication group and the set of participating nodes, op_code specifies the primitive type, segment_id indicates the segment index, and epoch / seq is used for rounds and ordered delivery. The gateway uses this information to perform traffic splitting, out-of-order reordering, and ordered merging, and aligns the segments in the local cache.

[0086] Reduce operations first complete partial reduction within the group through local full connections, then the gateway aligns the segments and performs pipelined reduction. Partial reduction units can perform accumulation / comparison or other reduction operations on multiple inputs of the same segment and generate partial reduction results. When the counter confirms that the contribution of the segment is complete, the gateway sends the reduction result to the upper layer, or directly executes reduce-scatter to feed the reduction result back to the corresponding node in the lower layer according to the fragment mapping table. All-Gather and Broadcast are distributed to the lower layer by the gateway caching segments and according to the topology-aware replication tree or multicast table, supporting segmented pipelined and parallel fan-out. All-Reduce adopts a hierarchical path of local reduction, cross-layer reduction and distribution to reduce cross-layer round trips and reduce upper-layer link occupancy. All-to-All adopts segmentation and time-slice scheduling, combined with credit / queue gating and priority arbitration to avoid cross-layer conflicts and head-of-line blocking, and supports bandwidth quotas based on traffic matrix.

[0087] The on-chip many-core CCL generates primitive execution graphs and time slot tables through the CCL engine, clarifying the participating sets, segmentation order, and routing priorities for each round. The gateway maps the orchestration results to the scheduling of local queues, cross-layer ports, and replication / reduction units, achieving fast convergence on set communication primitives and ensuring consistency and predictability of cross-layer and intra-layer execution. The set communication primitives include barrier synchronization, reduction, prefix scan reduction, and all-to-all communication (i.e., the corresponding English terms are barrier, reduce, scan / prefix-sum, and all-to-all).

[0088] In this embodiment, refer to Figure 5 It also has the function of coordinating heterogeneous nodes and communication nodes;

[0089] ComputeTile handles core computing; MemoryTile provides near-end storage and high-bandwidth access (SRAM / Cache / HBM / 3DMC); CommunicationTile serves as an on-chip / off-chip interconnect exit, including scale-up and scale-out ports. Gateways and communication tiles work together to schedule cross-layer and cross-chip communication, ensuring that high-priority LLM data streams are processed at the local level.

[0090] Furthermore, in other embodiments, based on the above technical solutions, without changing the overall framework of intra-layer rule 2D layout and cross-layer gateway aggregation, cross-layer connections adopt shared link time-division multiplexing to reduce cabling resource consumption, or adopt local cross switches or multi-port switching structures to improve concurrency and fine-grained bandwidth control capabilities.

[0091] Intra-layer connectivity can retain the 2D-Mesh skeleton while adding a small number of short or quasi-full-connection structures for hot communication areas to replace physical full connectivity and achieve a balance between feasibility and performance. For scenarios with a large K value, a combination of logical full connectivity and physical sparse connectivity can be used, achieving local full connectivity through small switching structures within the router or multi-hop short connections.

[0092] In summary, the key synergies of this invention are reflected in the following aspects:

[0093] Firstly, it adopts a regular 2D layout and a parameterized N-layer structure, which makes the topology naturally match the chip layout and wiring constraints, and can be flexibly expanded according to scale.

[0094] Secondly, by using K×K grouping and local full connectivity within the layer, short-range high-bandwidth and low-latency communication is achieved while maintaining physical feasibility;

[0095] Third, cross-layer traffic is aggregated through gateways to form a bandwidth gradient (high bandwidth at the lower layer and low bandwidth at the upper layer), which structurally controls cross-layer traffic and global interconnection costs.

[0096] Fourth, a data flow-driven joint scheduling mechanism of "compile-time estimation superimposed with runtime feedback" is introduced to achieve dynamic allocation of link resources and congestion adaptation.

[0097] Fifth, the gateway has built-in collection communication acceleration (reduction / replication / distribution / scheduling), which maps typical LLM communication primitives to hierarchical execution paths and reduces cross-layer round trips;

[0098] Sixth, it supports heterogeneous compute / storage / communication tiles and scale-up / scale-out collaboration, enabling on-chip and off-chip interconnects to be optimized under a unified hierarchical framework.

[0099] It should be noted that the parameterized N-layer structure refers to the following: the on-chip network starts from the bottom tile array and recursively constructs upper-layer nodes layer by layer according to the preset grouping size K until the Nth layer hierarchical structure is formed, where N represents the total number of layers and K represents the grouping size of each layer; N and K are both configurable parameters that can be selected according to chip size, wiring resources, power budget, and target communication bandwidth; by adjusting N, a balance can be achieved between local communication efficiency, cross-layer communication overhead, global wiring complexity, and scalability, so that the present invention can be adapted to both small-scale on-chip systems and large-scale multi-tile / multi-package computing chips, and each layer can be configured with different bandwidths or the same grouping rules.

[0100] The above solution, by simultaneously improving the performance and feasibility of on-chip networks at both the structural and control levels, brings the following beneficial effects:

[0101] 1. Intra-layer local full connectivity and K×K grouping significantly shorten the physical path of common hotspot communication, reducing the average number of hops and end-to-end latency; for neighbor and intra-group communication that dominates LLM inference and training, data exchange can be completed without entering the upper layer link, improving local throughput and reducing the probability of upper layer congestion.

[0102] 2. Cross-layer aggregation is achieved through gateways and explicit bandwidth gradient settings, ensuring that the upper-layer link bandwidth is lower than that of the lower layer. This design suppresses the disorderly spread of long-distance communication and avoids runaway global interconnection area and power consumption. Compared with full upper-layer interconnection or irregular virtual subnets, the regular 2D layout and bandwidth gradation of this invention make wiring and timing convergence more controllable, improving the feasibility and yield of chip physical implementation.

[0103] 3. The data flow-driven compile-time estimation and runtime feedback mechanism can pre-allocate communication based on the DP / TP / PP / EP / SP parallel strategy, and make adaptive adjustments based on congestion counters and hotspot detection during runtime. This improves link utilization and load balancing capabilities without changing the hardware structure, and reduces global performance jitter caused by sudden hotspots.

[0104] 4. The gateway has built-in aggregation communication acceleration logic, which transforms primitives such as All-Reduce, Reduce-Scatter, All-Gather, Broadcast, and All-to-All into hierarchical paths of local reduction / replication followed by cross-layer aggregation / distribution. This significantly reduces the number of cross-layer round trips and invalid data movement, improving the execution efficiency of aggregation communication. Combined with the barrier / reduce / scan / all-to-all orchestration capabilities of the on-chip many-core CCL, it can form shorter and more stable synchronization and reduction paths in multi-core arrays, improving the throughput of training and inference while reducing synchronization overhead.

[0105] 5. Communication Tile and gateway collaboration provide a unified scheduling entry point for scale-up and scale-out scenarios, enabling on-chip, in-package and off-chip interconnects to be collaboratively optimized under the same hierarchical framework, improving the efficiency of multi-chip collaborative training and inference;

[0106] 6. The parameterized N-layer structure has good scalability and can adjust the number of layers and K value under different chip sizes, different packaging forms and different power budgets, thereby achieving a controllable balance between performance, routing and cost. This parameterized structure is also easy to evolve in tandem with subsequent process evolution (higher bandwidth memory, denser interconnects) and has continuous scalability.

[0107] It should be noted that this invention is applicable to dedicated computing chips for large-scale model training and inference, and is particularly suitable for Transformer and MoE models employing parallel strategies such as DP / TP / PP / EP / SP. Intra-layer local full connectivity and gateway aggregation can limit aggregated communications such as All-Reduce, All-Gather, Reduce-Scatter, and All-to-All to be completed at lower layers as much as possible, and shorten communication paths through gateway-side reduction and replication logic, thereby reducing end-to-end latency and improving training throughput and batch processing capabilities for inference.

[0108] This invention is applicable to large model training and inference scenarios involving multi-chip / multi-package scale-up and cross-server scale-out. The communication tile, serving as the on-chip and off-chip interconnect exit, can work with the gateway to handle cross-layer and cross-chip bandwidth orchestration and traffic shaping, enabling on-chip 2D regular networks, high-speed in-package interconnects, and off-chip networks to work collaboratively under a unified hierarchical scheduling. It is suitable for chiplets, 2.5D interconnects, and 3D stacking packaging forms.

[0109] This invention is also applicable to heterogeneous SoCs and high-bandwidth storage systems. The division of labor between computing, storage, and communication tiles enables near-end storage (SRAM / Cache / HBM / 3D MC) to form a tighter data localization path with the computing core. Local high-bandwidth grids and cross-layer bandwidth gradients reduce long-distance data movement, which is more effective for memory-constrained and sparse models.

[0110] For power- and cost-sensitive systems, the bandwidth gradient and "compile-time estimation plus runtime feedback" mechanism of this invention can limit cross-layer link load while ensuring high local bandwidth, reduce global interconnect area and power consumption, and is suitable for data center AI accelerators, edge inference chips, and general-purpose parallel computing platforms that need to achieve scalable interconnect within limited resources.

[0111] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A dataflow-driven hierarchical 2D-Mesh network-on-chip architecture, characterized in that, The structure includes a node layer, a routing and gateway layer, a data flow mapping and control layer, and a collection communication acceleration layer; The node layer comprises a heterogeneous tile array consisting of computing / storage / communication tiles, used to provide computing and data residency capabilities; The routing and gateway layer is responsible for intra-layer forwarding and cross-layer aggregation, and is configured such that the upper-layer link bandwidth is lower than that of the lower layer to meet cabling and power consumption constraints. The data flow mapping and control layer is used to generate routes and bandwidth configurations based on parallel strategies during compilation, and to perform adaptive scheduling based on link counters and congestion feedback during runtime. The aggregated communication acceleration layer is used to embed reduction and replication logic in the gateway, mapping aggregated communication primitives to optimal paths and bandwidth allocation; In this chip, nodes are arranged in a regular 2D mesh at each layer. Upper-layer nodes are formed by pre-set K×K grouping, and N-layer hierarchical structure is recursively constructed, where K and N are both positive integers greater than one. Local full connectivity is used in the lower-layer group to provide high bandwidth over short distances. Cross-layer communication is aggregated from the gateway to the upper-layer 2D-Mesh, thus forming a communication path with bandwidth gradient and local priority.

2. The dataflow-driven hierarchical 2D-Mesh network-on-chip structure of claim 1, wherein, The bottom layer nodes are arranged in a regular 2D grid to form a tile array; each K×K tile is grouped to form an upper-layer node, and the upper layer continues to form a 2D regular grid, recursively forming an N-layer structure; among them, the node address adopts a field encoding method of hierarchical identifier, group coordinates, intra-group coordinates and node type to facilitate resolution and forwarding by intra-layer and cross-layer routers.

3. The dataflow-driven hierarchical 2D-Mesh network-on-chip structure of claim 2, wherein, Each K×K group employs local full connectivity to achieve low-latency, high-bandwidth data exchange. Local full connectivity is achieved through direct links or through small switches / logical interconnects to ensure cabling feasibility and maintain the advantage of low hop count.

4. The dataflow-driven hierarchical 2D-Mesh network-on-chip structure of claim 3, wherein, The routing and gateway layer is configured with gateway routers and intra-layer routers; each upper-layer node is configured with a gateway router, hereinafter referred to as the gateway, which is responsible for lower-layer traffic aggregation and upper-layer bandwidth control; the gateway includes an input queue, a congestion counter, a priority arbitrator and a cross-layer scheduler, and controls the upper-layer link bandwidth through port scheduling; it performs hierarchical and throttling on cross-layer traffic, so that the upper-layer link bandwidth is significantly less than that of the lower layer, thereby meeting cabling and power consumption constraints.

5. The dataflow-driven hierarchical 2D-Mesh network-on-chip structure of claim 4, wherein, The microarchitecture of the intra-layer router includes: Each tile is configured with a router and network interface. The router uses a pipelined processing architecture with input buffers, virtual channels, arbitrators, and switching structures. The network interface is responsible for packet fragmentation, header parsing, and reassembly. Routing uses dimensional / XY routing or equivalent deterministic path selection. When the destination node is within the same group, intra-layer forwarding is completed directly. When the destination node is in a different group, it first goes to the gateway and then enters the upper-layer 2D-Mesh routing.

6. The dataflow-driven hierarchical 2D-Mesh network-on-chip structure of claim 4, wherein, The data flow mapping and control layer employs a hybrid mechanism of compile-time estimation and runtime feedback in its control and scheduling. During compilation, the traffic structure is estimated based on a parallel strategy, and a mapping table of link priorities, bandwidth budgets, routing priorities, and aggregated communication is generated. During operation, bandwidth share and queue weight are adjusted based on link counters, congestion / hotspot monitoring and feedback control to achieve hotspot awareness and load balancing; lower-layer bandwidth is prioritized for local hotspot communication, and throttling / segmentation transmission strategies are adopted for cross-layer traffic to avoid upper-layer congestion; hotspot communication is prioritized to be completed locally or at lower layers, and cross-layer communication is carried out in a controlled manner.

7. The dataflow-driven hierarchical 2D-Mesh network-on-chip structure of claim 6, wherein, The aggregated communication acceleration layer is used to integrate reduction and replication logic into the gateway, specifically including: The gateway has a built-in CCL acceleration unit, which includes a set communication request parser, segmentation buffer, partial reduction unit, replication / distribution unit, and scheduling table / counter, and maintains session state by group_id and operation dimensions. Each set communication message carries group_id, op_code, segment_id, and epoch / seq fields on the network interface side. Among them, group_id identifies the communication group and the set of participating nodes, op_code specifies the primitive type, segment_id indicates the segment index, and epoch / seq is used for rounds and ordered delivery. The gateway uses this information to perform traffic splitting, out-of-order reordering, and ordered merging, and aligns the segments in the local cache. Reduce operations first complete partial reduction within the group through local full connections, then the gateway aligns the segments and performs pipelined reduction. Partial reduction units can perform accumulation / comparison or other reduction operations on multiple inputs of the same segment and generate partial reduction results. When the counter confirms that the contribution of the segment is complete, the gateway sends the reduction result to the upper layer, or directly executes reduce-scatter to feed the reduction result back to the corresponding node in the lower layer according to the fragment mapping table. All-Gather and Broadcast are distributed to the lower layer by the gateway caching segments and according to the topology-aware replication tree or multicast table, supporting segmented pipelined and parallel fan-out. All-Reduce adopts a hierarchical path of local reduction, cross-layer reduction and distribution to reduce cross-layer round trips and reduce upper-layer link occupancy. All-to-All adopts segmentation and time-slice scheduling, combined with credit / queue gating and priority arbitration to avoid cross-layer conflicts and head-of-line blocking, and supports bandwidth quotas based on traffic matrix. The on-chip many-core CCL generates primitive execution graphs and time slot tables through the CCL engine, clarifying the participating sets, segmentation order, and routing priority for each round. The gateway maps the orchestration results to the scheduling of local queues, cross-layer ports, and replication / reduction units, achieving fast convergence on set communication primitives and ensuring consistency and predictability of cross-layer and intra-layer execution. The set communication primitives include barrier synchronization, reduction, prefix scan reduction, and all-to-all communication.

8. A dataflow-driven hierarchical 2D-Mesh network-on-chip structure according to any one of claims 4 to 7, characterized in that, The communication tile serves as an on-chip / off-chip interconnection outlet, including scale-up and scale-out ports; the gateway and the communication tile work together to perform cross-layer and cross-chip communication scheduling to ensure that high-priority data streams are processed at the local level.

9. The dataflow-driven hierarchical 2D-Mesh network-on-chip structure of claim 8, wherein, Cross-layer connections employ shared link time-division multiplexing to reduce cabling resource consumption, or use local cross switches or multi-port switching structures to improve concurrency and fine-grained bandwidth control capabilities. Intra-layer connections can add a small number of short connections or quasi-full connections to hot communication areas while retaining the 2D-Mesh skeleton.

Images