A hybrid on-chip interconnection network architecture for MoBA-gated accelerators

Abstract

The application belongs to the field of integrated circuit design and relates to a hybrid on-chip interconnection network architecture for MoBA gate-controlled accelerator, which comprises macro, meso and micro interconnection networks, and hierarchical collaborative data flow transmission links are formed between the interconnection networks; the macro interconnection network adopts a hybrid topology structure combining central radiation and two-dimensional grid, is used for distributing input data to the meso interconnection network in a multicast manner, and outputs the screening result of the meso interconnection network to the display memory interaction core in a multi-path convergence manner; the meso interconnection network adopts a star and tree hybrid topology, is used for distributing data to the micro interconnection network, and returns the output of the micro interconnection network to the macro interconnection network; the micro interconnection network adopts an H-shaped tree fractal interconnection topology, is used for performing operation, and performs cascade accumulation on the partial sum and reduction data obtained by operation to obtain the screening result; and the overall computing throughput and energy efficiency of the heterogeneous core particle system are effectively improved.

Description

Technical Field

[0001] This invention relates to the field of integrated circuit design, and specifically discloses a hybrid on-chip interconnect network architecture for MoBA gated accelerators. Background Technology

[0002] As the scale of large language models (LLMs) continues to expand, sparse computing architectures based on expert hybrid or similar mechanisms are gradually becoming a key approach to improve model inference efficiency. The MoBA (Mixture-of-Block-Attention) algorithm, as an efficient attention mechanism, places extremely high demands on the parallel processing capabilities of the underlying hardware accelerators. In the design of gated accelerators for the MoBA algorithm, to cope with the large-scale parallel processing requirements of up to 128 computational blocks based on in-memory computing technology, a hierarchical and efficient interconnect network must be constructed.

[0003] Because the logic within the accelerator is divided into multiple layers, the internal data flow exhibits significant multi-granularity communication characteristics and phased changes. For example, in the input phase, the k and q data streams carrying the attention mechanism need to be broadcast or unicast to a specific destination block with high fan-out. This phase is characterized by a single source, high fan-out, and high throughput. After local computation is completed, the generated partial sum (PSUM) data stream needs to undergo many-to-one, high-concurrency spatial aggregation and reduction. The final sparse mask data stream needs to be written back to the output. This communication mode switches between broadcast and unicast modes in different computation phases.

[0004] However, traditional Network-on-Chip (NoC) topologies struggle to efficiently adapt to complex data flows with inherently hotspots and uniform distribution. The widely adopted two-dimensional mesh topology is ill-suited to this highly concentrated traffic pattern. In traditional mesh architectures, numerous horizontal and vertical links used for interconnecting peripheral edge nodes remain idle for extended periods, resulting in significant waste of chip area and static power consumption. More seriously, in a single flat mesh scheme, because transit traffic is primarily concentrated on specific backbone links, extreme network load imbalances and severe head-blocking effects are highly likely to occur, leading to global network deadlock and extremely high end-to-end latency.

[0005] Taking a traditional two-dimensional flat mesh on-chip network scheme as an example, assuming a standard 4x4 mesh topology is used to interconnect 16 computation blocks within a group, the network will expose a severe spatial traffic imbalance problem when executing the data flow of the gated operation circuit. During the algorithm's input phase, the k-stream or q-stream carrying the attention mechanism needs to be multicast or precisely unicast from a single network injection node to these 16 blocks. At this time, the routers adjacent to the injection source and their backbone links will experience extremely high concurrent fan-out pressure. Furthermore, when the computation enters the Merge phase, after completing the local inner product operation, these 16 blocks will simultaneously generate massive amounts of PSUM data, and this data needs to be spatially aggregated at high concurrency to specific TopK circuit nodes. This typical many-to-one communication mode forces all reduction traffic within the group to be highly concentrated on a few backbone links and the final aggregation node. The final local network state exhibits extreme polarization: routers located at data injection sources, aggregation centers, or backbone forwarding paths have traffic loads far exceeding the physical bandwidth limit, which can easily lead to severe queuing congestion and head-blocking effects, directly becoming the bottleneck restricting system throughput; in contrast, nodes located at the physical edge of the mesh or on non-backbone paths and their interconnection links are idle or have extremely low loads for a long time due to a lack of traffic.

[0006] Based on the quantitative analysis of gated accelerator data streams, existing conventional interconnect architectures face three major technical challenges: First, the high fan-out distribution and traffic redundancy of the input data stream. The input data stream includes the k-data stream and q-data stream carrying the attention mechanism, characterized by a single source, high fan-out, and high throughput. The k-data stream needs to be broadcast to a specific cluster and then replicated to all blocks within the cluster for storage, while the q-data stream flows as input data into the destination block for inner product operations. Taking the sequence length as S as an example, if the network can effectively utilize the cluster-level data reuse feature, the total injected traffic is 160S. If the interconnect architecture of the input data stream adopts a flat design, i.e., equipping 128 blocks with independent routers, without utilizing cluster-level data reuse, the total injected traffic will surge to 256S. This redundant traffic significantly increases the total injected load by 60%, not only greatly increasing the transmission cost of the input network but also leading to a sharp deterioration in system latency and power consumption. Second, the high-concurrency aggregation and high bandwidth pressure of the merge data stream. After the target block completes the local inner product calculation, the resulting partial sum (PSUM) data, as a merge data stream, needs to be accumulated with PSUM data from other corresponding blocks, and finally converged as the input of the TopK module. This process exhibits typical many-to-one, high-concurrency local convergence characteristics. During this process, the total merge traffic that the network needs to carry reaches 256 seconds, placing higher demands on the bandwidth of the on-chip network. Without targeted physical isolation design, the massive intermediate calculation results will cause serious network congestion bottlenecks. Third, there is the problem of dynamic heterogeneous communication matching at the micro-cell level. A single block integrates 32 basic computing cells, whose communication patterns include point-to-point distribution of independent k data and one-to-one full broadcast of shared q data, as well as the convergence of the output data of these cells. Different heterogeneous data distribution modes exist at different stages of the gated operation, which poses design requirements for the micro-interconnection topology within the block. Traditional simple internal buses are difficult to balance the contradictions of signal driving capability, wiring congestion, and timing alignment within a limited area.

[0007] In view of this, the present invention provides a hybrid on-chip interconnect network architecture for MoBA-gated accelerators, enabling the complex cross-layer, multi-mode communication needs of MoBA-gated accelerators to be met through a single traditional bus or simple on-chip network architecture. It can design hybrid topology networks that simultaneously support flexible routing and high-concurrency multicast for different physical layers, achieving non-blocking distribution and aggregation of tensor data streams at different granularities, thereby effectively improving the overall computing throughput and energy efficiency of heterogeneous chip systems. Summary of the Invention

[0008] The purpose of this invention is to provide a hybrid on-chip interconnect architecture for MoBA-gated accelerators, addressing the problems of multi-granularity, heterogeneous communication congestion and traffic redundancy arising from different computational stages in the gated accelerator circuit. Specifically, it includes:

[0009] A hybrid on-chip interconnect network architecture for MoBA-gated accelerators includes a macro-level interconnect network, a meso-level interconnect network, and a micro-level interconnect network, with hierarchical and collaborative data flow transmission links between each interconnect network. The macro-level interconnect network adopts a hybrid topology combining a central radial structure and a two-dimensional grid, used to distribute input data to the meso-level interconnect network via multicast, and output the filtering results returned by the meso-level interconnect network to the memory interaction core via multi-path aggregation. The meso-level interconnect network adopts a hybrid star and tree topology, used to distribute received data to the micro-level interconnect network, and return the filtering results output by the micro-level interconnect network to the macro-level interconnect network. The micro-level interconnect network adopts an H-shaped tree-like fractal interconnect topology, used to perform calculations on the received data, and concatenate and accumulate the partial and reduced data obtained from the calculations to obtain the filtering results.

[0010] Furthermore, the macro-interconnection network includes multiple macro groups and multiple macro routers; the macro groups include a central macro group and multiple peripheral macro groups; the macro routers include a central router and multiple peripheral routers, and the macro routers construct a backbone transmission network through a two-dimensional grid; the central macro group and the central router serve as the core hub for global data distribution and aggregation, used to distribute the input data stream from the memory interaction core to each peripheral macro group and peripheral router through a tree-structured divergent multicast routing mechanism, and to output the returned output data stream to the memory interaction core through a multi-path aggregation mechanism; the peripheral macro groups and their corresponding peripheral routers serve as data relay nodes between the macro interconnection network and the meso interconnection network, used to receive the input data stream and distribute it to the internal meso interconnection network, and to forward the output data stream returned from the meso interconnection network to the central macro group.

[0011] Furthermore, the macro router adopts a virtual channel-based multiple-input multiple-output architecture; the central router adopts a multi-port structure to connect the video memory interaction core and multiple peripheral macro groups; the multi-port structure has 5 or more ports; the peripheral router adopts a three-port structure to connect the central router, the meso-level interconnection network within the same macro group, and adjacent peripheral routers.

[0012] Furthermore, the data path of the macro router is equipped with a cross switch acceleration module; the cross switch acceleration module is used to concurrently output two micro-chips to the same physical output port each cycle during the multi-path aggregation stage of the output data stream.

[0013] Furthermore, the meso-level interconnection network is located within the macro-level group and includes a meso-level router, multiple computing blocks, and a filtering circuit. The meso-level router is used for centralized distribution and aggregation of regular tensor data, sending the input data stream to the computing blocks and forwarding the data stream returned by the computing blocks to the macro-level interconnection network. The computing blocks are used to receive the data stream sent by the meso-level router and accumulate the calculated partial and reduced data across the computing block space through a direct connection network independent of the meso-level router. The filtering circuit is used to receive the accumulated partial and reduced data, perform global sorting and sparse reduction, generate filtering results, and inject the filtering results into the meso-level router.

[0014] Furthermore, the mesoscopic router employs a non-standard cross-switch matrix; the input side of the mesoscopic router includes multiple cross-macrogroup input ports and filtering circuit input ports, the filtering circuit input ports being used to receive filtering results unidirectionally injected by the local filtering circuit; the output side of the mesoscopic router includes multiple output ports connected to the local computing block and multiple output ports connected to other mesoscopic routers.

[0015] Furthermore, the micro-interconnect network is located within the computing block and includes multiple adding units and multiple computing units. The computing units are used to receive and perform parallel computations on the distributed input data stream during the data input phase to generate partial and reduced data. The adding units are used to receive externally input reduced data during the reduction phase and cascade and accumulate it with the partial and reduced data output by the computing units. The micro-interconnect network physically isolates the data distribution network and the partial and reduced network, and dynamically switches between point-to-point unicast mode and global synchronous broadcast mode according to the computing phase through a gated distribution microarchitecture.

[0016] Furthermore, the gated distribution microarchitecture of the micro-interconnected network includes gated control logic, a data delivery network, and an AND gate array; the gated control logic includes a one-hot code ring counter and an all-one vector generator, which are switched by an external Mode signal; the AND gate array is set on each parallel branch of the data delivery network and is controlled by the chip select signal output by the gated control logic, used to selectively activate the input link of the target computing unit in point-to-point unicast mode, and to simultaneously activate the input links of all computing units in global synchronous broadcast mode.

[0017] Furthermore, the micro-interconnect network also includes a serial-in parallel-out module and a parallel-in serial-out module; the serial-in parallel-out module is used to receive externally input standard data, complete the serial-to-parallel format conversion, and then send the parallel data to each added unit.

[0018] Furthermore, the micro-interconnect network spatially isolates the data distribution network and the partial sum and reduction network in terms of physical cabling; the data distribution network adopts an H-shaped tree fractal topology to radially and synchronously distribute input data to each computing unit; the partial sum and reduction network adopts an H-shaped tree fractal topology symmetrical to the data distribution network to aggregate the partial sum and reduction data generated by each computing unit to the addition unit for cascading accumulation.

[0019] The present invention has the following advantages and beneficial effects:

[0020] This invention can significantly alleviate network congestion, achieve global traffic load balancing, and effectively utilize central radiating links to share the communication pressure of hotspot nodes. By efficiently distributing data to the meso-level cluster-level interconnection network through macro-level group-level routing, it significantly reduces the communication congestion problem inherent in traditional cluster-based communication units. This architecture eliminates the extreme head-blocking effect faced by a single node and its adjacent backbone link in a traditional flat grid, thus effectively solving the network traffic balancing problem.

[0021] This invention reduces end-to-end communication latency through deep optimization at the topology and microarchitecture levels, exhibiting superior latency performance under MoBA-gated data flow. The average packet latency of this invention's architecture is reduced by approximately 99.6% compared to the corner-injection flat mesh scheme and by approximately 91.0% compared to the center-injection single-layer mesh scheme, improving the adaptability and superiority of this design architecture under large-scale tensor data concurrent flow. Attached Figure Description

[0022] Figure 1 An exemplary structural diagram of the macroscopic interconnection network provided by the present invention;

[0023] Figure 2 The data flow diagram of the macroscopic interconnection network provided by this invention;

[0024] Figure 3 This is an exemplary structural diagram of the mesoscopic interconnection network provided by the present invention;

[0025] Figure 4 This is an exemplary structural diagram of the micro-interconnected network provided by the present invention;

[0026] Figure 5 An exemplary structural diagram of a macro router provided by the present invention;

[0027] Figure 6 This is an exemplary structural diagram of a mesoscopic router provided by the present invention;

[0028] Figure 7 An exemplary structural diagram of the gated distribution microarchitecture provided by the present invention;

[0029] Figure label: 1-Macro group (Group), 2-Macro group router, 3-Meso group (Cluster), 4-Meso group router, 5-Computation block (Block), 6-Filtering circuit (TopK circuit), 7-q data, k data and / or mask data, 8-Merge data, 9-Computation unit (Cell), 10-Add unit (Add). Detailed Implementation

[0030] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0031] The hybrid on-chip interconnect network architecture for MoBA-gated accelerators of this invention is deeply integrated with each computational stage of the MoBA algorithm in actual operation. To support efficient scheduling of complex data streams, this invention employs a customized router microarchitecture at both the macro and meso levels. This invention solves the traffic redundancy problem caused by high fan-out distribution of input data streams, mitigates the bandwidth pressure caused by high-concurrency aggregation of merge data streams, and achieves efficient matching of dynamic heterogeneous communication at the micro-cell level. The specific workflow of the entire data stream is as follows:

[0032] The macro-level interconnection network adopts a hybrid topology combining a central radial structure and a two-dimensional grid. It is used to distribute input data to the meso-level interconnection network in a multicast manner, and output the filtering results returned by the meso-level interconnection network to the memory interaction core in a multi-path aggregation manner.

[0033] A hybrid topology combining a central radial structure and a two-dimensional grid refers to a structure where a centrally located core module for video-memory interaction serves as the global data source and final aggregation point. Peripheral computing groups are distributed radially around this core module. Each macro-group connects to the backbone transmission network via a macro-group router. Simultaneously, adjacent macro-routers are interconnected through two-dimensional grid links, providing alternative paths for decentralized communication across macro-groups. For example... Figure 1As shown, a point-to-point direct link is established between the memory interaction core module and eight Group routers to form a central radial structure. Multicast refers to a transmission method where a single data packet originates from a source node, is replicated by network nodes, and is simultaneously sent to multiple target nodes. In a specific embodiment, data packets carrying k data streams are sent from the central memory interaction core, replicated by the central router, and simultaneously distributed to eight peripheral computing Groups. The filtering result refers to the sparse mask data generated after global sorting and sparse reduction. Multipath aggregation refers to a transmission method where data packets from multiple source nodes converge to the same target node through different paths. In a specific embodiment, the mask data packets generated by the eight peripheral computing Groups converge towards the central memory interaction core along their respective shortest paths. The memory interaction core refers to the memory interaction module located at the center of the macroscopic interconnection network, including a memory interface and a buffer, responsible for data exchange with external memory, serving as the source of global input data and the destination of output data.

[0034] The macro-interconnect network comprises multiple macro groups and multiple macro routers. Each macro group includes a central macro group and multiple peripheral macro groups; each macro router includes a central router and multiple peripheral routers. These macro routers form a backbone transmission network via a two-dimensional grid. For example, each macro group contains four nested clusters and connects to the backbone transmission network via a group router. The central macro group and central router serve as the core hub for global data distribution and aggregation. They distribute the input data stream from the memory interaction core to each peripheral macro group and peripheral router via a tree-structured divergent multicast routing mechanism, and output the returned output data through a multi-path aggregation mechanism back to the memory interaction core.

[0035] Tree-based divergent multicast routing refers to a routing method where data packets originate from the root node, are copied and distributed level by level along the tree topology to all leaf nodes. For example... Figure 2 As shown, the k data stream is injected from the central memory interaction core. The central router replicates and distributes the data packets to four peripheral Group routers. Each peripheral Group router continues to replicate and distribute the packets to its four internal Cluster routers, ultimately reaching all compute nodes. The multi-path aggregation mechanism refers to a routing method where data packets from multiple leaf nodes converge hierarchically upwards along a tree topology to the root node. For example, as... Figure 2 As shown, the mask data stream originates from each computing node, converges upwards along the original distribution path, and is finally forwarded by the peripheral Group router to the central router, and then output to the video memory interaction core.

[0036] The peripheral macro group and its corresponding peripheral routers serve as data relay nodes between the macro interconnection network and the meso interconnection network. They are used to receive input data streams and send them to the internal meso interconnection network, and forward the output data streams returned by the meso interconnection network to the central macro group.

[0037] The macro routers employ a virtual channel-based multiple-input multiple-output (MIMO) architecture; the central router uses a multi-port structure to connect the memory interaction core and multiple peripheral macro groups; the multi-port structure has 5 or more ports; the peripheral routers use a three-port structure to connect the central router, the meso-level interconnection network within the same macro group, and adjacent peripheral routers. The macro routers use a central five-port and radial path three-port structure to reduce global hardware overhead while ensuring core throughput. The virtual channel-based MIMO architecture means that the router internally configures multiple independent virtual channel queues for each physical input port. Different data streams can be allocated to different virtual channels for independent buffering and scheduling, thereby avoiding head-blocking. For example, ... Figure 5 As shown, after the data chip enters from the physical input port on the left, it is guided to the corresponding virtual channel queue (VC#0 to VC#N) according to its virtual channel ID. Each queue is independently controlled and scheduled.

[0038] The macro router's data path includes a crossbar switch acceleration module. This module is used to concurrently output two microchips to the same physical output port each cycle during the multi-path aggregation phase of the output data stream. For example, when multiple peripheral macro groups simultaneously transmit mask data back to the central router, the crossbar switch acceleration module allows microchips from two different input ports to be switched to the same output port within the same clock cycle, effectively alleviating the throughput bottleneck caused by multi-path aggregation.

[0039] The meso-level interconnection network adopts a hybrid star and tree topology to distribute received data to the micro-level interconnection network and send the filtering results output by the micro-level interconnection network back to the macro-level interconnection network.

[0040] A hybrid star-and-tree topology refers to a structure where, within a cluster, four compute blocks are connected point-to-point through a central meso-level router, forming a star topology. Simultaneously, for high-concurrency components and reduced data, a separate point-to-point direct connection network, independent of the meso-level (cluster) router, is constructed, forming a tree-like aggregation structure. For example, ... Figure 3 As shown, the Cluster router and the four Blocks are connected in a star topology for regular data distribution, while the Blocks are connected in a tree structure through independent direct links for spatial accumulation of parts and data.

[0041] The meso-level interconnection network is located within the macro-level group and includes meso-level routers, multiple computation blocks, and filtering circuits. The meso-level routers are used for centralized distribution and aggregation of regular tensor data, sending input data streams to computation blocks and forwarding the data streams returned by the computation blocks to the macro-level interconnection network. Regular tensor data refers to the k-stream, q-stream, and mask-stream indicating computation results used in the MoBA gating algorithm, which carry the attention mechanism. These data streams have deterministic routing characteristics and do not require independent physical link isolation. Centralized distribution and aggregation means that all regular tensor data is uniformly scheduled and forwarded through the central cluster router. The router is responsible for distributing input data to target computation blocks and aggregating the data returned by each computation block before forwarding it to the upper-layer network. For example, as... Figure 6 As shown, k data flows in from the Cluster input port, is switched by the cross switch inside the router, and is then sent to the target Block from the corresponding output port; the mask data calculated by the Block is aggregated by the router and forwarded to the macro network from the output port.

[0042] Computation blocks receive data streams from the mezzanine routers and accumulate the computed partial and protocol data across the computation block space via a direct-connection network independent of the mezzanine routers. This direct-connection network refers to a dedicated physical link established within the cluster between the four blocks that does not pass through the mezzanine routers, used for transmitting and accumulating partial and protocol data. For example, ... Figure 3 As shown, multiple blocks are connected via dedicated point-to-point links. Part of the output data from the first block can be directly sent to the second block for accumulation. The result accumulated by the second block is then sent to the third block, and so on. The final result is directly fed into the filtering circuit. The entire process does not consume the bandwidth resources of the mid-level router.

[0043] The filtering circuit receives the accumulated partial sums and reduced data, performs global sorting and sparse reduction, generates filtering results, and injects these results into the meso-level router. Global sorting and sparse reduction refer to uniformly sorting the partial sums from multiple computational blocks, filtering out the top K results with the largest values, and setting the remaining results to zero, thereby generating sparse mask data. For example, the TopK circuit module receives the accumulated partial sums from all blocks, sorts them according to their numerical values, selects the indices corresponding to the K largest values ​​as the filtering results, and outputs zero for the positions corresponding to the remaining indices.

[0044] The meso-level router employs a non-standard cross-switch matrix. Its input side includes multiple cross-macro-group input ports and filtering circuit input ports. The filtering circuit input ports receive the filtering results injected unidirectionally by the local filtering circuit. Cross-macro-group input ports are those connected to other cluster routers or macro-group routers, used to receive data packets from outside the cluster. For example, ... Figure 6 As shown, the three Cluster input ports on the input side are connected to adjacent Clusters or macro routers, respectively. The filtering circuit input port is a dedicated input port for receiving the filtering results (mask data) output from the local filtering circuit. This port is directly connected to the filtering circuit, and the data flow is unidirectional. The output side of the meso-level router includes multiple output ports connected to local computing blocks and multiple output ports connected to other meso-level routers. A local computing block refers to four basic computing blocks located within the same Cluster as the current meso-level router. Each block is responsible for performing local tensor operations.

[0045] The micro-interconnected network employs an H-shaped tree-like fractal interconnection topology to process the received data and concatenate and accumulate the processed partial and reduced data to obtain the filtering result. The H-shaped tree-like fractal interconnection utilizes a highly symmetrical spatial wiring strategy to eliminate signal skew and arrival delay differences during multi-node broadcasting and aggregation.

[0046] H-type tree-like fractal interconnect topology refers to a topology where 32 basic computing cells are interconnected within a block through a symmetrical H-type tree structure. The data delivery network and some protocol networks form a mirror-symmetric topology in terms of physical cabling, eliminating signal skew and arrival delay differences. For example, as... Figure 4 As shown, the data delivery network (blue) starts from the top Data input end, goes through two levels of H-shaped branches, and then radially connects to 32 cells; the partial and reduction network (orange) starts from the bottom Merge input end, also goes through two levels of H-shaped branches, and then connects to 32 cells. The two networks are symmetrically distributed in space.

[0047] The micro-interconnected network is located within the computation block and includes multiple addition units and multiple arithmetic units. The arithmetic units receive the distributed input data stream during the data input phase and perform parallel computations to generate partial and reduced data. The addition units receive externally input reduced data during the reduction phase and concatenate and accumulate it with the partial and reduced data output from the arithmetic units.

[0048] The micro-interconnect network physically isolates the data distribution network and the partial sum and reduction network, and dynamically switches between point-to-point unicast mode and global synchronous broadcast mode based on the computation stage through a gated distribution microarchitecture. The data distribution network is the physical network used to distribute input data from the Block input to each computing unit, employing an H-shaped tree topology, responsible for unidirectional data transmission. The partial sum and reduction network is the physical network used to aggregate the partial sum and reduction data generated by each computing unit to the adding unit level by level, employing an H-shaped tree topology symmetrical to the data distribution network, responsible for unidirectional data transmission. Point-to-point unicast mode means that during the data storage stage, input data is only written to a specific target computing unit, and the input links to other computing units are blocked. For example, in the k-data stream storage stage, different k-data need to be written to different Cells, in which case point-to-point unicast mode is used. Global synchronous broadcast mode means that during the data triggering stage, input data is sent to all computing units simultaneously, and all Cells receive the same data within the same clock cycle. For example, during the q data stream operation triggering phase, it is necessary to broadcast the same q data to all Cells simultaneously to trigger parallel inner product operations. In this case, a global synchronous broadcast mode is used.

[0049] The gating distribution microarchitecture dynamically switches between point-to-point unicast mode and global synchronous broadcast mode based on the computation stage. Specifically, the external Mode signal controls the output of the gating control logic. When the Mode signal indicates the k computation stage, the gating control logic outputs the result of the one-hot code ring counter, which activates the input links of individual cells in sequence. When the Mode signal indicates the q computation stage, the gating control logic outputs an all-one vector, which activates the input links of all cells simultaneously.

[0050] The gated distribution microarchitecture of the micro-interconnected network includes gating control logic, a data delivery network, and an AND gate array. The gating control logic comprises a one-hot code ring counter and an all-one vector generator, switched by an external Mode signal. A one-hot code ring counter is a ring counter whose output sequence has only one bit set to 1 and the rest to 0. Its output bits are cyclically shifted to activate specific computational units in batches. For example, ... Figure 7As shown, in a 32-cell scenario, the one-hot code ring counter sequentially outputs 32-bit one-hot codes, each bit being high in sequence, selecting the input links of the 1st, 2nd, ... 32nd cells. The all-one vector generator refers to a vector generation circuit where all output bits are 1, used to simultaneously activate the input links of all arithmetic units. The external Mode signal is a stage indication signal issued by the MoBA algorithm controller, used to identify whether the current stage is the k data storage stage or the q operation trigger stage. Specifically, the external Mode signal controls the switching as follows: when the Mode signal is low, the multiplexer selects the output of the one-hot code ring counter; when the Mode signal is high, the multiplexer selects the output of the all-one vector generator.

[0051] The AND gate array is positioned on each parallel branch of the data delivery network and controlled by the chip select signal output by the gating control logic. It is used to selectively activate the input links of target computational units in point-to-point unicast mode and simultaneously activate the input links of all computational units in global synchronous broadcast mode. The chip select signal refers to the 32-bit parallel control signal output by the gating control logic, with each bit controlling the on / off state of the AND gate on the corresponding computational unit's input link. In point-to-point unicast mode, selectively activating the input link of the target computational unit specifically involves setting only one bit of the 32-bit signal output by the one-hot code ring counter to 1, opening the corresponding AND gate, and allowing the input data stream to enter the corresponding computational unit through this gate; the remaining 31 bits are 0, closing the corresponding AND gate and blocking the data stream. In global synchronous broadcast mode, simultaneously activating the input links of all computational units specifically involves setting all 32 bits of the signal output by the all-one vector generator to 1, opening all AND gates simultaneously, and allowing the input data stream to pass through all AND gates and enter the 32 computational units in parallel.

[0052] The micro-interconnect network also includes serial-in-parallel-out (SPI) and parallel-in-serial-out (SPI) modules. The SPI module receives externally input protocol data, performs serial-to-parallel format conversion, and then sends the parallel data to each added unit. Externally input protocol data refers to partial protocol data (Merge-In data) from other computational blocks or upper-layer networks, input to the local block in the form of a serial bit stream. The micro-interconnect network spatially isolates the data transmission network and the partial protocol network in physical routing. Physical routing means that in chip layout design, the data transmission network and the partial protocol network occupy different metal layers or different routing areas, and they do not overlap spatially to avoid signal crosstalk.

[0053] The data distribution network employs an H-shaped tree-like fractal topology to radially and synchronously distribute input data to each processing unit. Specifically, the input data enters from the Data input at the top of the Block, is replicated into 2 paths after the first-level H-shaped branch, 4 paths after the second-level H-shaped branch, 8 paths after the third-level H-shaped branch, 16 paths after the fourth-level H-shaped branch, and 32 paths after the fifth-level H-shaped branch, ultimately reaching 32 processing units synchronously. The physical length of each path is equal, ensuring consistent data arrival times.

[0054] The partial sum and reduction network adopts an H-shaped tree-like fractal topology symmetrical to the data distribution network. This is used to progressively aggregate the partial sum and reduction data generated by each computational unit to the adding unit for cascading accumulation. Specifically, the partial sums and reduction data generated by the 32 computational units are first added pairwise at the fifth-level H-shaped node, generating 16 intermediate results. These 16 intermediate results are then added pairwise at the fourth-level H-shaped node, generating 8 intermediate results, and so on, until a single global partial sum and reduction data is generated at the top level. The adding units are distributed at each level of the H-shaped node. Each adding unit receives two inputs: the local cell computation result and the externally input reduction data. After accumulation, it outputs to the next level adding unit.

[0055] Example 1

[0056] This embodiment illustrates the process of the K-data stream storage stage. The k-data stream carrying the attention mechanism key vector is injected into the backbone network from the central memory interface. The central Group router first executes a tree-like diffusion mechanism, multicasting the data packets to the eight peripheral computing groups. The data stream first enters the router from the physical input ports (Input#0 to Input#N) on the left. At the input, data shards are guided and cached in the corresponding virtual channel (VC#0 to VC#N) queues according to their own virtual channel IDs through a multiplexer. Above the virtual channel queues is the RC (Routing Calculation) module, and the data header shards stored in the VC queues interact directly with the RC module. The RC module calculates and determines the target physical output port that the data packet needs to go to next by reading the control information of the header shards. The data shards first request output channel resources from the VA (Virtual Channel Assignment) module. Only data shards that successfully obtain VA arbitration are allowed to leave the queue. Data that obtains VA authorization then requests output port resources from the SA (Cross Switch Assignment) module. When the data shards obtain arbitration and clearance from the SA module, they enter the ST module. Through the control signals issued by the SA module, the crossbar switch network inside the ST module establishes a dedicated physical link from the input end to the target output end. Finally, the data passes through the ST module and flows out in parallel from the corresponding physical output ports on the right (Output#0 to Output#N) to the next destination node.

[0057] After the data reaches the mid-level, the Cluster router continues to perform multicast. To accurately match the highly asymmetric data interaction needs at this level, the core of the Cluster router uses a non-standard crossbar switch matrix with four inputs and seven outputs to accommodate the local interconnection requirements of the gated operation circuits. The router's input side contains four physical ports, three of which are Cluster inputs, through which cross-cluster data packets are sent; the remaining port is a TopK input, specifically used to receive data injected unidirectionally by the TopK module within the local Cluster. Similarly, input data packets enter the corresponding virtual channel based on the virtual channel ID. Only data fragments that obtain VA authorization can leave the virtual channel and request output channel resources. The SA+ST module in the middle of the diagram aggregates the requests from the four input data packets and performs global arbitration matching. Data packets that obtain SA authorization can enter the crossbar switch ST and flow out to their destination output port. The router's output side contains 7 physical ports. Four of these output ports radiate outwards and connect to the four blocks in the local cluster, responsible for sending data to the local computing unit. The remaining three output ports are linked to other cluster input ports and are responsible for forwarding the computation results to adjacent network nodes or writing them back to the upper-layer network.

[0058] During the k data stream storage phase, k data flows in from the Cluster input segment, is distributed via routers, and multicasts to the four local blocks.

[0059] Once traffic enters the Block, the communication requirement transforms into point-to-point unicast. The Cell-level gated distribution microarchitecture consists of four parts: gating control logic, data delivery network, computational Cell array, and PSUM protocol. The gating control logic includes two modes: a one-hot ring counter providing the unicast mask and an all-one vector providing the broadcast mask. An externally input Mode signal provides the control signal. The data delivery network, under gating control, divides the externally input Data into 32 parallel branches within the micronetwork. These branches are connected to the input sides of the 32 computational Cell arrays via AND gate arrays controlled by the Sel signal. Data only penetrates the AND gate array and enters the Cell computation unit when the corresponding Sel valid level is 1; otherwise, the link is blocked. The PSUM protocol network includes a SIPO (serial-in parallel-out) module for receiving external PSUM data, parallel-distributed Add modules, and a PISO (parallel-in serial-out) module for outputting PSUM results. The PSUM reduction network receives Merge In data from the outside, completes serial-to-parallel conversion, and sends it to the Add of the corresponding Cell for reduction. The global reduction result, after parallel-to-serial conversion, is finally output as Merge Out data from the Block.

[0060] During the k-data stream storage stage, the multiplexer at the input end selects the one-hot code ring counter under the control of the Mode signal, and activates the input link of the target cell in batches, writing independent k data into 32 cells to complete distributed storage. At this time, the unselected cell branches are in a static flip-off suppression state under the control of the AND gate array.

[0061] Example 2

[0062] This embodiment illustrates part of the process during the Q data stream operation triggering phase. q data is sent from the central memory interface. At this point, the macro and meso levels switch to unicast mode, and the data is routed along the shortest path to the target Cluster and the specific destination Block. Once the q data packet penetrates into the Block, the micro-communication requirement is converted to global broadcast: the Mode signal switches the multiplexer to output a single vector, and the q data is simultaneously delivered to all 32 Cells by the underlying network, thereby driving high-concurrency local cell inner product operations.

[0063] Example 3

[0064] This embodiment illustrates part of the Merge data stream reduction phase. After the inner product operation is completed, the partial sum of local sums (PSUM) data generated by the 32 cells is synchronously sent to a distributed adder array deeply embedded in an H-shaped tree network. After local pipeline reduction is completed within the Block, the data is reassembled and flows out of the Block port. The high-concurrency merge data stream does not enter the Cluster main data path, but is directly sent to a dedicated point-to-point direct connection network, where it is spatially accumulated across the Cluster with the PSUM data of other corresponding Blocks, and finally merged into a specific TopK circuit to complete global sorting and sparse reduction. This physical isolation completely eliminates the unnecessary bandwidth pressure on the central router.

[0065] Example 4

[0066] This embodiment illustrates part of the Mask data stream output phase. After the TopK module completes the filtering, it injects the generated sparse mask data unidirectionally into the TopK input port of the Cluster router. The Cluster router is specifically configured with a streamlined single-channel TopK input port on the input side for independently buffering the locally generated mask result data. Subsequently, this output data stream penetrates and converges from the peripheral nodes to the central node, hop-by-hop back to the Group router along the original topology path. To address the many-to-one communication pressure brought by multi-path convergence, the Group router's data path introduces an acceleration mechanism, allowing the crossbar switch to concurrently output two micro-slices to the same physical output port per cycle. This mechanism, combined with the downlink bus, significantly alleviates the throughput bottleneck under peak load, ultimately enabling efficient data write-back to the central memory output, completing the end-to-end data loop of the entire MoBA algorithm.

[0067] The architecture provided by this invention significantly alleviates network congestion and achieves global traffic load balancing: This architecture effectively utilizes central radiating links to share the communication pressure of hotspot nodes. Simulation heat map results show that it eliminates the extreme head-blocking effect of a single node and its adjacent backbone link in a traditional flat grid, thus achieving global traffic balancing.

[0068] The architecture provided by this invention reduces end-to-end communication latency: based on experimental data from the Booksim2 simulation platform, the average latency of this architecture package is reduced by approximately 99.6% compared to the corner injection flat mesh scheme and by approximately 91.0% compared to the center injection single-layer mesh scheme.

[0069] This invention reduces hardware overhead: the macro router adopts a central five-port and radial path three-port structure, which reduces global hardware overhead while ensuring throughput.

[0070] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A hybrid on-chip interconnect architecture for MoBA-gated accelerators, characterized in that, This includes macro-level interconnected networks, meso-level interconnected networks, and micro-level interconnected networks, with hierarchical and collaborative data flow transmission links formed between these networks; The macro-level interconnection network adopts a hybrid topology structure that combines central radiation and two-dimensional grid, which is used to distribute input data to the meso-level interconnection network in a multicast manner, and output the filtering results returned by the meso-level interconnection network to the video memory interaction core in a multi-path aggregation manner. The meso-level interconnection network adopts a hybrid star and tree topology to distribute the received data to the micro-level interconnection network and to send the filtering results output by the micro-level interconnection network back to the macro-level interconnection network. The micro-interconnection network adopts an H-shaped tree-like fractal interconnection topology to perform calculations on the received data, and to cascade and accumulate the calculated partial and reduced data to obtain the filtering results.

2. The hybrid on-chip interconnect network architecture for MoBA-gated accelerators according to claim 1, characterized in that, The macro-interconnection network includes multiple macro groups and multiple macro routers; the macro groups include a central macro group and multiple peripheral macro groups; the macro routers include a central router and multiple peripheral routers, and the macro routers construct a backbone transmission network through a two-dimensional grid. The central macro group and central router serve as the core hub for global data distribution and aggregation. They are used to distribute the input data stream from the memory interaction core to each peripheral macro group and peripheral router through a tree-structured divergent multicast routing mechanism, and to output the returned output data through a multi-path aggregation mechanism to the memory interaction core. The peripheral macro group and its corresponding peripheral routers serve as data relay nodes between the macro interconnection network and the meso interconnection network. They are used to receive input data streams and send them to the internal meso interconnection network, and forward the output data streams returned by the meso interconnection network to the central macro group.

3. The hybrid on-chip interconnect network architecture for MoBA-gated accelerators according to claim 2, characterized in that, The macro router adopts a multiple-input multiple-output architecture based on virtual channels; The central router adopts a multi-port structure to connect the video memory interaction core and multiple peripheral macro groups; The multi-port structure is greater than or equal to 5; The peripheral router adopts a three-port structure and is used to connect the central router, the meso-level interconnection network within the same macro group, and adjacent peripheral routers.

4. The hybrid on-chip interconnect network architecture for MoBA-gated accelerators according to claim 2, characterized in that, The data path of the macro router is equipped with a cross switch acceleration module; The cross switch acceleration module is used to concurrently output two microchips to the same physical output port each cycle during the multi-path aggregation stage of the output data stream.

5. The hybrid on-chip interconnect network architecture for MoBA-gated accelerators according to claim 2, characterized in that, The meso-level interconnection network is located within the macro-level group and includes meso-level routers, multiple computing blocks, and filtering circuits; The meso-level router is used to centrally distribute and aggregate conventional tensor data, send the input data stream to the computing block, and forward the data stream returned by the computing block to the macro-level interconnection network. The computing block is used to receive the data stream sent by the meso-level router, and to accumulate the calculated part and the protocol data across the computing block space through a direct connection network independent of the meso-level router; The filtering circuit is used to receive the accumulated partial sum and reduction data, perform global sorting and sparse reduction, generate filtering results, and inject the filtering results into the meso-level router.

6. The hybrid on-chip interconnect network architecture for MoBA-gated accelerators according to claim 5, characterized in that, The mid-level router uses a non-standard crossbar switch matrix; The input side of the meso-macro router includes multiple cross-macro group input ports and a filtering circuit input port. The filtering circuit input port is used to receive the filtering results unidirectionally injected by the local filtering circuit. The output side of the meso-level router includes multiple output ports connected to the local computing block and multiple output ports connected to other meso-level routers.

7. The hybrid on-chip interconnect network architecture for MoBA-gated accelerators according to claim 3, characterized in that, The micro-interconnected network is located inside the computing block and includes multiple addition units and multiple computing units; The computing unit is used to receive the distributed input data stream during the data input phase and perform parallel operations to generate partial and reduced data. The adding unit is used to receive externally input reduction data during the reduction phase and to cascade and accumulate it with the portion of the reduction data output by the computing unit. The micro-interconnected network physically isolates the data distribution network from the protocol network and dynamically switches between point-to-point unicast mode and global synchronous broadcast mode according to the computing stage through a gated distribution microarchitecture.

8. The hybrid on-chip interconnect architecture for MoBA-gated accelerators according to claim 7, characterized in that, The gated distribution microarchitecture of the micro-interconnected network includes gated control logic, a data delivery network, and an AND gate array. The gating control logic includes a one-hot code ring counter and an all-one vector generator, which are switched by an external Mode signal; The AND gate array is set on each parallel branch of the data delivery network and is controlled by the chip select signal output by the gating control logic. It is used to selectively activate the input link of the target computing unit in point-to-point unicast mode and to simultaneously activate the input links of all computing units in global synchronous broadcast mode.

9. The hybrid on-chip interconnect network architecture for MoBA-gated accelerators according to claim 7, characterized in that, The micro-interconnected network also includes a serial-in parallel-out module and a parallel-in serial-out module; The serial-in parallel-out module is used to receive externally input standard data, complete the serial-to-parallel format conversion, and then send the parallel data to each adding unit.

10. The hybrid on-chip interconnect network architecture for MoBA-gated accelerators according to claim 7, characterized in that, The micro-interconnection network spatially isolates the data delivery network and part of the protocol network in terms of physical cabling; The data distribution network adopts an H-shaped tree-like fractal topology to radially and synchronously distribute input data to each processing unit. The partial and reduction network adopts an H-shaped tree-like fractal topology that is symmetrical to the data distribution network. It is used to aggregate the partial and reduction data generated by each computing unit to the addition unit for cascading accumulation.

Images