Processing unit interconnection method, system, board and electronic device

Abstract

The application discloses a processing unit interconnection method, system, board and electronic equipment. The system comprises: a plurality of processing unit clusters, each processing unit cluster comprising a plurality of processing units and a local full interconnection network for realizing direct communication between any two processing units in the cluster; a segmented row-column routing network for realizing inter-cluster communication; a sparse awareness controller configured to generate a sparsity matrix according to a data dependency graph of a current computing layer in a configuration phase, preselect a routing path for a source processing unit cluster and a target processing unit cluster with inter-cluster communication and generate a path preselection table according to the sparsity matrix and a preset sparsity threshold, and write a sparse skip flag in the path preselection table for communication below the sparsity threshold in the sparsity matrix; and a zero value compression engine for merging continuous zero value tokens issued from the source processing unit cluster into a zero value compression package based on the sparse skip flag and sending the zero value compression package to the target processing unit cluster via the segmented row-column routing network.

Description

Technical Field

[0001] This application relates to the field of reconfigurable computing architecture technology, specifically to a processing unit interconnection method, system, board, and electronic device. Background Technology

[0002] Coarse-Grained Reconfigurable Architecture (CGRA) is an energy-efficient AI acceleration solution that uses an array of Processing Elements (PEs) as the computing core. It forms a reconfigurable computing array by interconnecting multiple PEs in a specific topology. The interconnection topology of the PE array directly affects data flow efficiency, array utilization, and chip energy efficiency, and is one of the core issues in CGRA design.

[0003] Currently, the mainstream PE array interconnection solutions in CGRA mainly include the following: (a) Pure Mesh (two-dimensional mesh) topology.

[0004] In this topology, each PE is directly connected to only four adjacent PEs (top, bottom, left, and right). Long-distance communication requires hop-by-hop forwarding through multiple intermediate PEs. Typical examples include Northeastern University's Cool Mega Array and some commercial CGRA solutions. When the array size is N×N, the furthest communication distance between any two PEs is 2N hops, and the communication latency increases linearly with the array size.

[0005] (ii) Pure Crossbar topology.

[0006] In this topology, all physical exciters (PEs) are directly interconnected via a global crossbar switch, enabling zero-hop direct communication between any two PEs. A representative solution is SambaNova Systems' SN series reconfigurable processors. However, the wiring complexity of the crossbar is O(N...). 2 As the number of PEs increases, the chip area and power consumption rise sharply. When the number of PEs exceeds 64, the area overhead becomes unsustainable.

[0007] (iii) Layered Mesh Topology.

[0008] This scheme groups PEs, with full interconnection within each group and mesh interconnection between groups. This method alleviates the long-distance communication latency problem of pure mesh to some extent, but inter-group communication still requires multi-hop forwarding and lacks specific adaptation for sparse data stream computing patterns.

[0009] In addition, existing technologies also include interconnection schemes based on segmented row-column (SRC) buses, which segment the row and column buses to reduce bus contention and power consumption. However, existing SRC schemes are only used for data transmission in regular communication modes and do not address the perception and utilization of data stream sparsity. In sparse scenarios, zero-value data still occupies communication resources on par with valid data.

[0010] With the rapid development of artificial intelligence technology, the computation mode of modern large AI models represented by the Transformer architecture exhibits significant sparsity characteristics: (1) the Attention matrix tends to be sparse in long sequence computation; (2) after activation functions such as ReLU and GELU, a large number of elements are close to zero; (3) in the hybrid expert (MoE) model, gated routing only activates a portion of the expert network. In such sparse computation scenarios, the existing CGRA interconnection scheme has the following problems and drawbacks: First, long-distance communication between PEs results in high latency. In a pure mesh topology, the furthest communication between an N×N array requires 2N hops. Based on a 16×16 array, the furthest communication latency reaches 32 cycles, leading to increased end-to-end computation latency and a reduction in PE utilization to 40%~55%.

[0011] Second, zero-value data in sparse data streams ineffectively consumes communication resources. In attention matrices with sparsity of 60% to 80%, effective data only accounts for 20% to 40% of the communication bandwidth. Zero-value data consumes the same amount of communication bandwidth and routing resources as traditional effective data, resulting in a significant decrease in effective throughput and a reduction in chip power efficiency ratio (TOPS / W).

[0012] Third, the routing paths lack dynamic sparsity awareness. Existing interconnection schemes use static routing paths, which cannot dynamically adjust communication strategies according to the sparsity of data flows. In long-distance communication, a large number of hop cycles are wasted on transmitting zero-value or low-contribution data, and there is a lack of runtime awareness and feedback mechanisms for data flow sparsity.

[0013] Fourth, the area overhead of pure Crossbar is not scalable. When the number of PEs reaches 64, the Crossbar area already accounts for more than 30% of the total chip area, which severely limits the array size and makes it difficult to scale up to large-scale computing arrays with more than 256 PEs.

[0014] In summary, existing CGRA interconnect technologies have significant shortcomings in terms of long-distance communication latency, sparse data stream processing efficiency, and area scalability. There is an urgent need for a new interconnect solution that can simultaneously achieve low-latency communication and sparse sensing optimization under controllable area conditions. Summary of the Invention

[0015] In order to solve at least one of the above-mentioned problems in the prior art, embodiments of this application provide a processing unit interconnection method, system, board, and electronic device.

[0016] According to a first aspect of this application, a processing unit interconnection system is provided, comprising: multiple processing unit clusters, each processing unit cluster including multiple processing units and a local fully interconnected network, the local fully interconnected network being used to enable direct communication between any two processing units within the cluster; a segmented row-column routing network coupled to the multiple processing unit clusters, used to enable inter-cluster communication between different processing unit clusters; and a sparsity-aware controller coupled to the segmented row-column routing network, the sparsity-aware controller being configured to: during a configuration phase, generate sparsity moments based on the data dependency graph of the current computation layer. The system uses a sparse matrix and a preset sparse threshold to pre-select routing paths for source processing unit clusters and target processing unit clusters with inter-cluster communication and generates a path pre-selection table. For communications in the sparse matrix that are below the sparse threshold, a sparse skip flag is written into the path pre-selection table. A zero-value compression engine is used to merge consecutive zero-value tokens issued from the source processing unit cluster into a zero-value compressed packet carrying a zero-value token count value, based on the sparse skip flag in the path pre-selection table, and send the zero-value compressed packet to the target processing unit cluster via the segmented row-column routing network.

[0017] In some embodiments of this application, the segmented row-column routing network includes multiple segmented row buses and multiple segmented column buses. Each segmented row bus connects multiple processing unit clusters located in the same row, and each segmented column bus connects multiple processing unit clusters located in the same column.

[0018] In some embodiments of this application, the segmented row-column routing network is also used to transmit data tokens and the zero-value compressed packets between processing unit clusters during the runtime phase according to the path preselection table.

[0019] In some embodiments of this application, the system further includes a router interface unit corresponding to the target processing unit cluster, the router interface unit being used to recover the zero-value token based on the zero-value compressed packet.

[0020] In some embodiments of this application, the sparse sensing controller generates the sparsity matrix by: comparing the expected number of tokens to be transmitted with a zero-value mask for each edge in the data dependency graph to obtain the data flow sparsity from each source processing unit cluster to each target processing unit cluster, wherein the data flow sparsity is the ratio of the number of valid tokens in the corresponding communication to the total number of tokens; and filling the data flow sparsity from each source processing unit cluster to each target processing unit cluster into the matrix according to the source-target cluster index to obtain the sparsity matrix.

[0021] In some embodiments of this application, the pre-selection of routing paths by the sparse sensing controller includes: when the source processing unit cluster and the target processing unit cluster are located in the same row, selecting the row segment bus as the routing path; when the source processing unit cluster and the target processing unit cluster are located in the same column, selecting the column segment bus as the routing path; when the source processing unit cluster and the target processing unit cluster are neither in the same row nor in the same column, selecting the row segment bus and the column segment bus as the routing paths in sequence.

[0022] In some embodiments of this application, the sparse sensing controller includes a path preselector, which is used to allocate time slots for communication between multiple processing unit clusters sharing the same bus segment. The path preselection table records the time slot index corresponding to each communication, and each processing unit cluster initiates communication in the corresponding time slot according to the time slot index.

[0023] In some embodiments of this application, the zero-value compression engine merges consecutive zero-value tokens into a zero-value compressed package, including: when the number of consecutive zero-value tokens reaches a preset jump step, merging the preset jump step number of zero-value tokens into one zero-value compressed package.

[0024] In some embodiments of this application, the aforementioned zero-value token is a data token whose validity flag is invalid or whose data payload is all zero.

[0025] In some embodiments of this application, the aforementioned zero-value compressed package further includes a type identifier field, which is used to distinguish between data tokens, zero-value compressed packages, and configuration packages.

[0026] According to a second aspect of this application, a method for interconnecting processing units is also provided, comprising: obtaining a data dependency graph of the current computing layer; generating a sparsity matrix based on the data dependency graph, wherein each element of the sparsity matrix represents the sparsity of the data flow from a source processing unit cluster to a target processing unit cluster; pre-selecting routing paths for a source processing unit cluster and a target processing unit cluster with inter-cluster communication based on the sparsity matrix and a preset sparsity threshold, and generating a path pre-selection table; for communication in the sparsity matrix that is below the sparsity threshold, writing a sparsity skip flag in the path pre-selection table; and, based on the sparsity skip flag in the path pre-selection table, merging consecutive zero-value tokens issued from the source processing unit cluster into a zero-value compressed packet carrying a zero-value token count value, and sending the zero-value compressed packet to the target processing unit cluster via a segmented row-column routing network.

[0027] According to a third aspect of this application, a board is also provided, which includes the processing unit interconnect system described above.

[0028] According to a fourth aspect of this application, an electronic device is also provided, which includes the board as described above.

[0029] The processing unit interconnection method, system, board, and electronic device proposed in this application constitute a two-level communication architecture of "intra-cluster full interconnection + inter-cluster segmented routing" by setting up multiple processing unit clusters and their internal local full interconnection networks, combined with the segmented row-column routing networks coupled between each processing unit cluster. Intra-cluster communication enables direct communication between any two processing units through the local full interconnection network, while inter-cluster communication is achieved through the segmented row-column routing network. This compresses the multi-hop long-distance communication (O(N) hops) in the original Mesh topology, which is related to the array size, into a fixed number of segments (maximum 2 segments) independent of the array size, thereby significantly reducing the communication latency between long-distance PEs and improving PE utilization. Meanwhile, this application generates a sparsity matrix based on the data dependency graph during the configuration phase using a sparse-aware controller. Based on this matrix, it pre-selects routing paths and writes sparse skip flags, advancing the sparsity-aware decision-making to the configuration phase. This eliminates the overhead of runtime packet-by-packet routing decisions. Furthermore, during runtime, a zero-value compression engine merges consecutive zero-value tokens into lightweight compressed packets carrying only count values, based on the sparse skip flags, and sends them to the target processing unit cluster. This prevents zero-value data in the sparse data stream from occupying bus time slots segment by segment, significantly improving the effective bandwidth utilization and chip energy efficiency in sparse data scenarios. This overcomes the technical shortcomings of existing technologies, such as the invalid occupation of communication resources by zero-value data and the lack of dynamic sparsity awareness in routing paths. Moreover, compared to a pure Crossbar topology, this application's segmented row-column routing network uses a segmented bus structure, resulting in significantly lower wiring complexity than a global crossbar switch, controllable area overhead, and support for larger-scale PE array expansion. Attached Figure Description

[0030] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. In the drawings: Figure 1 This is a schematic diagram of the structure of a processing unit interconnection system provided in the application embodiment; Figure 2 This is a specific hardware configuration diagram provided in the application embodiment; Figure 3 This is a flowchart illustrating a processing unit interconnection method provided in an embodiment of this application. Detailed Implementation

[0031] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the embodiments of this application will be further described in detail below with reference to the accompanying drawings. Here, the illustrative embodiments and descriptions of this application are used to explain this application, but are not intended to limit this application.

[0032] As indicated in this application and claims, unless the context clearly indicates otherwise, the words "a," "an," "an," and / or "the" are not specifically singular and may include plural forms. Generally speaking, the terms "comprising" and "including" only indicate the inclusion of explicitly identified steps and elements, which do not constitute an exclusive list, and the method or apparatus may also include other steps or elements.

[0033] Furthermore, it should be noted that the use of terms such as "first" and "second" to define related concepts is merely for the purpose of distinguishing them. Unless otherwise stated, these terms have no special meaning and therefore should not be construed as limiting the scope of protection of this application. Additionally, although the terminology used in this application is selected from commonly known and used terms, some terms mentioned in this application's specification may have been chosen by the applicant according to his or her judgment, and their detailed meanings are explained in the relevant sections of this description. Moreover, this application should be understood not only through the actual terms used but also through the meaning implied by each term.

[0034] It should also be noted that the "processing element" (PE) mentioned in this application refers to the basic computing unit in a reconfigurable computing array. Its specific structure can be, but is not limited to, an arithmetic logic unit containing a multiply-accumulate unit (such as a 16-bit fixed-point multiply-accumulate unit MAC) and a register file. Its internal structure and specific instruction set are not the core improvements of this application. Those skilled in the art can select and configure them according to actual needs, and this application does not limit them.

[0035] It should also be noted that the "token" mentioned in this application refers to the basic unit of data transmitted between processing unit clusters, which includes fields such as validity flag, data payload, and routing information. The specific bit width and format of the token can be flexibly configured according to the width of the data path, for example, a 32-bit, 64-bit, or 128-bit data path can be used, and this application does not limit this.

[0036] In this application's embodiments, the "configuration phase" refers to the phase in which the system pre-selects routing paths and writes configuration parameters to the interconnection network based on the computation task description (i.e., data dependency graph) of each computation layer before execution begins; the "running phase" refers to the phase in which each processing unit cluster actually performs computation and transmits data tokens between clusters after configuration is completed. The two phases are executed sequentially, with the running phase following the completion of the configuration phase.

[0037] It should also be noted that the various modules and their connections included in the accompanying drawings are only schematic representations. The specific physical implementation of each module (such as using ASIC logic gates, FPGA lookup tables, semiconductor memory, etc.) can be flexibly selected according to manufacturing process and performance requirements, and this application does not make specific limitations in this regard.

[0038] Figure 1 This is a schematic diagram of the structure of a processing unit interconnection system provided in an embodiment of this application. Figure 1 As shown in the figure, an embodiment of this application provides a processing unit interconnection system including: multiple processing unit clusters 110, a segmented row-column routing network, a sparsity-aware controller 120, and a zero-value compression engine 130. Wherein: Multiple processing unit clusters 110, each processing unit cluster 110 includes multiple processing units and a local fully interconnected network, the local fully interconnected network is used to realize direct communication between any two processing units within the cluster.

[0039] In this embodiment, the local fully interconnected network can be a crossbar network. As an optional implementation, each processing unit cluster 110 contains K×K processing units, where K can be 4 or 8, for example, each cluster contains 4×4=16 processing units or 8×8=64 processing units. Those skilled in the art can flexibly choose the specific value of K according to the target array size and chip area constraints, and this application does not limit it in this regard.

[0040] In the implementation of a locally fully interconnected network as a Crossbar network, the Crossbar uses a multi-level selector tree structure. Its basic working principle is as follows: each input port is connected to each output port through a selector matrix. Data from any input port can be directly transmitted to any output port within one cycle, without needing to be forwarded through intermediate nodes. This structure ensures that the communication delay between any two processing units within the cluster is constant at one cycle, regardless of the number of processing units in the cluster.

[0041] Since the size of the crossbar within a cluster is limited to a single processing unit cluster (e.g., K×K processing units), its M×M crossbar number (where M=K) is limited. 2The difference is much smaller than N required to expand the Crossbar to a global array. 2 Scale. Therefore, the area overhead of a local crossbar is controllable and does not increase dramatically with the expansion of the array size.

[0042] The segmented row-column routing network consists of multiple segmented row buses and multiple segmented column buses. For multiple processing unit clusters arranged in rows and columns, each segmented row bus connects multiple processing unit clusters located in the same row, and each segmented column bus connects multiple processing unit clusters located in the same column. The segmented row buses and segmented column buses transmit data only within their respective covered segments and do not cross the entire array, thereby reducing power consumption and signal integrity issues caused by long-distance bus driving.

[0043] by Figure 1 Taking the structure shown as an example, assume that the processing unit clusters are arranged in a 2×2 row-column configuration (i.e., two row segments and two column segments). The segmented row-column routing network includes multiple segmented row buses and multiple segmented column buses. It should be noted that... Figure 1 The illustration only shows four clusters of processing units arranged in a 2×2 pattern. This application is not limited to this; in actual implementation, it can be configured as 16 clusters arranged in a 4×4 pattern, 8 clusters arranged in a 2×4 pattern, or other array topologies with any number of rows and columns, as needed. Those skilled in the art will understand that the number of segmented row buses equals the number of rows in the processing unit cluster, and the number of segmented column buses equals the number of columns in the processing unit cluster. Each segmented bus connects all clusters in the corresponding row or column.

[0044] During operation, when data tokens are transmitted between processing unit clusters, the source processing unit cluster sends the data token to the segmented row bus or segmented column bus coupled to it. The segmented row bus or segmented column bus transmits the data token to the corresponding segment where the target processing unit cluster is located according to the path preselection table (see the description of the sparse sensing controller below). The target processing unit cluster receives the data token from the bus.

[0045] It should be understood that when the source processing unit cluster and the target processing unit cluster are in the same row, the data token only needs to pass through a segmented row bus corresponding to that row to reach the target cluster, without needing to pass through the column bus; when the source cluster and the target cluster are in the same column, the data token only needs to pass through a segmented column bus corresponding to that column; when the source cluster and the target cluster are in neither the same row nor the same column, the data token needs to pass through a segmented row bus and a segmented column bus in sequence (or vice versa, depending on the path pre-selection strategy).

[0046] In this embodiment, the data width of the segmented row bus and segmented column bus is 128 bits, used to carry data tokens and their additional control information. Multiple processing unit clusters (e.g., 4 or 8) can be connected to each bus segment, and multiple processing unit clusters share the same bus segment through time-division multiplexing.

[0047] A router interface unit (RT) is provided between each processing unit cluster and its connected segmented row bus and segmented column bus. The router interface unit is responsible for sending data tokens issued from within the cluster to the corresponding segmented row bus or segmented column bus according to the pre-selected path, and for receiving data tokens destined for the cluster from the bus.

[0048] The sparse sensing controller 120 is coupled to the segmented row-column routing network. Specifically, the sparse sensing controller 120 is connected to the router interface unit of each processing unit cluster and the zero-value compression engine 130 through control signal paths, and is used to write configuration information to the router interface unit.

[0049] In this embodiment, the sparse sensing controller 120 is configured to perform the following configuration phase operations: During the configuration phase, the sparse sensing controller 120 first obtains the data dependency graph of the current computing layer, which describes the data flow direction and data volume information between each cluster of processing units in the current computing layer. Then, the sparse sensing controller 120 generates a sparsity matrix based on the data dependency graph. Each element of the sparsity matrix represents the data flow sparsity from the source processing unit cluster to the target processing unit cluster. The data flow sparsity is used to reflect the ratio of valid data to zero-value data in a single inter-cluster communication. After generating the sparsity matrix, the sparse sensing controller 120 pre-selects routing paths for source processing unit clusters and target processing unit clusters with inter-cluster communication based on the sparsity matrix and a preset sparsity threshold, and generates a path pre-selection table. The path pre-selection table records the routing information between each pair of source-target clusters with communication, including the segmented row bus and / or segmented column bus that the communication should pass through. For communications in the sparse matrix that are below the preset sparse threshold, the sparse sensing controller 120 writes a sparse skip flag into the path preselection table entry corresponding to the communication. This flag is used to indicate that the communication is a high sparse communication, and the zero-value tokens in it should be compressed in the subsequent operation phase. After the above configuration phase is completed, the path preselection table is written into the router interface unit of each processing unit cluster, preparing for data transmission during the runtime phase.

[0050] In this embodiment, the zero-value compression engine 130 is configured to correspond to the segmented row-column routing network and is used to perform zero-value compression operations during the runtime phase based on the sparse skip flag in the path preselection table.

[0051] During the runtime phase, each processing unit cluster sends a data token to the target processing unit cluster according to the routing path pre-selected during the configuration phase. The data token contains flag information indicating whether the current token is a zero-value token, which is a data token with an invalid validity flag or all zero data payload.

[0052] For communications with a sparse skip flag in the path preselection table, the zero-value compression engine 130 is enabled. The zero-value compression engine 130 detects whether the data tokens issued from the source processing unit cluster are zero-value tokens and merges consecutively detected zero-value tokens into a zero-value compressed packet (ZCP) carrying a zero-value token count. This zero-value compressed packet is sent to the target processing unit cluster via a segmented row-column routing network. Because the zero-value compressed packet only carries the count value and not the actual data payload of the zero-value tokens, its data volume is much smaller than the data volume required to transmit the same number of zero-value tokens, thus saving bus bandwidth.

[0053] For regular communication where the sparse skip flag is not written in the path preselection table, the zero-value compression engine 130 does not perform compression operations, and all data tokens are transmitted to the target processing unit cluster segment by segment in the traditional manner.

[0054] It should be understood that the working logic of the zero-value compression engine 130 relies entirely on the sparse skip flags pre-written in the path preselection table, requiring no further intervention from the sparsity-aware controller 120 during runtime. This architecture, which advances sparse-aware decision-making to the configuration phase and allows it to be executed autonomously by the hardware at runtime, eliminates the overhead of runtime packet-by-packet routing decisions.

[0055] As described above, the processing unit interconnection system provided in this application embodiment constitutes a two-level communication architecture of "intra-cluster full interconnection + inter-cluster segmented routing" by setting up multiple processing unit clusters and their internal local full interconnection networks, combined with the segmented row-column routing networks coupled between each processing unit cluster. Intra-cluster communication enables direct communication between any two processing units through the local full interconnection network, while inter-cluster communication is achieved through the segmented row-column routing network. This compresses the multi-hop long-distance communication (O(N) hops) in the original Mesh topology, which is related to the array size, into a fixed number of segments (maximum 2 segments) independent of the array size, thereby significantly reducing the communication latency between long-distance PEs and improving PE utilization. Meanwhile, this application generates a sparsity matrix based on the data dependency graph during the configuration phase using a sparse-aware controller. Based on this matrix, it pre-selects routing paths and writes sparse skip flags, advancing the sparsity-aware decision-making to the configuration phase. This eliminates the overhead of runtime packet-by-packet routing decisions. Furthermore, during runtime, a zero-value compression engine merges consecutive zero-value tokens into lightweight compressed packets carrying only count values, based on the sparse skip flags, and sends them to the target processing unit cluster. This prevents zero-value data in the sparse data stream from occupying bus time slots segment by segment, significantly improving the effective bandwidth utilization and chip energy efficiency in sparse data scenarios. This overcomes the technical shortcomings of existing technologies, such as the invalid occupation of communication resources by zero-value data and the lack of dynamic sparsity awareness in routing paths. Moreover, compared to a pure Crossbar topology, this application's segmented row-column routing network uses a segmented bus structure, resulting in significantly lower wiring complexity than a global crossbar switch, controllable area overhead, and support for larger-scale PE array expansion.

[0056] In some embodiments of this application, the router interface unit corresponding to the target processing unit cluster is further configured to recover the zero-value token based on the zero-value compressed packet.

[0057] Specifically, during operation, the ingress router continuously monitors the segmented row bus and / or segmented column bus it is connected to. When the ingress router receives a transmission unit, it first parses the type identifier field (Type) of the transmission unit to determine whether it is a data token, a zero-value compressed packet, or a configuration packet. That is, a zero-value compressed packet also includes a type identifier field, which is used to distinguish between data tokens, zero-value compressed packets, and configuration packets.

[0058] If the type identifier indicates that the currently received transmission unit is ZCP, the ingress router enters the zero-value recovery process. The zero-value packet carries at least a zero-value token count (Count field), which represents the number of consecutive zero-value tokens represented by this zero-value packet. The ingress router extracts this count from the zero-value packet, then generates the corresponding number of zero-value tokens locally, and sequentially sends these zero-value tokens to the local fully interconnected network within the target processing unit cluster for use by each processing unit within the cluster.

[0059] It should be noted that the zero-value token generated by the ingress router has the same format specification as the original data token, including fields such as valid flag, sparse flag, cluster identifier, timestamp, and data payload. The valid flag is set to invalid (e.g., Valid Flag=0), and the data payload field is filled with all zeros to ensure that the recovered zero-value token is semantically identical to the original zero-value token. The computational logic within the target processing unit cluster does not need to distinguish whether the zero-value token arrived originally or was recovered.

[0060] In other embodiments, the zero-value compressed packet also carries position offset information (Offset field). This offset information indicates the starting token sequence number of the zero-value run represented by this zero-value compressed packet in its source-destination token stream. When recovering the zero-value token, the ingress router places the recovered zero-value token in the correct timing position according to this offset information, ensuring that the token sequence received by the target processing unit cluster corresponds one-to-one with the original token sequence sent by the source end in timing.

[0061] To help those skilled in the art better understand the above recovery process, a specific example is provided below.

[0062] In a certain sparse communication, suppose the original token sequence sent by the source processing unit cluster to the target processing unit cluster is (Z represents a zero-value token, and N represents a non-zero valid token): Location: 0 1 2 3 4 5 6 7 8 9; Token: ZZZZN a ZZZZN b ; At the source end, the zero-value compression engine follows a preset jump step size N. skip =4 Compressing consecutive zero-value tokens: The four consecutive zero-value tokens at positions 0 to 3 are merged into the first zero-value compressed packet ZCP1, with Count=4 and Offset=0; the non-zero token N at position 4 a The tokens are sent normally as regular data tokens; the four consecutive zero-value tokens at positions 5-8 are merged into a second zero-value compressed packet ZCP2, with Count=4 and Offset=5; the non-zero token N at position 9... b It was sent normally as a regular data token.

[0063] At the target end, the ingress router receives ZCP1 and N sequentially. a ZCP2, N bThe recovery process of the ingress router is as follows: First, the ingress router receives ZCP1, parses it to obtain Count=4 and Offset=0, then generates 4 zero-value tokens locally, and places them in the target token stream starting from position 0 according to Offset=0, thus recovering the 4 zero-value tokens at positions 0 to 3; then, the ingress router receives the ordinary data token N. a The ingress router receives ZCP2 and parses it to obtain Count=4 and Offset=5. It then generates four zero-value tokens locally and places them in the target token stream starting at position 5, based on Offset=5, thus recovering the four zero-value tokens at positions 5-8. Finally, the ingress router receives the ordinary data token N. b It is then sent directly to position 9 of the target token stream.

[0064] After the above recovery process, the target processing unit cluster obtains the complete token sequence: ZZZZN a ZZZ ZN b It is completely consistent with the original token sequence sent by the source, realizing lossless restoration of the zero-value compressed package.

[0065] In some embodiments of this application, since the segmented row bus and segmented column bus adopt the TDMA (Time Division Multiple Access) time slot mechanism, multiple data streams from different source-target cluster pairs may be interleaved on the same bus segment. When the ingress router receives a zero-value compressed packet, in addition to extracting the Count and Offset fields, it also needs to combine the source cluster identifier and target cluster identifier carried by the current transmission unit to determine the data stream to which the zero-value compressed packet belongs, thereby correctly sending the recovered zero-value token into the local fully interconnected network of the corresponding target processing unit cluster. The existence of the Offset field enables the ingress router to accurately restore each zero-value run to the correct timing position of its data stream even in cross-flow interleaving scenarios.

[0066] Compared to segmented transmission N skip N raw zero-value tokens (each token requires one bus time slot and one full data payload bandwidth), the ingress router receives a zero-value compressed packet and locally batch recovers N... skip Using zero-value tokens significantly reduces the transmission overhead of zero-value data on segmented row-column routing networks. With N skipTaking a data token payload of 128 bits as an example, four raw zero-value tokens require four bus time slots to transmit 512 bits of data, while a zero-value compressed packet only requires one bus time slot to transmit approximately 25 bits of control information. This reduces bus time slot usage by 75% and data transmission volume by approximately 95%. The local recovery operation of the ingress router is completed within the target cluster, without occupying inter-cluster bus resources, thereby improving the overall effective bandwidth utilization and energy efficiency of the system.

[0067] In some embodiments of this application, the recovery behavior of the ingress router is configurable. Specifically, the ingress router internally includes a recovery control register for storing parameters such as the current zero-value compressed packet format version, count value bit width, and offset bit width. These parameters are written by the sparse-aware controller during the configuration phase to ensure that the ingress router can correctly parse the zero-value compressed packet format under different configurations. When the system configuration changes (e.g., N...), the recovery behavior is configurable. skip (The value is reconfigured), the sparse sensing controller synchronously updates the recovery control register of the ingress router to ensure that the parameters of the compression end and the recovery end are consistent.

[0068] In some embodiments of this application, the sparse sensing controller 120 generates the sparsity matrix by: comparing the expected number of tokens to be transmitted with a zero-value mask for each edge in the data dependency graph to obtain the data flow sparsity from each source processing unit cluster to each target processing unit cluster, wherein the data flow sparsity is the ratio of the number of valid tokens in the corresponding communication to the total number of tokens; and filling the data flow sparsity from each source processing unit cluster to each target processing unit cluster into the matrix according to the source-target cluster index to obtain the sparsity matrix.

[0069] Specifically, the data dependency graph is generated by the compiler or runtime system during the task mapping phase and stored in the configuration context for the sparse awareness controller 120 to read. The data dependency graph contains multiple nodes and multiple directed edges. Each node corresponds to a cluster of processing units or a block of computational tasks, and each directed edge represents a data transfer from the source cluster of processing units to the target cluster of processing units. Each edge is also marked with the expected amount of data to be transferred (in bytes).

[0070] After acquiring the data dependency graph during the configuration phase, the sparse sensing controller 120 first converts the amount of communication data corresponding to each edge into the number of tokens. Since data is transmitted between processing unit clusters in units of tokens, and each token carries a fixed number of bytes of data payload, the conversion formula for the number of tokens is: N token = Communication data volume (bytes) ÷ Number of bytes per token payload; The payload bytes per token are equal to the payload width of the data token divided by 8. For example, when the data path is 128-bit wide, each token payload is 16 bytes; when the data path is 32-bit wide, each token payload is 4 bytes. Those skilled in the art can flexibly choose the payload width of the data token according to the actual system design, and this application does not impose specific limitations on it.

[0071] Simultaneously, the sparse-aware controller 120 acquires the zero-value mask corresponding to the communication. The zero-value mask is a bit sequence or array of tags of the same length as the token sequence, where each bit or tag indicates whether the token at the corresponding position is a zero-value token. For example, a tag "1" indicates that the token is a zero-value token (all data is zero or below the contribution threshold), and a tag "0" indicates that the token is a valid token. The zero-value mask can be generated by the compiler during the mapping of computational tasks to processing unit clusters, based on static data flow analysis or historical profile information, and provided to the sparse-aware controller 120. As an example, the compiler can predict the zero-value distribution on each communication edge by analyzing the data dependencies of each node in the computation graph and combining the output distribution characteristics of activation functions (such as ReLU and GELU), thereby generating the zero-value mask.

[0072] After obtaining the total number N of tokens expected to be transmitted. token After the zero-value mask, the sparse-aware controller 120 compares and counts the number of tokens marked as valid and the number of tokens marked as zero in the zero-value mask, and then calculates the sparsity S of the data stream for the communication according to the following formula: S = Number of valid tokens ÷ Total number of tokens (N) token ); Where the number of valid tokens equals N token Subtract the number of tokens marked as zero in the zero-value mask. The sparsity S ranges from 0.0 to 1.0. S=0.0 means that all tokens in the communication are zero-value tokens (all-zero communication), and S=1.0 means that all tokens in the communication are valid data (dense communication). The closer S is to 0.0, the sparser the communication, and the closer it is to 1.0, the denser the communication.

[0073] It should be understood that the aforementioned "valid token" refers to a token carrying valid data (non-zero value data). In some implementations, the criterion for determining a valid token can also be extended to "a token whose data contribution is higher than a preset contribution threshold." Even if the data is not entirely zero, if its value is extremely small and its impact on the final calculation result is negligible, it can still be marked as a "zero-value token" for compression. Those skilled in the art can flexibly set the specific value of this contribution threshold according to the accuracy requirements and computational error tolerance of the specific application scenario; this application does not limit this.

[0074] After performing the data flow sparsity calculation on each edge of the data dependency graph, the sparse-aware controller 120 obtains a series of sparsity values ​​between source-target cluster pairs. Then, the sparse-aware controller 120 arranges these sparsity values ​​into a matrix form according to the indices of the source processing unit cluster and the target processing unit cluster, obtaining the sparsity matrix S.

[0075] Specifically, let the total number of processing unit clusters be M, then the sparsity matrix S is an M×M matrix. S[i][j] represents the data flow sparsity from the i-th processing unit cluster to the j-th processing unit cluster, where i is the source cluster index and j is the target cluster index. For source-target cluster pairs in the data dependency graph that do not have a valid communication edge (i.e., there is no corresponding data transmission at this position), the corresponding position in the sparsity matrix is ​​marked as "no valid communication" (e.g., filled with an invalid value or left empty), and the sparsity-aware controller 120 does not perform subsequent routing path pre-selection for these positions.

[0076] As an example rather than a limitation, assume that there are 4 processing unit clusters in the system (denoted as C0, C1, C2, and C3), and the communication edges in the data dependency graph include C0→C1, C0→C2, C1→C3, and C2→C3. After the above calculation, the sparsity of each communication is S[0][1]=0.219, S[0][2]=0.883, S[1][3]=0.258, and S[2][3]=0.766, respectively, as shown in Table 1 below: Table 1

[0077] As shown in Table 1, for the communication edge C0→C1, the total number of tokens expected to be transmitted is 256, of which 200 are zero-value tokens and 56 are valid tokens. The data flow sparsity S[0][1] = number of valid tokens ÷ total number of tokens = 56 ÷ 256 = 0.219. This value is lower than the preset sparsity threshold Th_sparse = 0.3, so the communication is judged as high sparsity communication. The sparsity-aware controller 120 writes the SBI flag to 1, indicating that the communication is performed by the zero-value compression engine 130 during the running phase.

[0078] For the communication edge C0→C2, the total number of tokens expected to be transmitted is 256, of which 30 are zero-value tokens and 226 are valid tokens. The data flow sparsity S[0][2] = 226 ÷ 256 = 0.883. This value is higher than the preset sparsity threshold Th_sparse = 0.3. Therefore, this communication is judged as regular (dense) communication, the SBI flag is 0, and the zero-value compression engine 130 does not perform zero-value compression on it.

[0079] For the communication edge C1→C3, the total number of tokens expected to be transmitted is 256, of which 190 are zero-value tokens and 66 are valid tokens. The data flow sparsity S[1][3]=66÷256=0.258. This value is lower than Th_sparse=0.3, so the SBI flag is 1 (skip).

[0080] For the communication edge C2→C3, the total number of tokens expected to be transmitted is 256, of which 60 are zero-value tokens and 196 are valid tokens. The data flow sparsity S[2][3]=196÷256=0.766. This value is higher than Th_sparse=0.3, and the SBI flag is 0 (normal).

[0081] Arranging the above calculation results into a matrix form according to the source-target cluster index, we obtain the following 4×4 sparsity matrix S: S = [ [ — , 0.219, 0.883, ? ], [ — , — , ? , 0.258 ], [ — , — , — , 0.766 ], [ — , — , — , — ] ].

[0082] Where “—” indicates no effective communication (including self-communication between the source cluster and the target cluster, and cluster pairs without communication edges), and “?” indicates that the source-target cluster pair has no effective communication at this computation layer (the sparse sensing controller does not pre-select a path). It should be understood that the specific values ​​in the above examples are only for ease of understanding and do not constitute a limitation on the scope of protection of this application.

[0083] It should be noted that during the sparsity calculation process described above, when the zero-value mask for a certain communication is missing or the number of valid data samples within the scanning window is insufficient, the sparse sensing controller 120 cannot obtain enough information to accurately calculate the proportion of valid tokens. In this case, the sparse sensing controller 120 can directly use the lower bound P of the valid data proportion marked on the edge of the data dependency graph. min This serves as the sparsity of the data stream for this communication. Specifically, the sparsity-aware controller 120 sets the value of S[i][j] to be no less than P. min This is to avoid misclassifying communication known to the compiler as dense communication as high-sparse communication and incorrectly marking it as "skip zero data".

[0084] In some embodiments of this application, each element of the sparsity matrix S is stored in an 8-bit fixed-point fractional format with a resolution of 1 / 256, meaning each element can represent 256 different values ​​between 0 and 1 with a step size of 1 / 256. This fixed-point quantization format reduces storage overhead while maintaining the required sparsity discrimination accuracy. The sparsity matrix S is stored in a sparsity matrix memory within the sparsity sensing controller 120, with a capacity corresponding to M × M × 8-bit. For example, when M = 16, the size of the sparsity matrix memory is 16 × 16 × 8-bit = 2048-bit (i.e., 2Kb).

[0085] The generation process of the sparsity matrix S is now complete. This matrix serves as pivotal intermediate data for subsequent path pre-selection and sparse skip flag writing, providing a quantitative basis for sparse-aware communication throughout the interconnected system.

[0086] It should be noted that the above-mentioned data dependency graph structure, the conversion relationship between the number of tokens and the amount of data, the definition and acquisition method of the zero-value mask, and the lower bound P of the effective data ratio are not included. min The fallback use of the 8-bit fixed-point quantization format for the sparse matrix, and the handling method for situations where there are no effective communication positions in the matrix, are all optional implementations of this application. These features can be selected or modified in other embodiments according to actual needs, without departing from the protection scope of this application.

[0087] It should also be understood that the four clusters and four communication edges in the above example are only for the purpose of understanding the technical solution of this application and do not constitute a limitation on the scope of protection of this application. In practical applications, the number of processing unit clusters can be any positive integer (e.g., 16, 64 or more), and the number of communication edges in the data dependency graph can be more or less. The sparse sensing controller 120 processes each communication edge one by one according to the same principle, generates an M×M sparsity matrix, and completes the writing of the SBI flag.

[0088] In some embodiments of this application, the sparse sensing controller 120 further includes a path preselector, which is used to allocate time slots for communication between multiple processing unit clusters sharing the same bus segment. The path preselection table records the time slot index corresponding to each communication, and each processing unit cluster initiates communication in the corresponding time slot according to the time slot index.

[0089] Specifically, in a segmented row-column routing network, multiple processing unit clusters may share the same segmented row bus or the same segmented column bus. For example, when there are four processing unit clusters in a row, all four clusters are connected to the same segmented row bus; when there are four processing unit clusters in a column, all four clusters are connected to the same segmented column bus. If multiple clusters simultaneously attempt to send data on the same bus segment, bus contention will occur, leading to data collisions and transmission errors. To solve this problem, this application employs a Time Division Multiple Access (TDMA) mechanism. By pre-allocating different time slots for each communication sharing the same bus segment, each communication occupies the bus sequentially in different time periods, thereby avoiding bus collisions.

[0090] The path preselector is a logical unit within the sparse sensing controller 120, which performs time slot allocation operations during the configuration phase. The path preselector first obtains information on all source-target cluster pairs on the current segment bus that need to initiate communication, and then allocates a unique time slot index to each communication according to a preset time slot allocation strategy. The time slot allocation strategy can employ sequential allocation (i.e., allocation according to communication number or priority), round-robin allocation, or weighted allocation based on the amount of communication data, etc., and this application does not specifically limit its application to these methods.

[0091] Each segmented row bus and each segmented column bus maintains its own independent time slot allocation table. Assuming the number of clusters shared on a certain bus segment is K (K is a positive integer, e.g., K=4 or K=8), then the total number of time slots on that bus segment is K. The path preselector assigns a time slot index to each cluster mounted on that bus segment. The time slot index ranges from 0 to K-1 (or 1 to K), and each time slot index uniquely corresponds to one cluster. Each cluster can only initiate data transmission within its assigned time slot; in other time slots, it remains in a listening state and does not send data to the bus.

[0092] In some embodiments of this application, the width of each time slot is a fixed number of clock cycles. For example, the width of each time slot can be set to 32 clock cycles. Within each time slot, the cluster owning that time slot can send one or more data tokens. The time slot width can be flexibly configured according to the data token transmission rate and bus width, and this application does not specifically limit it.

[0093] The aforementioned time slot index is recorded in the entry of the path preselection table. In addition to recording the source cluster identifier, destination cluster identifier, routing path vector, and sparse skip flag (SBI), the path preselection table also includes a time slot index field (e.g., 3-bit wide, used to represent time slot numbers 0-7). Once a communication is assigned a specific time slot, this time slot index is written into the corresponding entry in the path preselection table. During runtime, this is read by the router interface units of each processing unit cluster to determine when the cluster initiates communication.

[0094] During operation, each processing unit cluster initiates communication in the corresponding time slot according to the time slot index recorded in the path preselection table. Specifically, each processing unit cluster's router interface unit has a local time slot counter, which is synchronized with the global time slot reference (e.g., via a global clock and frame synchronization signal). When the value of the local time slot counter equals the time slot index recorded in the path preselection table, the router interface unit allows data tokens to be issued from within the cluster and to occupy the bus; when the counter value does not equal the cluster's time slot index, the router interface unit maintains the bus driver in a tri-state (high impedance) state and does not send data to the bus. For the receiving end, all clusters connected to the same bus segment listen to the bus in each time slot and receive data tokens destined for their cluster.

[0095] Since time slots are pre-allocated by the path preselector during the configuration phase, each processing unit cluster only needs to send data sequentially according to the time slot index in the path preselection table during the runtime phase, without the need for runtime bus arbitration and retries. This static time slot allocation mechanism eliminates runtime arbitration overhead, making the initiation waiting time of each communication deterministic and independent of whether there is contention.

[0096] To help those skilled in the art better understand the impact of time slot allocation on communication latency, the following example illustrates the situation using a bus segment sharing 8 clusters, each with a time slot width of 32 cycles. When 8 clusters share the same bus segment, the path preselector allocates 8 time slots, numbered 0-7, with each time slot having a width of 32 cycles. A complete time slot frame (i.e., the sum of 8 time slots) is 8 × 32 = 256 cycles. If a communication is allocated to the s-th time slot (s = 1-8), then that communication must wait for the previous (s-1) time slots to finish before initiating transmission in each time slot frame; the initiation waiting time is (s-1) × 32 cycles. In the worst case, if a flow is queued in the 8th time slot, the initiation waiting time is 7 × 32 = 224 cycles; once it is the turn of that time slot, the single-hop transmission time of that flow on the segment bus is always 1 cycle (the bus carries one token per cycle). This deterministic delay allows system designers to accurately estimate the end-to-end delay of each communication during the configuration phase, which helps meet real-time requirements.

[0097] In some embodiments of this application, each bus segment supports a maximum of 8 processing unit clusters shared, corresponding to 8 time slots, with the time slot index field being 3 bits. When the number of clusters on a bus segment is less than 8, unused time slots can be marked as idle or allocated by the path preselector to communications requiring higher bandwidth (i.e., multiple time slots are allocated to this communication within the same frame). This flexible time slot allocation method allows bus bandwidth resources to be configured on demand according to the actual needs of each communication, improving bus utilization.

[0098] It should also be understood that the aforementioned time slot allocation is completed by the path preselector during the configuration phase and is a static allocation. In some embodiments of this application, when the data flow sparsity changes drastically during the runtime phase, the sparsity-aware controller 120 can trigger a path re-preselection process. The path preselector re-allocates time slots for the affected bus segments and writes the updated time slot index into the path preselection table. This dynamic reconfiguration capability allows time slot allocation to adapt to changes in computational load, further improving the system's flexibility.

[0099] It should be noted that the aforementioned path preselector and its time slot allocation function, as well as the related specific quantization parameters such as time slot width (32 cycles), maximum number of shared clusters (8), and time slot index bit width (3-bit), are all optional implementations of this application. These features can be flexibly adjusted in other embodiments according to the actual bus speed, clock frequency, and array size, without departing from the protection scope of this application.

[0100] In some embodiments of this application, the sparse sensing controller 120 described above is also configured to perform dynamic sparsity adaptive operation during the runtime phase.

[0101] Specifically, the sparse sensing controller 120 continuously monitors the real-time sparsity of each communication link during operation. The sparse sensing controller 120 internally includes a monitoring state machine, which enters the monitoring state from an idle state during operation, monitoring each communication link with a continuous monitoring window size (W). monitor Each data token is used as a monitoring window for sampling and statistics. After all tokens within each monitoring window have been transmitted, the sparse awareness controller 120 counts the number of valid tokens within that window, calculates the real-time sparsity within the current window, and updates the sparsity matrix S[i][j] to this real-time sparsity value. For example, W... monitor It can be set to 1024 consecutive tokens.

[0102] After updating the sparsity matrix S[i][j], the sparse-aware controller 120 calculates the deviation ΔS between the updated sparsity value and the estimated value during the configuration phase. If the deviation ΔS exceeds a preset deviation threshold ΔTh (e.g., ΔTh = 0.15), the sparse-aware controller 120 performs at least one of the following adaptive adjustment operations: (1) The sparse sensing controller 120 sends an update instruction to the sparse skip flag (SBI) register corresponding to the communication, adjusting the preset sparse threshold Th. sparse Or preset jump step size N skip The specific values ​​are adjusted to adapt to the actual changes in the sparsity of the current computing layer data flow.

[0103] (2) If the sparsity changes drastically, i.e. the deviation ΔS exceeds the threshold of drastic change (e.g. ΔS > 0.5), the sparsity sensing controller 120 temporarily stores the currently unprocessed data tokens, triggers the path re-pre-selection process, completes the partial refresh of the path pre-selection table within a preset number of cycles (e.g. 8 cycles), writes the updated path pre-selection table into the corresponding router interface unit, and then restores the normal transmission of data tokens.

[0104] In some embodiments of this application, the monitoring state machine includes four states: Idle, Monitor, Update, and Refresh. The Idle state is the initial state, waiting to enter the next monitoring cycle; the Monitor state is used to accumulate tokens within the monitoring window and calculate real-time sparsity; the Update state is used to send update instructions to the SBI register to adjust the Th... sparse or N skip The refresh status is used to trigger the path reselection process and wait for the path selection table to be refreshed.

[0105] It should be understood that the aforementioned dynamic sparsity adaptation operation is autonomously executed by the sparse-aware controller 120 during the runtime phase, running in parallel with the zero-value compression operation of the zero-value compression engine 130 without interference. Through this monitoring and adaptation mechanism, the sparse-aware controller 120 enables the interconnected system to adapt to changes in data flow sparsity between different computing layers and within different time windows of the same computing layer, without interrupting the computing task.

[0106] It should also be understood that the above W monitor The specific quantitative parameters such as ΔTh=0.15, ΔS>0.5 triggering path re-preselection, and local refresh delay not exceeding 8 cycles are all exemplary implementations of this application, and not mandatory limiting values. Those skilled in the art can flexibly configure the specific values ​​of the above parameters according to the latency tolerance and accuracy requirements of the actual application scenario, and this application does not make specific limitations in this regard.

[0107] In some embodiments of this application, the local fully interconnected network within each processing unit cluster also includes a Co-Arbiter. This Co-Arbiter is used to uniformly schedule and arbitrate data requests from intra-cluster processing units and data tokens from inter-cluster segmented row-column routing networks, thereby achieving the integration of cooperative data streams for intra-cluster and inter-cluster communication.

[0108] Specifically, at the entry point of the local fully interconnected network (such as the Crossbar) of the processing unit cluster, data may simultaneously originate from two directions: one is data from other processing units within the cluster (intra-cluster local communication); the other is data from the segmented row-column routing network (inter-cluster communication, including regular data tokens and zero-value compressed packets ZCP). The collaborative arbitrator arbitrates these two data streams according to a preset priority strategy, determining which data stream should be sent to the local fully interconnected network for distribution in the current cycle.

[0109] For example, the collaborative arbitrator arbitrates according to the following priorities from highest to lowest: First priority (highest): Zero-value compressed packets (ZCPs) from the inter-cluster segmented row-column routing network. Since ZCPs carry counter information, they need to be decompressed and restored to zero-value tokens before being sent into the cluster to prevent ZCPs from accumulating in the segmented row-column routing network and affecting the transmission of subsequent zero-value compressed packets.

[0110] Second priority: Local communication requests between processing units within the cluster. Intra-cluster communication involves computational tasks currently being executed in the current cycle, is sensitive to latency, and directly affects the current computation progress; therefore, it is given a higher priority.

[0111] Third priority (lowest): Regular data tokens (i.e., non-ZCP data tokens) from the inter-cluster segmented row-column routing network. Regular data tokens have a relatively high latency tolerance and can be cached in the inter-cluster network, thus they are given a lower priority.

[0112] In some embodiments of this application, the collaborative arbitrator employs a weighted round-robin (WRR) scheduling strategy, which allocates access opportunities to the local fully interconnected network among data of different priority categories according to a preset weight ratio. For example, the WRR weight register of the collaborative arbitrator stores two sets of weight values, corresponding to inter-cluster communication weight (weight A) and intra-cluster communication weight (weight B), respectively. The default value can be inter-cluster:intra-cluster = 2:1. The configurable range can be from 1:9 to 9:1, and those skilled in the art can flexibly adjust it according to the actual load conditions.

[0113] In some embodiments of this application, the weights of the collaborative arbitrator are dynamically adjusted based on the current busy status of the processing units within the cluster. Specifically, the collaborative arbitrator obtains the current working status of each processing unit within the cluster (such as computing unit utilization or idle time ratio). When the average utilization of the processing units within the cluster is lower than a first utilization threshold (e.g., 70%), it indicates that the computing resources within the cluster are relatively idle. At this time, the collaborative arbitrator increases the weight of inter-cluster communication, allowing more external data to quickly enter the cluster to fill idle processing units. When the average utilization of the processing units within the cluster is higher than a second utilization threshold (e.g., 90%), it indicates that the computing resources within the cluster are under high load. At this time, the collaborative arbitrator reduces the weight of inter-cluster communication, prioritizing ensuring that local computing is not interrupted and avoiding excessive influx of external data that could cause cluster congestion. For example, when the cluster utilization is lower than 70%, the collaborative arbitrator adjusts the ratio of inter-cluster weight to intra-cluster weight from the default 2:1 to 3:1 or higher; when the intra-cluster utilization is higher than 90%, the ratio is adjusted to 1:1 or lower. The first and second utilization thresholds mentioned above can be flexibly configured according to actual application scenarios, and this application does not impose specific limitations on them.

[0114] In some embodiments of this application, the arbitration decision delay of the collaborative arbitrator is a fixed number of cycles (e.g., 2 cycles). This arbitration decision delay includes: receiving request signals from both intra-cluster and inter-cluster data streams, comparing priorities, executing a weighted round-robin scheduling algorithm, and outputting the arbitration result to the selector control terminal of the local fully interconnected network. Since the arbitration decision delay is deterministic, system designers can accurately estimate the arbitration overhead, which is beneficial for meeting real-time requirements.

[0115] It should be understood that the specific implementation methods of the above-mentioned collaborative arbitrator, such as the three-level priority strategy, WRR scheduling mechanism, dynamic weight adjustment, utilization thresholds (70% and 90%), and arbitration decision delay (2 cycles), are all exemplary implementations of this application, and not mandatory limiting schemes. Those skilled in the art can flexibly select and configure arbitration strategies and parameters according to different needs for arbitration delay, bandwidth allocation fairness, and computational load balancing in actual application scenarios, without departing from the protection scope of this application.

[0116] As described above, this application continuously monitors the real-time sparsity of each communication link during the operation phase through a sparsity-aware controller and adjusts or reselects paths based on deviation trigger thresholds. This enables the interconnect system to adaptively respond to changes in data flow sparsity between different computing layers and within the same computing layer at different time windows, without interrupting computing tasks. Simultaneously, by setting a collaborative arbitrator at the local full interconnection network entry point of the processing unit cluster, priority arbitration and weighted round-robin scheduling are performed on inter-cluster zero-value compressed packets, intra-cluster local communication requests, and inter-cluster regular data tokens. The weights of inter-cluster and intra-cluster communication are dynamically adjusted according to the busy status of the intra-cluster processing units. While ensuring timely decompression of zero-value compressed packets and preventing ZCP accumulation, collaborative optimization of intra-cluster computing and inter-cluster communication is achieved, further improving the utilization rate of processing units and the overall system throughput.

[0117] Thus far, the above embodiments of this application have described the core architecture of the processing unit interconnection system and the basic working principle of each module. In the above embodiments, the specific parameters such as the number of processing unit clusters, the number of processing units contained in each cluster, the bus width, and the storage capacity are all exemplary descriptions. Those skilled in the art can flexibly select and configure them according to the target array size, chip area constraints, and actual needs of the application scenario.

[0118] To help those skilled in the art better understand the implementation of the technical solutions of this application in practical applications, the following description, in conjunction with several specific embodiments, provides further details. Embodiment 1 describes an implementation of a 4×4 cluster structure (256 processing units) and its application in LLM inference; Embodiment 2 describes an implementation of an 8×8 cluster structure (512 processing units) and its application in three-dimensional integrated circuits; Embodiment 3 describes the application of an asymmetric cluster structure in a MoE (Hybrid Expert) model; and Embodiment 4 describes an optical interconnect implementation when the chip size is scaled up to 1024 processing units or more. These embodiments can be combined or implemented individually, and identical parts will not be repeated.

[0119] Example 1 (4×4 cluster structure, 256 PE array).

[0120] In this embodiment, the above-mentioned processing unit interconnection system can adopt a 4×4 cluster structure to form an array of 256 processing units.

[0121] Specifically, each processing unit cluster consists of 4×4=16 processing units. Each processing unit includes a 16-bit fixed-point multiply-accumulate unit (MAC), a 16KB local register file (RF), and a routing interface unit (RIU). The 16 processing units are interconnected through a 16×16 local crossbar. The crossbar is implemented using a MUX-Tree structure, with a wiring area of ​​approximately 0.08 mm².2 (28nm process).

[0122] Sixteen clusters of processing units form a 4×4 array topology, totaling 256 processing units. The physical size of the array is approximately 2.8×2.8mm. 2 (Including the area of ​​routing and sparse-aware controllers). Inter-cluster SRT routing is achieved through 4 row buses (each connecting 4 clusters in a row) and 4 column buses (each connecting 4 clusters in a column).

[0123] In this embodiment, the segmented row-column routing network includes four row bus segments and four column bus segments, each bus segment being 128 data bits plus 25 control bits wide. Each bus segment is connected to the egress of each processing unit cluster via a router interface unit (RT). The router interface unit internally contains a 4×4 crosspoint node for switching connections between the four processing unit clusters in the same row or column. A detailed hardware configuration diagram of this embodiment can be found in [reference needed]. Figure 2 .

[0124] The sparse sensing controller is integrated on one side of the array and contains: a sparse matrix memory (256×8-bit SRAM, 2Kb in total), a path preselection lookup table (256×28-bit ROM, 7Kb in total) and a monitoring state machine (4 states, approximately 12K gates in total).

[0125] The following uses Attention calculation in LLM inference as an example to illustrate the workflow of this embodiment.

[0126] During the configuration phase (approximately 2000 cycles), the configuration context contains a description of the data dependency graph of the current computation layer (e.g., Attention Layer 5). The sparse perceptual controller extracts the block mapping relationships of the Q, K, and V matrices from the data dependency graph. In the QK... T In the computation, the Q matrix is ​​mapped row-wise to four processing unit clusters, and the K matrix is ​​mapped column-wise to another four processing unit clusters. The sparsity-aware controller analyzes the sparsity of communication between these clusters—in scenarios with long sequences (such as 4K tokens), QK... T The upper triangular portion of the matrix (the causal mask coverage area) has a sparsity of 100% (all skipped), while the lower triangular portion has a sparsity of approximately 30% to 50%. Based on this, the sparsity-aware controller generates a sparse skip flag (SBI) and a routing path for each communication pair.

[0127] During operation, data tokens are transmitted between processing unit clusters along pre-selected paths. For zero-value regions covered by causal masks, the zero-value compression engine skips all tokens, transmitting only a single zero-value compressed packet (ZCP) representing the zero values ​​of the entire region; a single skip can reach 128 to 512 consecutive zero-value tokens. T After the phase is completed, the Softmax results are delivered to all target processing unit clusters within one cycle through the broadcast mode of the segmented row-column routing network—multicast via the column segment bus.

[0128] Example 2 (8×8 cluster structure, 512 PE array).

[0129] In this embodiment, the above-mentioned processing unit interconnection system can adopt an 8×8 cluster structure, which is suitable for larger-scale LLM inference scenarios.

[0130] Specifically, each processing unit cluster consists of 8×8=64 processing units, and the local fully interconnected network within the cluster is expanded to a scale of 64×64 (implemented using a multi-level MUX tree, with an area of ​​approximately 0.45 mm²). 2 Eight clusters of processing units form a 2×4 or 8×1 array topology, for a total of 512 processing units.

[0131] The inter-cluster SRT network is correspondingly expanded to 2 row bus segments and 4 column bus segments (corresponding to a 2×4 array topology), or 8 row bus segments and 1 column bus segment (corresponding to an 8×1 array topology). The path preselection table depth of the sparse sensing controller is expanded to 64 entries (i.e., 8×8=64 cluster pairs).

[0132] In 3D integrated circuit (3D IC) implementations, SRAM dies can be vertically stacked below the logic dies via through-silicon vias (TSVs). The number of TSVs is approximately 4000, and the inter-layer communication bandwidth is approximately 1TB / s. This vertical stacking method can effectively reduce memory access latency and further improve overall system performance.

[0133] Example 3 (Asymmetric cluster structure, heterogeneous array).

[0134] In this embodiment, the above-mentioned processing unit interconnection system can adopt an asymmetric cluster structure, which is suitable for the MoE (Mixture-of-Experts) model.

[0135] Specifically, different processing unit clusters can be configured with different sizes. In the MoE model, the core expert cluster (carrying the resident / high-frequency calling expert network that is continuously activated by gated routes with a high probability in the MoE model) uses an 8×8 cluster (64 processing units), and the routing expert cluster (carrying the expert network that is dynamically and sparsely selected by gated routes and is only active when a token is routed to the expert) uses a 4×4 cluster (16 processing units).

[0136] Introducing the "H" factor in the sparsity analysis of the sparse sensing controller e This factor is defined as H. e = (1 / T)·Σ{t=1..T} ge(t), where ge(t) is the gating probability (or indicator function: 1 if successful) of the MoE gating network routing the input token to expert e at time t, and T is the statistical window (obtained during the configuration phase from calibration run or static analysis of gating weights). H e ∈[0,1], the larger the value, the more frequently the expert is routed.

[0137] For expert clusters with popularity exceeding a preset threshold (core expert clusters), the sparse sensing controller increases the time slot allocation weight of the bus in the row / column segment where it is located; for expert clusters with popularity below the preset threshold (routing expert clusters), the sparse sensing controller merges their time slots to reduce bus resource waste.

[0138] The weights of the Co-Arbiter are also adjusted accordingly: the ratio of intra-cluster communication weight to inter-cluster communication weight of the core expert cluster is set to 1:2 (prioritizing remote data reception), and the weight of the routing expert cluster is set to 3:1 (prioritizing local routing decisions).

[0139] Example 4 (Optical Interconnect Extension).

[0140] In this embodiment, when the chip size is expanded to more than 1024 processing units, the row / column segment bus of the inter-cluster SRT can be implemented through silicon photonic waveguides.

[0141] Specifically, each bus segment corresponds to a wavelength division multiplexing (WDM) channel. A microring resonator array achieves 8-channel parallel transmission across 8 wavelengths, with a single-wavelength bandwidth of 25Gbps and a total bandwidth of 200Gbps. The electrical cross-connect nodes in the router interface unit (RT) are replaced with microring resonator and photodetector pairs.

[0142] This embodiment can replace electrical interconnect with optical interconnect, which can significantly reduce the power consumption and latency of long-distance bus transmission and further overcome the physical bottlenecks of electrical interconnect in terms of bandwidth and power consumption.

[0143] The beneficial effects of the above-mentioned interconnection system of processing units can be further verified by comparing the following technical indicators. Table 2 shows the comparison results of the embodiments of this application and the prior art (pure Mesh topology) on several key indicators.

[0144] Table 2

[0145] Based on the above comparative data, the beneficial effects of the embodiments of this application are as follows.

[0146] Effect 1: Significantly reduces communication latency between long-distance PEs.

[0147] This application utilizes a two-level communication architecture of "intra-cluster Crossbar direct connection + inter-cluster SRT segmented traversal" to compress long-distance communication that requires hop-by-hop forwarding in the original Mesh topology into a maximum of two routes (row segment + column segment). The communication latency is reduced from being linearly positively correlated with array size (O(N)) to being independent of array size (O(1)). In a 16×16 array, the longest communication distance is reduced from 32 cycles to 2 cycles, a reduction of 93.7%, effectively eliminating the data flow bottleneck problem caused by long-distance communication.

[0148] Effect 2: Significantly improves the effective bandwidth utilization in sparse data scenarios.

[0149] This application utilizes zero-value detection in a zero-value compression engine and the ZCP compression mechanism to merge zero-value data that originally occupied an equal amount of bandwidth in sparse communication into lightweight compressed packets carrying only count values, significantly reducing bandwidth overhead. In high-sparseness scenarios such as Attention and MoE, the effective bandwidth utilization rate is increased from 30% to over 85%. Simultaneously, the dynamic sparsity adaptation mechanism executed by the sparse perception controller during runtime ensures a smooth transition between different sparsity characteristics without interrupting computational tasks.

[0150] Effect 3: Comprehensive improvement of PE utilization rate and energy efficiency ratio.

[0151] This application achieves balanced utilization of intra-cluster and inter-cluster communication resources through weighted round-robin scheduling of a collaborative arbitrator, significantly reducing PE idle waiting time. Under typical AI inference workloads, the average PE utilization rate increases from 55% to 82%. Simultaneously, due to the power reduction effect brought about by zero-value skipping and ZCP compression, the overall system energy efficiency ratio (TOPS / W) is improved by approximately 2.1 times, demonstrating significant application advantages in power-sensitive edge computing and data center scenarios.

[0152] It should be understood that the values ​​in Table 2 above are exemplary data obtained under specific process nodes (28nm) and specific test loads, and do not constitute a limitation on the scope of protection of this application. Specific values ​​may vary under different process nodes, different array sizes, and different application scenarios, but the beneficial effects of latency reduction, bandwidth utilization improvement, and area optimization brought about by the core technical solution of this application are universally applicable.

[0153] like Figure 3 The diagram shown is a flowchart of a processing unit interconnection method provided in an embodiment of this application. The method includes the following steps: Step S301: Obtain the data dependency graph of the current computing layer, and generate a sparsity matrix based on the data dependency graph. Each element of the sparsity matrix represents the sparsity of the data flow from the source processing unit cluster to the target processing unit cluster.

[0154] Step S302: Based on the sparsity matrix and the preset sparsity threshold, pre-select routing paths for the source processing unit cluster and the target processing unit cluster that have inter-cluster communication and generate a path pre-selection table.

[0155] Step S303: For communications in the sparsity matrix that are below the sparsity threshold, write a sparse skip flag in the path preselection table.

[0156] Step S304: Based on the sparse skip flag in the path preselection table, merge the consecutive zero-value tokens issued from the source processing unit cluster into a zero-value compressed packet carrying a zero-value token count value, and send the zero-value compressed packet to the target processing unit cluster via a segmented row-column routing network.

[0157] For a detailed description of each of the above steps, please refer to the relevant descriptions in the foregoing system embodiments, which will not be repeated here.

[0158] This application also provides a board that includes the aforementioned wafer-level chip. Therefore, the board can achieve the same technical effects as the wafer-level chip, which will not be described in detail here.

[0159] This application also provides an electronic device, which includes the aforementioned circuit board. Since the electronic device provided in this application includes the aforementioned circuit board, it can achieve the same technical effects as the circuit board described above, and will not be described again here.

[0160] In the description of this specification, the references to terms such as "an embodiment," "a specific embodiment," "some embodiments," "for example," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0161] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of this application. It should be understood that the above descriptions are merely specific embodiments of this application and are not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A processing unit interconnection system, characterized in that, include: Multiple processing unit clusters, each of the processing unit clusters including multiple processing units and a local fully interconnected network, wherein the local fully interconnected network is used to enable direct communication between any two processing units within the cluster; A segmented row-column routing network is coupled between the multiple processing unit clusters to enable inter-cluster communication between different processing unit clusters; A sparsity-aware controller, coupled to the segmented row-column routing network, is configured to: during the configuration phase, generate a sparsity matrix based on the data dependency graph of the current computation layer; pre-select routing paths for source processing unit clusters and target processing unit clusters with inter-cluster communication based on the sparsity matrix and a preset sparsity threshold, and generate a path pre-selection table; and For communications in the sparsity matrix that are below the sparsity threshold, a sparse skip flag is written into the path preselection table; A zero-value compression engine is used to merge consecutive zero-value tokens issued from the source processing unit cluster into a zero-value compressed packet carrying a zero-value token count value based on the sparse skip flag in the path preselection table, and send the zero-value compressed packet to the target processing unit cluster via the segmented row-column routing network.

2. The processing unit interconnection system as described in claim 1, characterized in that, The segmented row-column routing network includes multiple segmented row buses and multiple segmented column buses. Each segmented row bus connects multiple processing unit clusters located in the same row, and each segmented column bus connects multiple processing unit clusters located in the same column.

3. The processing unit interconnection system as described in claim 1, characterized in that, The segmented row-column routing network is also used during runtime to transmit data tokens and the zero-value compressed packets between processing unit clusters according to the path preselection table.

4. The processing unit interconnection system as described in claim 1, characterized in that, It also includes a router interface unit corresponding to the target processing unit cluster, the router interface unit being used to recover the zero-value token based on the zero-value compressed packet.

5. The processing unit interconnection system as described in claim 1, characterized in that, The sparse-aware controller generates the sparsity matrix by: for each edge in the data dependency graph, comparing the expected number of tokens to be transmitted with a zero-value mask to obtain the data flow sparsity from each source processing unit cluster to each target processing unit cluster, wherein the data flow sparsity is the ratio of the number of valid tokens in the corresponding communication to the total number of tokens; filling the data flow sparsity from each source processing unit cluster to each target processing unit cluster into the matrix according to the source-target cluster index to obtain the sparsity matrix.

6. The processing unit interconnection system as described in claim 1, characterized in that, The sparse sensing controller pre-selects the routing path as follows: when the source processing unit cluster and the target processing unit cluster are in the same row, the row segment bus is selected as the routing path; when the source processing unit cluster and the target processing unit cluster are in the same column, the column segment bus is selected as the routing path; when the source processing unit cluster and the target processing unit cluster are in different rows and different columns, the row segment bus and the column segment bus are selected as the routing paths in sequence.

7. The processing unit interconnection system as described in claim 6, characterized in that, The sparse sensing controller includes a path preselector, which is used to allocate time slots for communication between multiple processing unit clusters sharing the same bus segment. The path preselection table records the time slot index corresponding to each communication, and each processing unit cluster initiates communication in the corresponding time slot according to the time slot index.

8. The processing unit interconnection system as described in claim 1, characterized in that, The zero-value compression engine merges consecutive zero-value tokens into a zero-value compressed package, including: when the number of consecutive zero-value tokens reaches a preset jump step, merging the preset jump step number of zero-value tokens into one zero-value compressed package.

9. The processing unit interconnection system as described in claim 1, characterized in that, The zero-value token is a data token whose validity flag is invalid or whose data load is all zero.

10. The processing unit interconnection system as claimed in claim 1, characterized in that, The zero-value compressed package also includes a type identifier field, which is used to distinguish between data tokens, zero-value compressed packages, and configuration packages.

11. A method for interconnecting processing units, characterized in that, include: Obtain the data dependency graph of the current computing layer, and generate a sparsity matrix based on the data dependency graph. Each element of the sparsity matrix represents the sparsity of the data flow from the source processing unit cluster to the target processing unit cluster. Based on the sparsity matrix and the preset sparsity threshold, a path pre-selection table is generated for the source processing unit cluster and the target processing unit cluster that have inter-cluster communication. For communications in the sparsity matrix that are below the sparsity threshold, a sparse skip flag is written into the path preselection table; Based on the sparse skip flag in the path preselection table, consecutive zero-value tokens issued from the source processing unit cluster are merged into a zero-value compressed packet carrying a zero-value token count value, and the zero-value compressed packet is sent to the target processing unit cluster via a segmented row-column routing network.

12. A circuit board, characterized in that, Includes the processing unit interconnection system as described in any one of claims 1-10.

13. An electronic device, characterized in that, Includes the board as described in claim 12.

Images