Input and output system and method supporting multicast transmission

Abstract

The invention provides an input and output system and method supporting multicast transmission, a multicast routing engine is added in an input and output chip, the multicast routing engine is inserted between an interconnection bus and a computing power exchange interface, and the multicast routing engine comprises an address resolution module, a multicast routing module, a multicast routing module and a multicast routing module, obtaining a multicast label of the to-be-processed data, and matching a target computing power unit according to the multicast label; and the data copying module copies the to-be-processed data according to the number of the matched target computing power units, and sends the to-be-processed data to the target computing power units through the corresponding computing power exchange interfaces. A hardware-level multicast routing engine is introduced into an input / output chip, so that one-time copying and simultaneous transmission of data are realized, the data transmission efficiency is improved, the delay is reduced, efficient computing power parallel operation is supported, and the balance between data handling efficiency and computing power parallelism is realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of chip design and artificial intelligence operation, and particularly relates to an input / output system and method supporting multicast transmission. BACKGROUND

[0002] In a heterogeneous AI computing system, the IO DIE (Input / Output Die) plays a crucial role as the key hub connecting the computing power DIE (such as NPU / TPU) with external storage and network. Traditional data path design usually only supports Unicast transmission mode, which means that the same data source can only transmit data to one target computing power DIE within the same time window. This limitation poses a serious challenge to scenarios that require multi-operator parallel operation on the same data source (such as convolution, pooling, normalization, etc. on the same image data).

[0003] Firstly, when using the traditional Unicast mode, in order to achieve multi-operator parallel operation, the data source must be copied multiple times (such as 3 times of Unicast transmission) through the IO DIE to be sent to different computing power DIEs. This not only leads to three times the occupation of bus bandwidth, but also makes the computing units of the computing power DIEs have low utilization due to waiting for data, with actual utilization rate less than 40%. In addition, in the case of multi-operator serial processing, the total delay is equal to the sum of the processing times of each operator; even if parallel processing is attempted, data synchronization relies on a complex handshake protocol, which introduces additional nanosecond-level delay, further affecting overall efficiency.

[0004] With the expansion of system size, especially when the number of computing power DIEs increases to more than 8, the size of the routing table under the Unicast mode will grow in proportion to O(N2), leading to a sharp increase in the complexity of the IO DIE control logic and increasing the difficulty and cost of system maintenance.

[0005] In order to solve the above problems, existing technologies have proposed several optimization schemes, but each method has its limitations. On-chip bus broadcasting can transmit data to all computing power DIEs at once, but due to the lack of selective multicast capability, it will cause non-related computing power DIEs to process invalid data, resulting in unnecessary power consumption increase of more than 50%. Data replication at the software layer relies on the CPU to control the IO DIE multiple times of Unicast, which has microsecond-level delay and cannot meet the real-time AI inference requirements, especially in application scenarios such as autonomous driving that require response time not exceeding 100 nanoseconds. Finally, the method of using a dedicated multicast chip externally increases hardware cost and system complexity, and due to the round-trip delay of PCIe interface being greater than 1 microsecond, it cannot meet the performance standards required by chip-level parallel computing.

[0006] In summary, the current solution fails to effectively solve the problems of low data transfer efficiency, high operator coordination delay and poor system scalability in heterogeneous AI computing systems, and there is an urgent need for a new mechanism to improve data transfer efficiency, reduce delay and support efficient multi-operator parallel operation. SUMMARY

[0007] Based on the above, the present application provides an input-output system and method supporting multicast transmission, aiming to solve the technical problems of low data transmission efficiency and high delay in the prior art.

[0008] An input-output system supporting multicast transmission, comprising an input-output chip for input-output management of data of each computing power unit, characterized in that a multicast routing engine is added to the input-output chip, the multicast routing engine is inserted between an interconnection bus and a computing power exchange interface, each computing power exchange interface corresponds to a computing power unit, and the multicast routing engine comprises:

[0009] An address resolution module is configured to resolve the received data to be processed, obtain a multicast tag of the data to be processed, and match a target computing power unit according to the multicast tag;

[0010] A data duplication module is connected to the address resolution module and is configured to duplicate the data to be processed according to the number of matched target computing power units when the number of matched target computing power units is greater than 1, and send the data to be processed to the target computing power units through the corresponding computing power exchange interfaces.

[0011] Further, the multicast routing engine further comprises:

[0012] A traffic arbitrator connected to the data duplication module is configured to dynamically allocate bandwidth to the target computing power units to be sent;

[0013] The data duplication module sends the data to be processed to the target computing power units through the corresponding computing power exchange interfaces according to the allocated bandwidth.

[0014] Further, the input-output chip pre-stores a multicast routing table formed based on a hash algorithm, and the multicast routing table maintains a plurality of groups of computing power units; the address resolution module is configured to match the computing power units in the multicast routing table according to the multicast tag to obtain the target computing power units.

[0015] Further, the address resolution module is further configured to receive heartbeat signals of each computing power unit and update the multicast routing table in real time based on the heartbeat signals of each computing power unit.

[0016] Further, the computing power units connected to the input-output chip include at least two heterogeneous computing power units.

[0017] An input-output method supporting multicast transmission, using the aforementioned input-output system supporting multicast transmission, comprising:

[0018] Step A1, parsing the received to-be-processed data to obtain the multicast label of the to-be-processed data;

[0019] Step A2, matching the target computing power unit according to the multicast label;

[0020] Step A3, judging whether to-be-processed data replication is needed according to the matched target computing power unit:

[0021] If yes, step A4 is executed;

[0022] If no, step A5 is executed;

[0023] Step A4, replicating the to-be-processed data according to the number of matched target computing power units, and then entering step A5;

[0024] Step A5, sending the to-be-processed data to the target computing power unit through the corresponding computing power exchange interface.

[0025] Further, in step A5, dynamic bandwidth allocation is performed on the target computing power unit to be sent, and the to-be-processed data is sent to the target computing power unit through the corresponding computing power exchange interface according to the allocated bandwidth.

[0026] Further, a multicast routing table formed based on a hash algorithm is pre-stored, and the multicast routing table maintains multiple groups of computing power units;

[0027] In step A2, the computing power units in the multicast routing table are matched according to the multicast label to obtain the target computing power unit.

[0028] Further, it further includes step B1, receiving the heartbeat signals of each computing power unit, and dynamically updating the multicast routing table based on the heartbeat signals of each computing power unit.

[0029] The beneficial technical effect of the present application is that by introducing a hardware-level multicast routing engine in the input-output chip, one-time replication and simultaneous transmission of data are realized, the data transmission efficiency is improved, the delay is reduced, efficient computing power parallel operation is supported, and the balance between data transfer efficiency and computing power parallelism is realized. BRIEF DESCRIPTION OF DRAWINGS

[0030] Figure 1 It is a module schematic diagram of the input-output system supporting multicast transmission of the present application;

[0031] Figure 2 It is a step flowchart of the input-output method supporting multicast transmission of the present application;

[0032] Figure 3A scenario diagram for sending data to a computing power unit based on PCIe according to the application;

[0033] Figure 4 A scenario diagram for receiving data from a computing power unit based on PCIe according to the application;

[0034] Figure 5 A scenario diagram for sending data to a computing power unit based on Ethernet and RDMA according to the application;

[0035] Figure 6 A scenario diagram for receiving data from a computing power unit based on Ethernet and RDMA according to the application. DETAILED DESCRIPTION

[0036] The technical solutions in the embodiments of the application will be clearly and completely described below with reference to the drawings in the embodiments of the application. Obviously, the described embodiments are only part of the embodiments of the application, rather than all the embodiments of the application. Based on the embodiments in the application, all other embodiments obtained by a person of ordinary skill in the art without creative work fall within the protection scope of the application.

[0037] It should be noted that the embodiments in the application and the features in the embodiments can be combined with each other without conflict.

[0038] The application will be further described below with reference to the drawings and specific embodiments, but the application is not limited by the embodiments.

[0039] Referring to Figure 1 The application provides an input and output system supporting multicast transmission, comprising an input and output chip, the input and output chip performing input and output management on data of each computing power unit, a multicast routing engine is added to the input and output chip, the multicast routing engine is inserted between an interconnection bus and a computing power exchange interface, each computing power exchange interface is connected to one computing power unit, and the multicast routing engine comprises:

[0040] An address resolution module (1) is configured to resolve received to-be-processed data, obtain a multicast tag of the to-be-processed data, and match a target computing power unit according to the multicast tag.

[0041] A data duplication module (2) is connected to the address resolution module (1) and is configured to, when the number of matched target computing power units is greater than 1, duplicate the to-be-processed data according to the number of matched target computing power units, and send the to-be-processed data to the target computing power units through corresponding computing power exchange interfaces.

[0042] Specifically, the multicast routing engine is a DDU, namely a Distribution&Duplication Unit.

[0043] By introducing a hardware-level multicast routing engine in the traditional IO DIE data path, the contradiction between data transfer and computing power in a heterogeneous system is broken through, and single-source multi-destination efficient transmission is realized, that is, the same data to be processed is distributed to multiple target computing power DIEs at one time, which greatly reduces the bus bandwidth occupancy rate and improves the operator parallelism, supports different operator operations (such as convolution + pooling parallel) on the same data to be processed by multiple computing power DIEs, and the test shows that the computing power utilization rate is improved.

[0044] Specifically, the multicast tag of the data to be processed is set in the packet header part of the data to be processed.

[0045] Specifically, the data replication module (2) replicates the single-input data to be processed into N-way output based on the Crossbar switching structure.

[0046] Specifically, the data to be processed is image data.

[0047] Further, the multicast routing engine further comprises:

[0048] The traffic arbitrator (3) is connected to the data replication module (2) and is used for dynamically allocating bandwidth to the target computing power unit to be sent.

[0049] The data replication module (2) sends the data to be processed to the target computing power unit through the corresponding computing power switching interface according to the allocated bandwidth.

[0050] The traffic arbitrator (3) performs dynamic bandwidth allocation based on the Round-Robin algorithm to avoid congestion.

[0051] In the unicast mode, transmitting 3 copies of the same data requires 3 times the bus bandwidth, resulting in an actual utilization rate of only 35% of the high-speed interface such as PCIe / Chiplet. The data shunting arbitration mechanism in the multicast routing engine can reduce the bandwidth occupancy of multicast transmission to the same as unicast, improving the bandwidth utilization rate. Taking the gradient update scenario in AI training as an example, when distributing data to 8 computing power DIEs, 700GB of bandwidth consumption can be saved, significantly alleviating the "bandwidth wall" problem of the on-chip bus.

[0052] As a specific application scenario of this invention, for example, the input data to be processed is 512×512 pixel image data, which needs to simultaneously perform: Die1: 3×3 convolution (extracting edge features); Die2: max pooling (downsampling); Die3: normalization (standardizing pixel values). A multicast group {Die1, Die2, Die3} is configured. The IO DIE copies the image data to be processed twice and then distributes it to the three computing power DIEs at once. Each computing power DIE executes the operators in parallel, and the results are summarized through the IO DIE after the computation is completed. The total processing time is reduced from 300ns in the unicast scheme to 120ns, an improvement of 60%; the bandwidth is reduced from 3×512MB to 512MB, saving 66%.

[0053] Furthermore, the input / output chips pre-store a multicast routing table (MRT) based on a hash algorithm, which maintains multiple sets of computing power units;

[0054] The address resolution module (1) is used to match the computing units in the multicast routing table according to the multicast label to obtain the target computing unit.

[0055] Currently, this invention supports simultaneous data transmission from up to 8 computing power DIEs. Therefore, the data replication module (2) is based on the Crossbar exchange structure to replicate the single-channel input data to be processed into N-channel outputs, where N≤8.

[0056] In traditional unicast mode, transmitting the same data to be processed to three target computing power DIEs requires three independent transmissions, resulting in a total latency of up to 300ns (including bus arbitration and data transfer). This invention, through the data replication module and multicast routing table of the multicast routing engine, distributes single-source data to up to eight target computing power units at once, achieving an end-to-end latency of only 50ns, a reduction of 83% compared to traditional solutions. This performance improvement stems from zero-copy register slicing technology and a synchronous handshake protocol, ensuring that data replication and routing resolution are completed within a single clock cycle, meeting the latency-sensitive requirements of scenarios such as autonomous driving and real-time inference.

[0057] Furthermore, when the number of computing power DIEs is expanded to 8, the number of routing table entries in the traditional unicast mode is O(N). 2 The original method had 64 routing entries, but this invention uses multicast group abstraction (M = number of groups), so the number of routing table entries is only O(M+N) = M+8 (24 routing table entries when supporting 16 multicast groups), reducing complexity by more than 80%.

[0058] By introducing a hardware-level multicast routing engine into the traditional IO DIE data path, the IO DIE is upgraded from a simple data relay node to a computing hub with intelligent distribution capabilities, providing underlying architectural support for improving the computing efficiency of heterogeneous AI chips.

[0059] Furthermore, the address resolution module (1) is also used to receive the heartbeat signals of each computing unit and update the multicast routing table in real time based on the heartbeat signals of each computing unit.

[0060] Furthermore, the address resolution module can dynamically configure the multicast routing table according to the AI ​​computing tasks. The hardware-level integration of the multicast routing engine and the dynamic updating of the multicast routing table enable the flexibility of dynamic routing, allowing multicast groups to be configured in real time according to the needs of AI tasks and adapting to different data distribution strategies during the training / inference phases.

[0061] The multicast routing table is updated in real time based on heartbeat signals to achieve load balancing. For example, in distributed training, gradient data is distributed to 8 slave computing units via IO DIE. The multicast group needs to be dynamically adjusted according to the load of the slave computing units. The slave computing units report their load status via heartbeat signals (e.g., utilization > 80% is marked as busy). The DDU updates the multicast routing table in real time, diverting data to idle nodes (e.g., the original group {Die1-Die8} is adjusted to {Die1-Die4, Die6-Die8}, skipping the busy Die5).

[0062] like Figure 3 As shown, data is sent to computing power DIE (i.e., computing unit) 6 via the PCIe high-speed interface. The multicast routing engine (DDU) can dynamically determine the distribution to several (including all) computing power DIEs, as indicated by the arrows. In the transmission link, DDU slave 8 mainly aggregates the data from PCIe, while DDU master 7 contains an address resolution module, a data replication module, and a traffic arbitrator, performing main tasks such as data parsing, data replication, data distribution, and bandwidth allocation.

[0063] like Figure 4 As shown, data is received from computing power DIE 6 (i.e. computing power unit), the calculation results of computing power DIE are aggregated by the multicast routing engine (DDU), and then transmitted out through the PCIe high-speed interface, as indicated by the arrow flow direction.

[0064] like Figure 5As shown, data is sent to computing power DIE (i.e., computing power unit) 6 via Ethernet and RDMA (Remote Direct Memory Access). The multicast routing engine (DDU) 12 can dynamically determine and distribute the data to several (including all) computing power DIEs 6. As indicated by the arrow, the data is serially transmitted through MAC layer 9, ETH Controller (Ethernet Controller) 10, and RDMA Subsystem (including RDMA 11 and DDU 12). Finally, it is sent to computing power DIE 6 through the computing power exchange interface of bus matrix 5.

[0065] like Figure 6 As shown, data is received from the computing power DIE 6, and after the calculation results of the computing power DIE are aggregated by the multicast routing engine (DDU) 12, it is transmitted out via Ethernet and RDMA, as indicated by the arrow flow direction.

[0066] Furthermore, at least two of the computing units connected to the input / output chips are heterogeneous units.

[0067] See Figure 2 This invention provides an input / output method supporting multicast transmission, characterized by using a aforementioned input / output system supporting multicast transmission, wherein the input / output chip manages the input / output of data from each computing unit, a multicast routing engine is incorporated into the input / output chip, the multicast routing engine is inserted between the interconnect bus and the computing power exchange interface, and each computing power exchange interface is connected to a corresponding computing unit. The method includes:

[0068] Step A1: Parse the received data to be processed and obtain the multicast tag of the data to be processed;

[0069] Step A2: Match the target computing unit based on the multicast tag;

[0070] Step A3: Determine whether data copying is required based on the matched target computing unit.

[0071] If so, proceed to step A4;

[0072] If not, proceed to step A5;

[0073] Step A4: Copy the data to be processed according to the number of matched target computing units, and then proceed to step A5;

[0074] Step A5: Send the data to be processed to the target computing unit through the corresponding computing power exchange interface.

[0075] By introducing a hardware-level multicast routing engine into the traditional IO DIE data path, the contradiction between data transport and computing power parallelism in heterogeneous systems is overcome, achieving efficient single-source multi-destination transmission. That is, the same data to be processed is distributed to multiple target computing power DIEs at once, which greatly reduces the bus bandwidth utilization, improves the operator parallelism, and supports multiple computing power DIEs to perform different operator operations (such as convolution + pooling parallelism) on the same data to be processed. The computing power utilization has been tested.

[0076] Specifically, the multicast tag for the data to be processed is set in the header of the data packet.

[0077] Specifically, in step A4, based on the Crossbar exchange structure, the data to be processed from a single input is copied into N outputs.

[0078] Specifically, the data to be processed is image data.

[0079] Furthermore, in step A5, dynamic bandwidth allocation is performed on the target computing unit to be sent, and the data to be processed is sent to the target computing unit through the corresponding computing power exchange interface according to the allocated bandwidth.

[0080] The traffic arbitrator (3) uses the Round-Robin algorithm to dynamically allocate bandwidth and avoid congestion.

[0081] This solution utilizes the data routing arbitration mechanism within the multicast routing engine to reduce the bandwidth usage of multicast transmission to the same level as unicast, thereby improving bandwidth utilization.

[0082] Furthermore, a multicast routing table based on a hash algorithm is pre-stored, and the multicast routing table maintains multiple sets of computing power units;

[0083] In step A2, the target computing unit is obtained by matching the computing units in the multicast routing table according to the multicast label.

[0084] In traditional unicast mode, transmitting the same data to be processed to three target computing power DIEs requires three independent transmissions, resulting in a total latency of up to 300ns (including bus arbitration and data transfer). This invention, through the data replication module and multicast routing table of the multicast routing engine, distributes single-source data to up to eight target computing power units at once, achieving an end-to-end latency of only 50ns, a reduction of 83% compared to traditional solutions. This performance improvement stems from zero-copy register slicing technology and a synchronous handshake protocol, ensuring that data replication and routing resolution are completed within a single clock cycle, meeting the latency-sensitive requirements of scenarios such as autonomous driving and real-time inference.

[0085] Furthermore, it also includes step B1, receiving the heartbeat signal of each computing unit and dynamically updating the multicast routing table based on the heartbeat signal of each computing unit.

[0086] The multicast routing table is updated in real time based on heartbeat signals to achieve load balancing.

[0087] The above are merely preferred embodiments of the present invention and are not intended to limit the implementation methods and protection scope of the present invention. Those skilled in the art should recognize that any equivalent substitutions and obvious changes made based on the description and illustrations of the present invention should be included within the protection scope of the present invention.

Claims

1. An input / output system supporting multicast transmission, comprising an input / output chip, wherein the input / output chip manages the input and output of data from each computing unit, characterized in that, A multicast routing engine is incorporated into the input / output chip. This multicast routing engine is inserted between the interconnect bus and the computing power exchange interface. Each computing power exchange interface is connected to one computing power unit. The multicast routing engine includes: The address resolution module is used to parse the received data to be processed, obtain the multicast tag of the data to be processed, and match the target computing unit according to the multicast tag; The data replication module, connected to the address resolution module, is used to replicate the data to be processed according to the number of the target computing power units when the number of matched target computing power units is greater than 1, and to send the data to be processed to the target computing power units through the corresponding computing power exchange interface, so as to realize the one-time distribution of the same data to be processed to multiple target computing power units.

2. The input / output system supporting multicast transmission as described in claim 1, characterized in that, The multicast routing engine also includes: A traffic arbitrator, connected to the data replication module, is used to dynamically allocate bandwidth to the target computing unit to be sent. The data replication module sends the data to be processed to the target computing unit through the corresponding computing power exchange interface according to the allocated bandwidth.

3. The input / output system supporting multicast transmission as described in claim 1, characterized in that, The input / output chip pre-stores a multicast routing table based on a hash algorithm, and the multicast routing table maintains multiple sets of computing power units. The address resolution module is used to match the computing power units in the multicast routing table according to the multicast label to obtain the target computing power unit.

4. The input / output system supporting multicast transmission as described in claim 3, characterized in that, The address resolution module is also used to receive the heartbeat signal of each computing unit and update the multicast routing table in real time based on the heartbeat signal of each computing unit.

5. An input / output system supporting multicast transmission as described in claim 1, characterized in that, At least two of the computing units connected to the input / output chip are heterogeneous units.

6. An input / output method supporting multicast transmission, characterized in that, Using an input / output system supporting multicast transmission as described in any one of claims 1-5, comprising: Step A1: Parse the received data to be processed to obtain the multicast tag of the data to be processed; Step A2: Match the target computing unit according to the multicast tag; Step A3: Determine whether data copying is required based on the matched target computing unit. If so, proceed to step A4; If not, proceed to step A5; Step A4: Copy the data to be processed according to the number of the matched target computing units, and then proceed to step A5; Step A5: Send the data to be processed to the target computing power unit through the corresponding computing power exchange interface, so as to distribute the same data to multiple target computing power units at one time.

7. The input / output method supporting multicast transmission as described in claim 6, characterized in that, In step A5, dynamic bandwidth is allocated to the target computing power unit to be sent, and the data to be processed is sent to the target computing power unit through the corresponding computing power exchange interface according to the allocated bandwidth.

8. The input / output method supporting multicast transmission as described in claim 6, characterized in that, A multicast routing table based on a hash algorithm is pre-stored, and the multicast routing table maintains multiple sets of computing power units; In step A2, the target computing unit is obtained by matching the computing units in the multicast routing table according to the multicast label.

9. The input / output method supporting multicast transmission as described in claim 8, characterized in that, It also includes step B1, receiving the heartbeat signal of each computing unit, and dynamically updating the multicast routing table based on the heartbeat signal of each computing unit.

Images