Cross-domain heterogeneous communication library system based on conversion plug-in type and working method of cross-domain heterogeneous communication library system
By designing a conversion plug-in cross-domain heterogeneous communication library system, the problem of slow communication between chips of different brands and models is solved, and fast and accurate cross-domain heterogeneous communication is achieved to meet the needs of large-scale model training and inference.
Patent Information
- Application Number
- CN202510670391.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-05-23
- Publication Date
- 2025-09-16
AI Technical Summary
Existing technologies cannot efficiently achieve cross-domain heterogeneous communication between chips of different brands and models, resulting in slow communication speeds and difficulty meeting the needs of large-scale model training and inference.
A cross-domain heterogeneous communication library system based on conversion plug-ins is designed, including user access layer, topology management layer, communication operator layer and underlying interface layer. GPU related interfaces are directly called through conversion plug-ins to achieve cross-domain and heterogeneous communication.
It enables fast and accurate communication between chips of different brands and models, meets the training and inference needs across different chip brands and models, and reduces CPU-related memory copies and delays.
Smart Images
Figure CN120653595A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of cross-domain heterogeneous communication, and in particular to a cross-domain heterogeneous communication library system based on a conversion plug-in and a working method thereof. Background Art
[0002] With the rapid development of computing chips, various brands and models of chips have emerged. At the same time, larger-scale model training deployment and computing power usage scenarios have emerged, and the demand for training and inference across different chip brands and models has gradually emerged. For example:
[0003] Large model training scenarios: In actual applications, chips of the same brand and model are insufficient, or a computing cluster cannot accommodate a large number of chips. In this case, cross-cluster (cross-domain) and cross-chip type (heterogeneous) joint training is required.
[0004] Large model inference scenarios: The two stages of inference—prefill and decode—have different hardware priorities and therefore require different types of chips.
[0005] Cloud platform computing power management scenario: For platforms that provide computing power, if they can support different types and models of chips for computing activities, it will help save costs.
[0006] The above application scenarios all require that data between chips of different brands and models can be sent and received. Traditional communication libraries often support communication between chips of the same brand, but do not support heterogeneous and cross-domain communication. Existing research uses the CPU transfer method (such as Figure 1 As shown, different communication libraries exchange data through the CPU, or employ network interfaces similar to CPU operations to achieve cross-domain heterogeneous interconnection. However, this approach inevitably increases CPU-related memory copies or requires CPU operations without CPU copies, resulting in slower communication speeds and difficulty in quickly achieving communication between chips of different brands and models. Summary of the Invention
[0007] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and provide a cross-domain heterogeneous communication library system based on a conversion plug-in and its working method, which can quickly and accurately realize communication between cross-domain heterogeneous chips.
[0008] The object of the present invention can be achieved by the following technical solution: a cross-domain heterogeneous communication library system based on a conversion plug-in, comprising a user access layer, a topology management layer, a communication operator layer and an underlying interface layer, wherein the user access layer is used to access different chips;
[0009] The topology management layer is used to establish a communication topology map, manage communication equipment, and perform communication scheduling;
[0010] The communication operator layer is used to provide different communication operators;
[0011] The bottom interface layer includes a network interface and a chip adapter interface.
[0012] Furthermore, the user access layer is installed with a running platform for implementing corresponding setup and compilation, implementing Pytorch independent backend, implementing user reference examples and corresponding operator performance testing.
[0013] Furthermore, the topology management layer includes a topology module and a management module, and the topology module is used to realize node discovery, topology management and communication scheduling, build a topology map, query heterogeneous types and maintain node heterogeneous types;
[0014] The management module is used to manage relevant selections according to heterogeneous types, handle errors and exceptions, manage multi-threading, multi-process, and parallel acceleration, and perform compression and communication calculation overlap optimization operations.
[0015] Furthermore, the communication operator layer includes a plurality of different communication operators, and the communication operator layer is provided with an auxiliary module for recording the communication process and error information, and providing a debugging tool.
[0016] Furthermore, the different communication operators include: Send, Recv, Broadcast, Scatter, Gather, All-Gather, Reduce, All-Reduce, All-to-All, and ReduceScatter.
[0017] Furthermore, the network interface includes: an IB verb interface, a RoCE interface, a Socket interface, a driver interface, and a network card interface.
[0018] Furthermore, the chip adapter interface includes: a memory management interface, a hardware computing management interface, a RoCE / IB and chip handover interface, and a communication interface of the chip itself.
[0019] A working method of a cross-domain heterogeneous communication library system based on a conversion plug-in, comprising an initialization phase, a phase for running a collective communication operator, and a destruction phase. The initialization phase is used to perform a preliminary handshake connection, construct a heterogeneous topology graph, set a network interface, and select a transmission algorithm and transmission path.
[0020] The said running collective communication operator stage calls the chip related hardware interface through the chip adapter interface, and implements the collective communication operation in combination with the heterogeneous topology graph;
[0021] The destruction phase is used to release related resources.
[0022] Furthermore, the specific process of the initialization phase is as follows: setting the master addr (master node address) and master port (master node port), each node initially finds the master node master through the master addr and master port, and performs a preliminary handshake connection;
[0023] Collect information between nodes, construct a topology map, and set network cards and IB (InfiniBand) underlying network interfaces with default or configured values;
[0024] Select the transmission algorithm and transmission path based on the topology information of the communication library;
[0025] Perform initialization for the selected transmission algorithm.
[0026] Furthermore, the specific process of running the collective communication segment stage is as follows:
[0027] Perform initial buffer processing for the selected transmission algorithm;
[0028] Call the kernel for calculation and call the corresponding IB / Roce (RDMA over ConvergedEthernet, lossless transmission based on Ethernet) transmission interface.
[0029] Compared with the prior art, the present invention has the following advantages:
[0030] The present invention designs a user access layer, a topology management layer, a communication operator layer, and a bottom-level interface layer. The user access layer is used to access different chips; the topology management layer is used to establish a communication topology, manage communication equipment, and perform communication scheduling; the communication operator layer is used to provide different communication operators; and the bottom-level interface layer includes a network interface and a chip adapter interface. The present invention achieves communication between different chips through a conversion plug-in (Adapter) format. It can directly call GPU-related interfaces such as memset and memcpy, and has corresponding format compatibility and operation processing. This enables cross-domain and heterogeneous communication, while ensuring communication speed.
[0031] In the present invention, the operation process of the cross-domain heterogeneous communication library system includes an initialization phase, a collective communication operator operation phase, and a deconstruction phase. In the initialization phase, the user access layer, topology management layer, underlying interface layer, and communication operator layer are respectively called to realize the preliminary handshake connection, construct the heterogeneous topology map, set the network interface, and select the transmission algorithm and transmission path; the collective communication operator operation phase calls the communication operator layer to call the chip-related hardware interface through the chip adapter interface and realize the collective communication operation in combination with the heterogeneous topology map; the deconstruction phase calls the user access layer, topology management layer, underlying interface layer, and communication operator layer to release related resources. This enables data transmission between chips of different brands and models, and can meet the current training and reasoning needs across different chip brands and models. BRIEF DESCRIPTION OF THE DRAWINGS
[0032] Figure 1 This is a schematic diagram of the traditional cross-domain heterogeneous CPU transit communication principle;
[0033] Figure 2 This is a schematic diagram of the communication library system architecture of the present invention;
[0034] Figure 3 Schematic diagram of the cross-domain heterogeneous communication principle of the present invention. DETAILED DESCRIPTION
[0035] The present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.
[0036] Example
[0037] A cross-domain heterogeneous communication library system based on conversion plug-ins, such as Figure 2 As shown in the figure, it includes the user access layer, topology management layer, communication operator layer, and underlying interface layer. The user access layer is used to access different chips. The user access layer is installed with a running platform for implementing corresponding setup and compilation, implementing Pytorch independent backend, implementing user reference examples and corresponding operator performance testing;
[0038] The topology management layer is used to establish a communication topology, manage communication devices, and perform communication scheduling. The topology management layer includes a topology module and a management module. The topology module is used to implement node discovery, topology management, and communication scheduling, build a topology, query heterogeneous types, and maintain heterogeneous node types.
[0039] The management module is used to manage relevant selections based on heterogeneous types, handle errors and exceptions, manage multi-threading, multi-process, and parallel acceleration, and perform compression and communication and computing overlap optimization operations;
[0040] The communication operator layer is used to provide different communication operators, including multiple different communication operators (Send, Recv, Broadcast, Scatter, Gather, All-Gather, Reduce, All-Reduce, All-to-All, ReduceScatter). The communication operator layer is equipped with auxiliary modules for recording communication processes and error information, and providing debugging tools.
[0041] The bottom interface layer includes the network interface and the chip adapter interface. The network interface includes: IB verb interface, RoCE interface, socket interface, driver interface, and network card interface.
[0042] The chip adapter interface includes: memory management interface, hardware computing management interface, RoCE / IB and chip handover interface, and the chip's own communication interface.
[0043] The working method of the above-mentioned cross-domain heterogeneous communication library system based on the conversion plug-in includes an initialization phase, a phase of running a collective communication operator, and a destruction phase. The initialization phase is used to perform a preliminary handshake connection, construct a heterogeneous topology map, set the network interface, and select a transmission algorithm and transmission path by calling the user access layer, the topology management layer, the underlying interface layer, and the communication operator layer;
[0044] The communication operator layer is called during the collective communication operator phase to call the chip-related hardware interface through the chip adapter interface and implement collective communication operations in combination with the heterogeneous topology graph.
[0045] The destruction phase releases related resources by calling the user access layer, topology management layer, underlying interface layer, and communication operator layer.
[0046] This embodiment applies the above solution and first builds four software layers, including:
[0047] 1. Plugin layer (i.e. user access layer): the software layer that installs, runs, and interacts with PyTorch and Python pip;
[0048] 2. Topology management layer: This layer manages the communication network for the AI training plan, including establishing the communication topology, managing communication devices, and selecting computing kernels.
[0049] 3. Communication operator layer: The operator software layer that implements various communication operators and related auxiliary tools;
[0050] 4. Bottom interface layer: including network interface and chip interface.
[0051] Specifically:
[0052] 1. Pytorch Plugin (Pytorch platform plug-in)
[0053] Used to implement corresponding setup and compilation, so that users can install it with pip, or run python setup.py to compile and run;
[0054] Used to implement Pytorch independent backend and create Pytorch custom Backend;
[0055] Used to implement user reference examples and corresponding operator performance tests.
[0056] 2. Topology Management Layer
[0057] 2.1 Topology Module
[0058] Used to implement node discovery, topology management, and communication scheduling;
[0059] Construct topological graphs using common graph structures;
[0060] For heterogeneous chips, the node heterogeneity type is maintained so that subsequent operators can determine the heterogeneous machine types. For example, whether the send and receive primitives "send recv" are isomorphic.
[0061] Use the query function to query heterogeneous types when running the communication operator later.
[0062] 2.2 Management Module
[0063] For the computational operations that occur in the communication operator, manage the relevant selections based on the heterogeneous types;
[0064] Carry out corresponding error and exception handling for possible network or node failures;
[0065] Multi-threading, multi-process, and parallel acceleration management;
[0066] Optimization functions such as compression, communication and calculation overlap, etc.
[0067] 3. Communication Operator Layer
[0068] 3.1. Communication operators include:
[0069] 1.Send
[0070] 2.Recv
[0071] 3.Broadcast
[0072] 4. Scatter
[0073] 5. Gather
[0074] 6.All-Gather
[0075] 7.Reduce
[0076] 8.All-Reduce
[0077] 9. All-to-All
[0078] 10.ReduceScatter
[0079] 11. Expand to the required collective communication operator.
[0080] 3.2 Auxiliary modules
[0081] Log: records communication processes and error information, and monitors communication delay, throughput and other indicators when necessary;
[0082] Debugging tools: tools for developers to locate problems.
[0083] 4. Bottom-level interface
[0084] 4.1 Network Interface
[0085] IB verb interface;
[0086] RoCE interface;
[0087] Socket interface;
[0088] Other networks, drivers, and network card interfaces.
[0089] 4.2 Chip Adapter Interface:
[0090] Video memory management memset, memcpy, malloc, free, etc.;
[0091] The hardware's own computing interface, such as reduce, etc.
[0092] RoCE / IB and chip handover interface;
[0093] The communication interface of the chip itself.
[0094] It should be noted that this solution is designed to handle adapters of different chip types at the bottom layer, such as Figure 3 As shown in the figure, the adapter's approach is to directly call GPU-related interfaces such as memset and memcpy, and has corresponding format compatibility and runtime processing, thereby achieving the purpose of cross-domain and heterogeneous communication. In addition, new chip hardware types can be easily integrated into the existing communication library system using the newly added adapter.
[0095] The operation process of the above communication library is divided into the following steps:
[0096] S1, initialization stage dist.initialize:
[0097] Set the master addr (master node address) and master port (master port). Each node will initially find the master through the masteraddr and port and perform a preliminary handshake connection.
[0098] Collect information between nodes, construct a topology map, and set the underlying network interfaces such as network cards and IB (InfiniBand) with default or configured values;
[0099] Select the transmission algorithm (such as ring reduce, tree reduce), transmission path, and initialize the relevant transmission algorithms based on the topology information of the communication library;
[0100] S2. Run the specific dist. collection communication operator:
[0101] Perform initial processing of the corresponding buffer;
[0102] Call the kernel to perform calculations and call the corresponding IB / Roce (RDMA over ConvergedEthernet) transmission interface;
[0103] S3, destruction phase: release related resources.
Claims
1. A cross-domain heterogeneous communication library system based on conversion plug-in, characterized in that: It includes a user access layer, a topology management layer, a communication operator layer, and an underlying interface layer. The user access layer is used to access different chips. The topology management layer is used to establish a communication topology map, manage communication equipment, and perform communication scheduling; The communication operator layer is used to provide different communication operators; The bottom interface layer includes a network interface and a chip adapter interface.
2. A cross-domain heterogeneous communication library system based on conversion plug-in according to claim 1, characterized in that: The user access layer is installed with a running platform for implementing corresponding setup and compilation, implementing Pytorch independent backend, implementing user reference examples and corresponding operator performance testing.
3. A cross-domain heterogeneous communication library system based on conversion plug-in according to claim 1, characterized in that: The topology management layer includes a topology module and a management module, wherein the topology module is used to realize node discovery, topology management and communication scheduling, build a topology map, query heterogeneous types and maintain node heterogeneous types; The management module is used to manage relevant selections according to heterogeneous types, handle errors and exceptions, manage multi-threading, multi-process, and parallel acceleration, and perform compression and communication calculation overlap optimization operations.
4. A cross-domain heterogeneous communication library system based on conversion plug-in according to claim 1, characterized in that: The communication operator layer includes a plurality of different communication operators. The communication operator layer is provided with an auxiliary module for recording the communication process and error information, and providing a debugging tool.
5. A cross-domain heterogeneous communication library system based on conversion plug-in according to claim 4, characterized in that: The different communication operators include: Send, Recv, Broadcast, Scatter, Gather, All-Gather, Reduce, All-Reduce, All-to-All, and ReduceScatter.
6. A cross-domain heterogeneous communication library system based on conversion plug-in according to claim 1, characterized in that: The network interface includes: IB verb interface, RoCE interface, Socket interface, driver interface, and network card interface.
7. A cross-domain heterogeneous communication library system based on conversion plug-in according to claim 1, characterized in that: The chip adapter interface includes: a memory management interface, a hardware computing management interface, a RoCE / IB and chip handover interface, and a communication interface of the chip itself.
8. A working method of a cross-domain heterogeneous communication library system based on a conversion plug-in type, applied to a cross-domain heterogeneous communication library system based on a conversion plug-in type as claimed in claim 1, characterized in that: It includes an initialization phase, a phase for running collective communication operators, and a destruction phase. The initialization phase is used to perform a preliminary handshake connection, construct a heterogeneous topology graph, set the network interface, and select a transmission algorithm and transmission path. The said running collective communication operator stage calls the chip related hardware interface through the chip adapter interface, and implements the collective communication operation in combination with the heterogeneous topology graph; The destruction phase is used to release related resources.
9. A working method of a cross-domain heterogeneous communication library system based on a conversion plug-in according to claim 8, characterized in that: The specific process of the initialization phase is as follows: setting the master addr master node address and master port master node port, each node node initially finds the master node master through the master addr and master port, and performs a preliminary handshake connection; Collect information between nodes, construct a topology map, and set the network card and IB underlying network interface with default or configured values; Select the transmission algorithm and transmission path based on the topology information of the communication library; Perform initialization for the selected transmission algorithm.
10. A working method of a cross-domain heterogeneous communication library system based on a conversion plug-in according to claim 9, characterized in that: The specific process of running the collective communication segment stage is as follows: Perform initial buffer processing based on the selected transmission algorithm; Call the kernel for calculation and call the corresponding IB / Roce transmission interface.