Communication optimization method, system and equipment based on tensor parallelization, and storage medium

Abstract

The invention provides a communication optimization method and system based on tensor parallelism, equipment and a storage medium. The method comprises the following steps: segmenting local tensor data into a plurality of tensor data blocks; retaining part of the tensor data blocks to the local, performing quantization operation on the rest of the tensor data blocks to obtain first quantized data, and sending the first quantized data to the target node; receiving second quantized data of the target node, and performing dequantization operation on the second quantized data to obtain a far-end tensor data block; executing local reduction on the local tensor data block and the far-end tensor data block to obtain a local reduction result; and performing quantization operation on the local reduction result to obtain third quantized data, and broadcasting the third quantized data to the distributed cluster. According to the method, quantization communication optimization is realized by adopting a two-step protocol, the overall overhead of quantization operation is relatively low, and the precision loss of the tensor data in a multi-round intermediate transmission and processing process is effectively reduced. And the communication path is shortened, so that the linear speed-up ratio is greatly improved.

Description

Technical Field

[0001] This disclosure relates to the fields of artificial intelligence and distributed computing technology, and in particular to a communication optimization method, system, device, and storage medium based on tensor parallelism. Background Technology

[0002] In the deployment of Large Language Models (LLMs), the sheer number of parameters in mainstream models such as LLaMA-3-70B and GPT-4 far exceeds the memory capacity of a single GPU due to the massive parameter count at a single level. Therefore, Tensor Parallelism (TP) has become an indispensable core technology in LLM deployment. The core principle of TP lies in partitioning the weight matrix of each layer of the LLM (including but not limited to linear layer weight matrices and self-attention layer weight matrices) and distributing these partitioned weight matrices across multiple GPUs. During model inference, each GPU is only responsible for calculating the output component corresponding to its weight partition. GPU nodes must exchange and aggregate their calculated output components through Collective Communication to obtain the complete output tensor for that layer, thus ensuring the correctness of model inference. However, currently, there is no effective way to overcome the communication performance bottleneck in real-time inference scenarios for large models. Summary of the Invention

[0003] The technical problem to be solved by this disclosure is to overcome the above-mentioned defects in the prior art and provide a communication optimization method, system, device, and storage medium based on tensor parallelism.

[0004] This disclosure solves the above-mentioned technical problems through the following technical solution:

[0005] Firstly, a communication optimization method based on tensor parallelism is provided, applicable to any node in a distributed cluster employing tensor parallelism. The communication optimization method includes:

[0006] The local tensor data is divided into multiple tensor data blocks corresponding to the total number of nodes in the tensor parallel group; the rest.

[0007] A portion of the tensor data blocks is retained locally, and quantization is performed on the remaining tensor data blocks to obtain the first quantized data. The first quantized data is then sent to the target node; wherein, the target node is a cooperating node within the same tensor parallel group.

[0008] Receive the second quantized data from the target node and perform a dequantization operation on the second quantized data to obtain a remote tensor data block;

[0009] Perform local reduction on the local tensor data block and the remote tensor data block to obtain the local reduction result;

[0010] A quantization operation is performed on the local reduction result to obtain third quantized data, and the third quantized data is broadcast to the distributed cluster.

[0011] Optionally, quantization is performed on the remaining tensor data blocks, including: performing quantization on the remaining tensor data blocks at the transport layer;

[0012] And / or, perform a dequantization operation on the second quantized data, including: performing a dequantization operation on the second quantized data at the transport layer.

[0013] Optionally, quantization is performed on the remaining tensor data blocks at the transport layer, including:

[0014] Symmetric quantization is performed on the remaining tensor data blocks at the transport layer;

[0015] And / or, perform quantization operations on the remaining tensor data blocks at the hardware transport layer.

[0016] Optionally, a dequantization operation is performed on the second quantized data at the transport layer, including:

[0017] A symmetric dequantization operation is performed on the second quantized data at the transport layer;

[0018] And / or, perform a dequantization operation on the second quantized data at the hardware transport layer.

[0019] Secondly, a communication optimization system based on tensor parallelism is provided, applicable to any node in a distributed cluster employing tensor parallelism, wherein the communication optimization system includes:

[0020] The acquisition module is used to divide the local tensor data into multiple tensor data blocks corresponding to the total number of nodes in the tensor parallel group;

[0021] A quantization module is used to perform quantization operations on the remaining tensor data blocks to obtain first quantized data, and send the first quantized data to the target node; wherein, the target node is a collaborative node within the same tensor parallel group; the remaining tensor data blocks are the data blocks other than the tensor data blocks retained locally among the multiple tensor data blocks; a dequantization module is used to receive the second quantized data from the target node, and perform dequantization operations on the second quantized data to obtain a remote tensor data block;

[0022] The reduction module is used to perform local reduction on the local tensor data block and the remote tensor data block to obtain the local reduction result;

[0023] The quantization module is further configured to perform a quantization operation on the local reduction result to obtain third quantized data, and broadcast the third quantized data to the distributed cluster.

[0024] Optionally, the quantization module is specifically used to: perform quantization operations on the remaining tensor data blocks at the transport layer;

[0025] And / or, the dequantization module is specifically used to: perform a dequantization operation on the second quantized data at the transport layer.

[0026] Optionally, symmetric quantization is performed on the remaining tensor data blocks at the transport layer;

[0027] And / or, perform quantization operations on the remaining tensor data blocks at the hardware transport layer.

[0028] Optionally, a symmetric dequantization operation is performed on the second quantized data at the transport layer;

[0029] And / or, perform a dequantization operation on the second quantized data at the hardware transport layer.

[0030] Thirdly, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and for running on the processor, wherein the processor, when executing the computer program, implements the tensor-parallel-based communication optimization method described in any one of the first aspects.

[0031] Fourthly, a computer-readable storage medium is provided, on which a computer program is stored, wherein the computer program, when executed by a processor, implements the tensor-parallel-based communication optimization method described in any one of the first aspects.

[0032] Based on common knowledge in the field, the above-mentioned preferred conditions can be combined arbitrarily to obtain various preferred embodiments of this disclosure.

[0033] The positive improvements of this disclosure are as follows: In this embodiment, the two-step reduction method (i.e., performing local reduction first, then executing global broadcast) optimizes quantization communication. This means that each data block (Chunk) only needs to perform two quantization operations throughout the entire processing flow. The overall overhead of quantization operations is reduced by N-1 compared to the original scheme, while effectively reducing the accuracy loss of tensor data during multiple rounds of intermediate transmission and processing. Furthermore, the two-step reduction method significantly shortens the communication path, resulting in a substantial improvement in linear speedup in multi-machine, multi-card scenarios. When processing ultra-large-scale parameter models, the multi-machine communication performance is improved by more than 2 times. The two-step reduction method (i.e., performing local reduction first, then executing global broadcast) optimizes quantization communication, meaning that each data block (Chunk) only needs to perform two quantization operations throughout the entire processing flow. The overall overhead of quantization operations is reduced by N-1 compared to the original scheme, while effectively reducing the accuracy loss of tensor data during multiple rounds of intermediate transmission and processing. Furthermore, the two-step reduction method greatly shortens the communication path, resulting in a significant improvement in the linear speedup ratio in multi-machine and multi-card scenarios. When processing ultra-large-scale parameter models, the multi-machine communication performance is improved by more than 2 times. Attached Figure Description

[0034] Figure 1 A schematic diagram illustrating an application scenario of a ring communication method provided by existing technology;

[0035] Figure 2a A flowchart illustrating a communication optimization method based on tensor parallelism, provided as an exemplary embodiment of this disclosure;

[0036] Figure 2b A schematic diagram illustrating an application scenario of a communication optimization method based on tensor parallelism, provided as an exemplary embodiment of this disclosure;

[0037] Figure 3 A partial architecture diagram of a hardware integration for implementing communication optimization is provided as an exemplary embodiment of this disclosure;

[0038] Figure 4 A flowchart illustrating a quantization operation in a tensor-parallel-based communication optimization process, provided as an exemplary embodiment of this disclosure;

[0039] Figure 5 A flowchart illustrating a dequantization operation in a tensor-parallel communication optimization process, provided as an exemplary embodiment of this disclosure;

[0040] Figure 6a This is a schematic diagram illustrating the execution of an in-machine reduction during a communication optimization process based on tensor parallelism, provided as an exemplary embodiment of this disclosure.

[0041] Figure 6bThis is a schematic diagram illustrating the execution of inter-machine specifications during communication optimization based on tensor parallelism, provided as an exemplary embodiment of this disclosure.

[0042] Figure 6c This is a schematic diagram illustrating the execution of intra-machine synchronization during communication optimization based on tensor parallelism, provided as an exemplary embodiment of this disclosure.

[0043] Figure 7 This is a schematic diagram of the structure of an electronic device provided as an exemplary embodiment of the present disclosure. Detailed Implementation

[0044] The present disclosure is further illustrated below by way of embodiments, but the present disclosure is not limited to the scope of the embodiments described herein.

[0045] The prefixes such as "first" and "second" used in this disclosure are merely for distinguishing different descriptive objects and do not limit the position, order, priority, quantity, or content of the described objects. The use of ordinal numbers and other prefixes used to distinguish descriptive objects in this disclosure does not constitute a limitation on the described objects. The description of the described objects is given in the claims or the context of the embodiments, and should not be construed as an unnecessary limitation. Furthermore, in the description of this embodiment, unless otherwise stated, "multiple" means two or more.

[0046] The following are some terms that will be used in the embodiments of this disclosure:

[0047] 1) Reduce scatter

[0048] This set communication primitive combines the reduce and scatter operations. It performs a global reduce operation (such as summation or maximization) on the data in an input array, and then distributes the results across the nodes.

[0049] 2) Allgather

[0050] AllGather communication is a many-to-many communication primitive that allows each node to send its data to all other nodes and collect data from all other nodes. After the AllGather operation, each node holds the complete dataset from all nodes.

[0051] 3) AllReduce

[0052] AllReduce is a commonly used aggregation communication primitive in distributed computing, playing a crucial role, especially in distributed training in deep learning. This primitive is primarily used to synchronize data across multiple nodes or devices (such as an NPU). Typically, it aggregates data from multiple nodes, performs some reduction operation (such as summation, maximization, or averaging), and then broadcasts the result back to all nodes. In scenarios involving distributed training of large models, this primitive is mainly applied to synchronizing activation values ​​during tensor-parallel forward and backward propagation, and synchronizing gradient data in data-parallel systems. It's easy to see that the AllReduce primitive is equivalent to the superposition of the Reduce scatter and Allgather communication primitives.

[0053] 4) All to All

[0054] The All-to-All operation allows each node to send its data to all other nodes and receive data from all other nodes. This is a many-to-many communication pattern that ensures each node receives data from all other nodes. This set communication primitive consists of two phases: data distribution and data aggregation. In the data distribution phase, on each node, the input data is split into `split_count` chunks according to the specified `split_dim` (split dimension). These chunks are then sent to other nodes in a certain order (e.g., chunk index); for example, the i-th chunk might be sent to the i-th node. In the data aggregation phase, each node receives the data chunks from all other nodes. The received data chunks are then reassembled into a complete Tensor according to `concat_dim` (connection dimension). Tensors on all nodes must have the same shape and format to ensure correct data aggregation.

[0055] 5) Tensor Parallelism

[0056] Tensor parallelism (TP) in distributed training of large models is a technique that divides the model parameters into multiple tensors and assigns each tensor to a different computing unit (such as an NPU) for computation.

[0057] In large-model tensor parallel (TP) inference, the AllReduce operator, as the core communication operator, directly determines the overall efficiency of large language model tensor parallel inference. Currently, industry optimization schemes for the AllReduce operator and related set communication operations are mainly divided into two categories: topology-based set communication optimization schemes and data compression-based set communication optimization schemes, as detailed below:

[0058] I. Optimization Scheme for Collective Communication Based on Topology

[0059] The core idea of ​​this type of optimization scheme is to optimize the communication topology between GPU nodes, reduce communication path redundancy, improve the communication exchange efficiency between multiple GPUs within a node, and thus optimize aggregate communication performance. The mainstream technical solutions are as follows:

[0060] (1) Ring AllReduce: This is one of the most mainstream communication implementation schemes currently used in the industry. See also Figure 1 In the Reduce-Scatter phase, the algorithm connects N computing nodes (or GPUs) to form a logical ring structure. The tensor data to be communicated is divided into N data chunks. During the transfer of each chunk between adjacent nodes, a loop of "online quantization-network transmission-dequantization-summation" must be performed at each hop. After 2(N-1) steps of data flow, a global reduction operation is completed for the data across all nodes, ultimately achieving consistent aggregation of data across all nodes. In the All-Gather phase, the quantization and dequantization processes are repeated for each hop during cross-node synchronization of tensor data. This scheme has significant drawbacks in large model inference scenarios: large model inference typically uses a small batch size, resulting in small and scattered data chunks for each communication; the ring communication algorithm itself has many communication steps, leading to accumulated latency and failing to fully utilize the high bandwidth of modern GPUs, resulting in idle GPU bandwidth resources and severely limiting inference efficiency.

[0061] (2) Hierarchical Topology: For GPU clusters with non-fully connected architectures (such as PCIe cascaded architectures and some NVLink connection architectures), existing technologies attempt to divide multiple GPUs into different communication groups based on the affinity of the physical links between GPUs. Data reduction operations are first completed within each group, and then data exchange is performed between groups, thereby reducing the amount of communication across physical links. However, this approach still has significant limitations: it introduces complex synchronization barriers at the junctions of intra-group and inter-group communication; in high-performance large model inference tasks, frequent synchronization operations will significantly increase the context switching overhead between the CPU and GPU, making it difficult to fully utilize the GPU's computing resources and reach the theoretical peak bandwidth of the GPU hardware.

[0062] (3) Peer-to-Peer Direct Access: This scheme is a communication optimization scheme for existing high-end GPU clusters. It utilizes the NVLink bus feature supported by high-end GPUs to enable direct reading and writing of each other's video memory data between different GPUs (i.e., Peer-to-Peer access), thereby reducing the overhead of data transfer between the CPU and GPU. However, the application scenarios of this scheme are limited: at present, this P2P access mode is mainly used for simple data copying scenarios; in the complex AllReduce calculation process, how to avoid cache consistency conflicts when multiple GPUs write data concurrently, and how to efficiently handle address mapping in large-scale concurrent scenarios, are still the challenges faced by this type of topology optimization scheme.

[0063] II. Optimization Scheme for Collective Communication Based on Data Compression

[0064] Since the physical communication bandwidth of GPU clusters is limited by hardware costs (e.g., there are differences in communication bandwidth between different GPU models), there is limited room for improving communication efficiency simply through topology optimization. Therefore, another research direction in the industry is to use compression technology at the data transmission level to reduce the amount of data in a single communication, thereby reducing communication bandwidth pressure. The mainstream technical solutions are as follows:

[0065] (1) Coarse-grained quantization: The core idea of ​​existing technologies is to convert the FP16 format activation values ​​generated during large model inference into INT8 or FP8 format before data transmission, thereby reducing data transmission volume and bandwidth pressure. However, it suffers from precision loss: such schemes usually use a uniform quantization scaling factor for the entire tensor or the entire channel; however, during large model inference, there are often a large number of outliers in the activation values. Using a coarse-grained quantization method with a uniform scaling factor will lead to a significant increase in the quantization error of outliers, which in turn will cause the model precision to collapse, resulting in problems such as garbled characters and logical errors in the model output, which cannot meet the precision requirements of large model inference.

[0066] (2) Compression process decoupling computation and communication: Currently, the industry's conventional compression-communication implementation logic is "quantization first, then communication, and finally dequantization," that is, quantization and communication operations are independent of each other and executed step by step. This process has obvious latency overhead defects: both quantization and dequantization operations need to call independent GPU kernel functions to execute; this causes the data to be transmitted to have to be converted back and forth between video memory and registers multiple times, generating a large amount of memory access overhead (I / O overhead); in high-frequency inference scenarios of large models with trillions of parameters, the additional kernel function startup time and video memory access latency often offset the communication benefits brought by the reduction of transmission volume due to data compression, resulting in no significant improvement in the overall inference efficiency of the system and failing to achieve the expected effect of communication optimization.

[0067] (3) Static compression strategy: Some simple compression optimization schemes use a fixed compression ratio to perform the same degree of compression processing on the data to be transmitted in all inference scenarios. However, it has the drawback of poor flexibility: the inference process of large language models is divided into the Prefill stage and the Decode stage. The characteristics of the data to be transmitted in the two stages are significantly different (the data volume in the Prefill stage is large and the granularity is coarse, while the data volume in the Decode stage is small and the granularity is fine). The static compression strategy cannot adapt to this dynamic fluctuation of inference load and it is difficult to achieve a dynamic balance between the communication gain brought by data compression and the computational overhead brought by compression / decompression. The optimization effect is limited.

[0068] In summary, current optimization schemes for set communication in tensor parallel inference of large language models generally exhibit a disconnect between "topology optimization" and "data compression optimization," failing to achieve deep integration between the two. This situation makes it difficult for existing technologies to overcome the communication performance bottleneck in real-time inference scenarios of large models and to meet the performance requirements of actual deployments.

[0069] In demanding application scenarios where real-time inference of large models requires low TTFT (Time To First Token) output time, there is an urgent need for an acceleration system that can deeply co-design topology optimization and underlying fine-grained quantification operators to achieve the organic integration of topology optimization and data compression, thereby breaking through existing technical bottlenecks and achieving end-to-end performance improvement for large model tensor parallel inference.

[0070] To address the significant technical overhead of traditional ring communication algorithms, this disclosure provides a communication optimization method based on tensor parallelism for large model inference. This method employs an optimization strategy of pre-quantization before communication and unified reduction after communication. Furthermore, it is suitable for multi-machine clusters with M computing nodes, and each computing node is configured with N GPUs.

[0071] Figure 2a The flowchart illustrates a communication optimization method based on tensor parallelism, provided as an exemplary embodiment of this disclosure. This method is applied to any node in a distributed cluster employing tensor parallelism, such as a GPU. See also... Figure 2a The communication optimization method includes:

[0072] Step 201: Divide the local tensor data into multiple tensor data blocks corresponding to the total number of nodes in the tensor parallel group.

[0073] The following describes the generation process of tensor data blocks: The complete original tensor of the entire cluster is generated by the cluster master node / core computing node during the forward computation or backward propagation phase of the model; then, the cluster unified scheduling layer, based on the preset parallelizable dimension of the tensor and combined with the total number of nodes in the cluster, uniformly and coarsely divides the complete original tensor data into basic fragments of equal size; then, the cluster scheduling layer coordinates the scheduling, combined with the intra-machine-inter-machine coupling topology, and links the intra-machine communication scheduling module to accurately distribute each basic fragment to each GPU in the cluster through the collaborative access path; finally, the local computing unit of each GPU, according to the data block granularity uniformly preset by the cluster, further uniformly divides its own received basic fragment (local tensor data) into standardized tensor data blocks (Chunks).

[0074] See Figure 2b The diagram illustrates an example of a tensor parallel group containing four GPUs: GPU0, GPU1, GPU2, and GPU3. GPU0 divides its local tensor data A into four parts (since there are four GPUs in total): A0, A1, A2, and A3. GPU1 divides its local tensor data B into four parts: B0, B1, B2, and B3. GPU2 divides its local tensor data C into four parts: C0, C1, C2, and C3. GPU3 divides its local tensor data C into four parts: D0, D1, D2, and D3. Data blocks marked with different colors in the diagram represent data grouped together that require local reduction operations (e.g., summation).

[0075] Step 202: Reserve a portion of the tensor data blocks locally, perform quantization operations on the remaining tensor data blocks to obtain the first quantized data, and send the first quantized data to the target node.

[0076] In this context, target nodes are collaborative nodes within the same tensor parallel group. For GPU0, its target nodes include GPU1 through GPU3. For GPU1, its target nodes include GPU0, GPU2, and GPU3. For GPU2, its target nodes include GPU0, GPU1, and GPU3. For GPU3, its target nodes include GPU0 through GPU2.

[0077] Generally, each node retains one copy of the tensor data block locally (the tensor data block retained locally is called the local tensor data block), and the remaining tensor data blocks are sent to the target node through All-to-All communication.

[0078] Step 202 implements pre-quantization and all-to-all exchange. Each node first performs a one-time quantization operation on the tensor data block chunk. Then, using P2P (peer-to-peer memory access) technology, each node directly sends its first quantized data to the target node through a single all-to-all communication.

[0079] After all-to-all communication, each GPU card has different data blocks (such as a part of a matrix), which need to be precisely exchanged. Let's take... Figure 2b Taking the four GPU cards shown as an example, GPU0 sends data block A1 to the corresponding position of GPU1, data block A2 to the corresponding position of GPU2, and data block A3 to the corresponding position of GPU3. Data block A0 is held by GPU0 itself and is its local tensor data block.

[0080] Step 203: Receive the second quantized data from the target node and perform a dequantization operation on the second quantized data to obtain the remote tensor data block.

[0081] The second quantized data is obtained by the target node performing a quantization operation on its local tensor data block.

[0082] Or with Figure 2b Taking the four GPUs shown as an example, GPU0 receives the second quantized data from GPUs 1 through 3 respectively, performs dequantization operations to obtain remote tensor data blocks (B0, C0, and D0), and stores them in their corresponding locations. The location originally storing A1 in GPU0 receives data (B0) from the corresponding location in GPU1, the location originally storing A2 receives data (C0) from the corresponding location in GPU2, and the location originally storing A3 receives data (D0) from the corresponding location in GPU3. Through All-to-All communication, each GPU gathers all the scattered data blocks, but the data blocks are different for each GPU.

[0083] Step 204: Perform local reduction on the local tensor data block and the remote tensor data block to obtain the local reduction result.

[0084] In one embodiment, local reduction is achieved through a summation operation, which is performed in the local memory of each node. See also Figure 2bAfter step 204, GPU0 obtains A0+B0+C0+D0, GPU1 obtains A1+B1+C1+D1, GPU2 obtains A2+B2+C2+D2, and GPU3 obtains A3+B3+C3+D3.

[0085] Step 205: Perform quantization on the local reduction result to obtain the third quantized data, and broadcast the third quantized data to the distributed cluster.

[0086] Step 205 involves synchronizing the third quantized data to all other nodes in the distributed cluster via a broadcast communication. For example, GPU0 copies the model parameters and sends them to GPU1, GPU2, and GPU3, ensuring that each GPU receives the same data. Each node then performs dequantization and finally executes a full reduction and summation operation based on the transmitted data to obtain the global tensor data.

[0087] In this embodiment, a two-step reduction method (i.e., performing local reduction first, then global broadcasting) is used to optimize quantization communication. This means that each data chunk only needs to undergo two quantization operations throughout the entire processing flow. The overall overhead of quantization operations is reduced by N-1 compared to the original scheme, while effectively reducing the accuracy loss of tensor data during multiple rounds of intermediate transmission and processing. Furthermore, the two-step reduction method significantly shortens the communication path, resulting in a substantial improvement in linear speedup in multi-machine, multi-GPU scenarios. When processing ultra-large-scale parameter models, multi-machine communication performance is improved by more than 2 times.

[0088] "Multi-machine" refers to a computing cluster composed of multiple independent physical computing servers (physical nodes of a distributed cluster). High-speed interconnect technology establishes high-bandwidth, low-latency cross-machine communication links, handling operations such as tensor data block interaction and global reduction / aggregation across physical nodes. "Multi-GPU" refers to multiple graphics processing units (GPUs, the core execution units of computing) within each physical server. GPUs communicate with each other via intra-machine P2P direct access, establishing low-overhead communication paths and handling operations such as fine-grained tensor computation, local reduction, and chunk-level quantization / dequantization. The multi-machine, multi-GPU architecture achieves distributed parallel processing of ultra-large-scale parameter models through hierarchical intra-machine and inter-machine communication and computational collaboration.

[0089] In one embodiment, a single machine uses a high-speed interconnect to complete a reduction, while multiple machines use a slower speed to achieve reduction.

[0090] In this embodiment, based on the high efficiency of two-step full protocol using P2P direct access within a single machine, an intra-machine-inter-machine coupled topology is further constructed to effectively integrate the low latency advantage of intra-machine P2P communication with the scalability of inter-machine collaborative transmission, breaking the scenario boundary of single-machine full protocol and realizing deep coupling and collaborative scheduling of intra-machine and inter-machine full protocol processes in a distributed cluster.

[0091] For multi-machine distributed scenarios, this embodiment automatically splits the global AllReduce into a collaborative execution path of fine-grained intra-machine sharding aggregation and high-speed inter-machine interconnection (such as RDMA). It maintains the minimal hop count of the two-step method within the node and reduces cross-machine bandwidth pressure through segmented pipelines between nodes. This not only significantly improves the overall execution efficiency of the full reduction operation in multi-machine distributed scenarios and optimizes the hierarchical utilization efficiency of cluster communication resources, but also realizes the smooth expansion of tensor parallelism from single-machine multi-card to multi-machine cluster.

[0092] It should be noted that although high-speed interconnection is used between machines, the speed of internal interconnection is much higher than that of inter-machine interconnection.

[0093] In one embodiment, the step of performing a quantization operation on the remaining tensor data blocks includes: performing a quantization operation on the remaining tensor data blocks at the transport layer.

[0094] In one embodiment, the step of performing a dequantization operation on the second quantized data includes: performing a dequantization operation on the second quantized data at the transport layer.

[0095] In this embodiment, quantization and dequantization are performed at the transport layer, realizing a data processing mechanism of "decompressing as soon as it is dropped and packaging as soon as it is sent". This completely eliminates the startup overhead of the software kernel and avoids redundant video memory loading / Store latency, significantly improving the hardware execution efficiency of tensor data communication processing.

[0096] In one embodiment, quantization is performed on the remaining tensor data blocks at the hardware transport layer.

[0097] In one embodiment, a dequantization operation is performed on the second quantized data at the hardware transport layer.

[0098] Figure 3 This is a partial architecture diagram of a hardware integration for implementing communication optimization, provided as an exemplary embodiment of the present disclosure.

[0099] In this embodiment, the quantization operation occurs before the source-side C2CSS (chip interconnect subsystem) sends data to the Ethernet interface; the dequantization operation occurs after the destination-side C2CSS receives data from the Ethernet. By integrating the quantization logic into the C2CSS, the compression operation evolves from a "separate software step" to a "streaming hardware behavior," eliminating memory access overhead. Simultaneously, fundamental operator functions such as internal hardware numerical comparisons (e.g., maximum value acquisition), division operations (e.g., scaling factor calculation), and bit-width conversions (e.g., FP16 / BF16 and INT8 / FP8 conversions) are all implemented through a high-performance hardware acceleration library, ensuring computational determinism and low latency performance in the high-speed clock domain.

[0100] Since C2CSS supports the standard AXI4 protocol, the system has designed independent data paths for read and write operations: For write operations (Write Transaction), C2CSS intercepts the AXI4 write data stream and triggers quantization and packet assembly logic in real time before the data leaves the chip and enters the Ethernet physical link; for read operations, C2CSS receives compressed packets from Ethernet feedback and performs unpacking and dequantization logic before the data is delivered to the AXI4 read channel.

[0101] In this embodiment, the quantization and dequantization logic is directly embedded or integrated into the hardware transmission layer (such as a dedicated communication module or network card interface logic). This allows data to be quantized and packaged synchronously in the hardware pipeline when it is sent from the register via the bus. Before reaching the target node's memory, the hardware automatically unpacks the data, eliminating the need to call independent kernel functions. This achieves deep pipelined integration of quantization, dequantization, and cross-node data transmission at the hardware layer, without the need for software layer intervention. This completely eliminates the kernel startup overhead and redundant memory load / store latency at the software level, significantly reduces GPU computing power consumption, and frees up computing unit resources to focus on core tasks such as tensor reduction and model calculation. At the same time, it achieves hardware-level microsecond-level execution of quantization / dequantization, significantly improving the end-to-end processing efficiency of tensor data cross-node communication and fully leveraging the native high-performance transmission characteristics of the hardware transmission layer.

[0102] In one embodiment, the quantization operation employs a symmetric quantization operation.

[0103] In one embodiment, the dequantization operation employs a symmetric dequantization operation.

[0104] Preferably, a numerical compression mechanism of grouped symmetric 8-bit quantization is used to achieve symmetric quantization and symmetric dequantization.

[0105] In this embodiment, a fine-grained symmetric quantization scheme based on a 512-bit hardware width increases the effective data throughput by 2 times while accurately locking out outliers through extremely small quantization groups (the scale factor is aligned with the hardware width), ensuring that the accuracy loss of the trillion-level model during inference is less than 1% (or almost lossless).

[0106] In order to adapt to the unified 512-bit data width of the hardware interface, this embodiment has designed a special numerical processing flow.

[0107] Figure 4 A flowchart of a quantization operation provided for an exemplary embodiment of this disclosure.

[0108] For high-precision quantization and scaling to low-bit levels, 512 bits (corresponding to the amount of data transmitted in one VLD signal cycle) are used as the smallest hardware processing unit. Based on software configuration parameters, the hardware logic performs real-time quantization on BF16 / FP16 format data, converting it to INT8 / FP8 low-bit format, and generating a corresponding quantization scaling factor for each 512-bit data block. Innovatively, the quantization unit is aligned with the 512-bit physical width of the hardware interface. By generating a scaling factor per cycle, an optimal physical balance is found between hardware implementation complexity, data throughput, and numerical fidelity.

[0109] For hardware packet processing, after the original BF16 / FP16 format data is quantized into low-bit data such as INT8, the bit width is greatly reduced, resulting in a bit width idle space in the 512-bit transmission link. After the quantization operation is completed, the hardware module starts the packet processing logic to encapsulate multiple quantized low-bit short vectors into the same 512-bit physical frame, thereby multiplying the actual effective payload ratio of the Ethernet transmission link.

[0110] Figure 5 A flowchart of a dequantization operation is provided for an exemplary embodiment of this disclosure.

[0111] For hardware unpacking, after receiving a 512-bit compressed frame from the Ethernet link, the receiving node first performs unpacking based on the identifier information in the message header, and then parses and restores the INT8 / FP8 low-bit data stream.

[0112] For online dequantization, the hardware module extracts the quantization scaling factor (scale) transmitted with the packet, and relies on the computing unit of the DWFC high-performance hardware library to perform real-time dequantization processing on the low-bit data, restoring it to the original BF16 / FP16 format.

[0113] For aligned output, the dequantized data is restored to a standard 512-bit width, and the data is delivered to the chip's internal logic in accordance with the AXI4 bus protocol.

[0114] In this embodiment, fine-grained symmetric quantization, which is more computationally compatible, is adopted. Compared with asymmetric quantization, symmetric quantization eliminates the computational overhead of zero-point offset, which can make more efficient use of hardware computing power. At the same time, fine-grained grouping can effectively lock out outliers in the activation values, ensuring that the trillion-parameter model still maintains near-lossless inference accuracy after compression.

[0115] In one embodiment, quantization / dequantization is performed at the hardware transport layer.

[0116] In this embodiment, the hardware operation circuit is simplified by using symmetric quantization logic (no zero-point calculation is required), which minimizes the hardware area and power consumption of the quantization unit integrated on the AXI4 channel, achieving an extremely high energy efficiency ratio.

[0117] See Figures 6a-6c The diagram illustrates the communication optimization process in this embodiment by taking a cluster containing four computing device nodes (Node0-Node3), each of which contains four GPUs (GPU0-GPU3) for tensor parallelism.

[0118] IntraNode: See Figure 6a Within each compute node, a Reduce-Scatter operation is performed once (implemented through steps 201-205 above). At this point, each server's N GPU cards each hold a different, pre-reduced data block (Chunk_idx). The transmission medium is PCIe or NVLink high-speed interconnect. Different colored data blocks in the diagram represent the same set of data that requires global reduction (such as summation).

[0119] InterNode Protocol Layer: See Figure 6b GPUs holding the same Chunk_idx across different compute nodes form a communication group and perform a cross-machine AllReduce via an RDMA / InfiniBand link. This step ensures that a specific GPU within each server holds the "global summation result" of that data block.

[0120] IntraNode All Gather: See Figure 6c After cross-machine aggregation is completed, each machine performs another All-Gather operation. In the final state, all GPUs on each server have obtained the complete, globally summed tensor.

[0121] In this embodiment, the high-bandwidth / low-latency intra-machine links and high-performance / cross-node inter-machine links are cleverly separated, avoiding frequent quantization at the cross-machine level and greatly optimizing the inference response time of large-scale clusters.

[0122] In this embodiment, a hierarchical communication architecture combining hardware and software is employed. Unlike traditional single communication algorithms, this invention constructs a hierarchical model of "intra-machine two-step method + inter-machine distributed aggregation." By fixing the number of communication hops to a constant level, the problem of latency accumulation caused by the increase in the number of nodes in distributed inference is completely solved.

[0123] Corresponding to the aforementioned embodiments of the communication optimization method based on tensor parallelism, this disclosure also provides embodiments of the communication optimization system based on tensor parallelism.

[0124] This disclosure provides a communication optimization system based on tensor parallelism, which is used to implement the communication optimization method based on tensor parallelism provided in any of the above embodiments. The system includes:

[0125] The acquisition module is used to divide local tensor data into multiple tensor data blocks corresponding to the total number of nodes in the tensor parallel group;

[0126] The quantization module is used to perform quantization operations on the remaining tensor data blocks to obtain the first quantized data, and send the first quantized data to the target node; wherein, the target node is a cooperating node within the same tensor parallel group; the remaining tensor data blocks are the data blocks other than the tensor data blocks retained locally among the multiple tensor data blocks;

[0127] The dequantization module is used to receive the second quantized data from the target node and perform a dequantization operation on the second quantized data to obtain a remote tensor data block.

[0128] The reduction module is used to perform local reduction on the local tensor data block and the remote tensor data block to obtain the local reduction result;

[0129] The quantization module is further configured to perform a quantization operation on the local reduction result to obtain third quantized data, and broadcast the third quantized data to the distributed cluster.

[0130] Optionally, the quantization module is specifically used to: perform quantization operations on the remaining tensor data blocks at the transport layer;

[0131] And / or, the dequantization module is specifically used to: perform a dequantization operation on the second quantized data at the transport layer.

[0132] Optionally, symmetric quantization is performed on the remaining tensor data blocks at the transport layer;

[0133] And / or, perform quantization operations on the remaining tensor data blocks at the hardware transport layer.

[0134] Optionally, a symmetric dequantization operation is performed on the second quantized data at the transport layer;

[0135] And / or, perform a dequantization operation on the second quantized data at the hardware transport layer.

[0136] For the system embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this disclosure according to actual needs.

[0137] Figure 7 This is a schematic diagram of the structure of an electronic device according to an example embodiment of the present disclosure. The electronic device includes a memory, a processor, and a computer program stored in the memory and used to run on the processor. When the processor executes the computer program, it implements the communication optimization method based on tensor parallelism described in any of the above embodiments. Figure 7 The electronic device 70 shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments disclosed herein.

[0138] like Figure 7 As shown, the electronic device 70 can be manifested as a general-purpose computing device, such as a server device. The components of the electronic device 70 may include, but are not limited to: at least one processor 71, at least one memory 72, and a bus 73 connecting different system components (including memory 72 and processor 71).

[0139] Bus 73 includes a data bus, an address bus, and a control bus.

[0140] The memory 72 may include volatile memory, such as random access memory (RAM) 721 and / or cache memory 722, and may further include read-only memory (ROM) 723.

[0141] The memory 72 may also include a program tool 725 (or utility) having a set (at least one) program module 724, such program module 724 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each or some combination of these examples may include an implementation of a network environment.

[0142] The processor 71 executes various functional applications and data processing by running computer programs stored in the memory 72, such as the tensor-parallel-based communication optimization method provided in any of the above embodiments.

[0143] Electronic device 70 can also communicate with one or more external devices 74 (e.g., keyboard, pointing device, etc.). This communication can be performed via input / output (I / O) interface 75. Furthermore, electronic device 70 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public network, such as the Internet) via network adapter 76. As shown, network adapter 76 communicates with other modules of electronic device 70 via bus 73. It should be understood that, although not shown in the figure, other hardware and / or software modules can be used in conjunction with electronic device 70, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID (disk array) systems, tape drives, and data backup storage systems.

[0144] It should be noted that although several units / modules or sub-units / modules of the electronic device have been mentioned in the detailed description above, this division is merely exemplary and not mandatory. In fact, according to embodiments of this disclosure, the features and functions of two or more units / modules described above can be embodied in one unit / module. Conversely, the features and functions of one unit / module described above can be further divided and embodied by multiple units / modules.

[0145] This disclosure also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the tensor-parallel-based communication optimization method provided in any of the above embodiments.

[0146] The readable storage medium may be more specifically adopted, including but not limited to: portable disk, hard disk, random access memory, read-only memory, erasable programmable read-only memory, optical storage device, magnetic storage device, or any suitable combination thereof.

[0147] This disclosure also provides a computer program product, including a computer program that, when executed by a processor, implements the tensor-parallel-based communication optimization method described above.

[0148] The program code for executing the computer program product of this disclosure can be written in any combination of one or more programming languages, and the program code can be executed entirely on a user device, partially on a user device, as a stand-alone software package, partially on a user device and partially on a remote device, or entirely on a remote device.

[0149] While specific embodiments of this disclosure have been described above, those skilled in the art should understand that these are merely illustrative examples, and the scope of protection of this disclosure is defined by the appended claims. Those skilled in the art can make various changes or modifications to these embodiments without departing from the principles and essence of this disclosure, but all such changes and modifications fall within the scope of protection of this disclosure.

Claims

1. A communication optimization method based on tensor parallelism, characterized in that, The communication optimization method, applicable to any node in a distributed cluster employing tensor parallelism, includes: The local tensor data is divided into multiple tensor data blocks corresponding to the total number of nodes in the tensor parallel group; A portion of the tensor data blocks is retained locally, and quantization is performed on the remaining tensor data blocks to obtain the first quantized data. The first quantized data is then sent to the target node; wherein, the target node is a cooperating node within the same tensor parallel group. Receive the second quantized data from the target node and perform a dequantization operation on the second quantized data to obtain a remote tensor data block; Perform local reduction on the local tensor data block and the remote tensor data block to obtain the local reduction result; A quantization operation is performed on the local reduction result to obtain third quantized data, and the third quantized data is broadcast to the distributed cluster.

2. The communication optimization method based on tensor parallelism according to claim 1, characterized in that, Performing quantization operations on the remaining tensor data blocks includes: performing quantization operations on the remaining tensor data blocks at the transport layer; And / or, perform a dequantization operation on the second quantized data, including: performing a dequantization operation on the second quantized data at the transport layer.

3. The communication optimization method based on tensor parallelism according to claim 2, characterized in that, Performing quantization operations on the remaining tensor data blocks at the transport layer includes: Symmetric quantization is performed on the remaining tensor data blocks at the transport layer; And / or, perform quantization operations on the remaining tensor data blocks at the hardware transport layer.

4. The communication optimization method based on tensor parallelism according to claim 2, characterized in that, Performing a dequantization operation on the second quantized data at the transport layer includes: A symmetric dequantization operation is performed on the second quantized data at the transport layer; And / or, perform a dequantization operation on the second quantized data at the hardware transport layer.

5. A communication optimization system based on tensor parallelism, characterized in that, The communication optimization system, applicable to any node in a distributed cluster employing tensor parallelism, includes: The acquisition module is used to divide the local tensor data into multiple tensor data blocks corresponding to the total number of nodes in the tensor parallel group; The quantization module is used to perform quantization operations on the remaining tensor data blocks to obtain the first quantized data, and send the first quantized data to the target node; wherein, the target node is a cooperating node within the same tensor parallel group; the remaining tensor data blocks are the data blocks other than the tensor data blocks retained locally among the multiple tensor data blocks; The dequantization module is used to receive the second quantized data from the target node and perform a dequantization operation on the second quantized data to obtain a remote tensor data block. The reduction module is used to perform local reduction on the local tensor data block and the remote tensor data block to obtain the local reduction result. The quantization module is further configured to perform a quantization operation on the local reduction result to obtain third quantized data, and broadcast the third quantized data to the distributed cluster.

6. The communication optimization system based on tensor parallelism according to claim 5, characterized in that, The quantization module is specifically used to: perform quantization operations on the remaining tensor data blocks at the transport layer; And / or, the dequantization module is specifically used to: perform a dequantization operation on the second quantized data at the transport layer.

7. The communication optimization system based on tensor parallelism according to claim 6, characterized in that, Symmetric quantization is performed on the remaining tensor data blocks at the transport layer; And / or, perform quantization operations on the remaining tensor data blocks at the hardware transport layer.

8. The communication optimization system based on tensor parallelism according to claim 6, characterized in that, A symmetric dequantization operation is performed on the second quantized data at the transport layer; And / or, perform a dequantization operation on the second quantized data at the hardware transport layer.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and for running on the processor, characterized in that, When the processor executes the computer program, it implements the communication optimization method based on tensor parallelism as described in any one of claims 1 to 4.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the communication optimization method based on tensor parallelism as described in any one of claims 1 to 4.

Images