Online aggregation method, device and system and storage medium

Abstract

The invention discloses an on-network aggregation method, device and system and a storage medium. The method comprises the following steps: S1, executing gradient aggregation logic by adopting a configurable matching table architecture; and S2, in the process of executing the gradient aggregation logic, Hash conflicts, resource overflow and floating point accumulation / sparse coding calculation are processed through an RISC-V calculation module. By adopting the technical scheme of the invention, a customized protocol and line speed processing specific to a data packet can be realized; and the abnormal condition which cannot be processed by the network card with limited network card flexibility can be processed.

Description

Technical Field

[0001] This invention belongs to the field of online aggregation technology, and particularly relates to an online aggregation method, apparatus, system and storage medium. Background Technology

[0002] The rapid development of artificial intelligence technology is driving deep industrial integration, and its core supporting technology—deep learning—has entered the era of large-scale models with hundreds of billions of parameters. For example, the Qwen3 series includes the flagship model Qwen3-235B-A22B with 235B (235 billion parameters), and the Deepseek-R1 model has a total of 671 billion parameters, employing a hybrid expert architecture (MoE), with each token activating 37 billion parameters. The training of such models requires exponentially increasing computing power, and a single server can no longer meet the computational and storage needs, making distributed training an inevitable choice.

[0003] Traditional distributed training employs a parameter server (PS) architecture, where worker nodes send local gradients to a centralized server for aggregation and parameter updates. However, with the improvement of hardware accelerators such as GPUs, the ratio of computation to communication time has become severely unbalanced, with communication latency accounting for more than 60% of the total training time. Especially in large-scale data parallel scenarios, the multi-round gradient synchronization traffic generated by ensemble communication operations such as All-Reduce leads to network congestion, becoming a core bottleneck restricting training efficiency.

[0004] To alleviate communication bottlenecks, academia and industry have proposed on-network aggregation technology, which directly performs gradient aggregation through programmable network devices, reducing the number of end-to-end communication rounds. Early solutions relied on fixed-function ASIC switches, such as gradient synchronization based on RDMA, but their protocols were rigid and lacked flexibility, making it difficult to adapt to dynamically changing aggregation algorithms (such as sparse gradient optimization).

[0005] In recent years, the rise of programmable data plane devices has provided new ideas for on-network aggregation. For example: Intel Tofino switches support P4 programming, enabling customized aggregation logic in the data plane. However, due to the fixed pipeline structure of the switching chip, they struggle to handle large gradient tensors. Furthermore, the Tofino switch chip is not open source, preventing developers from observing its internal logic and operating status. RMT architecture devices: adopt a reconfigurable match-action table pipeline, which allows dynamic reconstruction of match-action rules, supports protocol-independent packet processing, and significantly improves flexibility.

[0006] However, existing on-the-net aggregation solutions still have two major drawbacks: Insufficient computational offloading capacity: The switch lacks a general-purpose computing unit, and complex aggregation operations (such as floating-point accumulation and sparse coding) need to be transmitted back to the host CPU, which offsets the benefits of communication optimization. Low efficiency of end-to-end network collaboration: Although traditional smart network cards (such as Mellanox BlueField) integrate ARM cores, their firmware ecosystem is closed, making it difficult to achieve deep collaboration with the P4 data plane. Summary of the Invention

[0007] The technical problem to be solved by the present invention is to provide an online aggregation method, apparatus, system and storage medium.

[0008] To achieve the above objectives, the present invention adopts the following technical solution: An online aggregation method, comprising: Step S1: Execute gradient aggregation logic using a configurable matching table architecture; Step S2: In the execution of gradient aggregation logic, hash collisions, resource overflows, and floating-point accumulation / sparse encoding calculations are handled by the RISC-V computing module.

[0009] Preferably, the configurable matching table adopts a multi-stage pipeline structure, including: a protocol-independent parsing unit, a keyword extraction unit, an action matching unit, an aggregator, and a protocol-independent inverse parser. When a message enters the protocol-independent parsing unit, the message header is extracted. The message header contains the task ID and packet sequence number ID, and the message data is temporarily stored in the data buffer. In each stage, the message header enters the keyword extraction unit, which extracts the first N bytes and hashes them into an index. The action matching unit uses the extracted first N bytes as the index to find the corresponding aggregator. The aggregator performs gradient aggregation, judging whether to update or discard the table entry based on the metadata extracted from the message header and the information read from the corresponding position in the aggregation table. After all stages have completed the processing of the message header, the protocol-independent inverse parsing unit concatenates the processed message header with the message data stored in the data buffer.

[0010] The present invention also provides an on-grid aggregation apparatus, comprising: The first processing module is used to execute gradient aggregation logic using a configurable matching table architecture; The second processing module is used to handle hash collisions, resource overflows, and floating-point accumulation / sparse coding calculations through the RISC-V computing module during the execution of gradient aggregation logic.

[0011] Preferably, the configurable matching table adopts a multi-stage pipeline structure, including: a protocol-independent parsing unit, a keyword extraction unit, an action matching unit, an aggregator, and a protocol-independent inverse parser. When a message enters the protocol-independent parsing unit, the message header is extracted. The message header contains the task ID and packet sequence number ID, and the message data is temporarily stored in the data buffer. In each stage, the message header enters the keyword extraction unit, which extracts the first N bytes and hashes them into an index. The action matching unit uses the extracted first N bytes as the index to find the corresponding aggregator. The aggregator performs gradient aggregation, judging whether to update or discard the table entry based on the metadata extracted from the message header and the information read from the corresponding position in the aggregation table. After all stages have completed the processing of the message header, the protocol-independent inverse parsing unit concatenates the processed message header with the message data stored in the data buffer.

[0012] This invention also provides an online aggregation system, comprising: a worker end and a PS end, the PS end including: an online aggregation device and a broadcast module; the online aggregation device including: a first processing module for executing gradient aggregation logic using a configurable matching table architecture; and a second processing module for handling hash collisions, resource overflows, and floating-point accumulation / sparse coding calculations through a RISC-V computing module during the execution of the gradient aggregation logic; wherein, The worker sends message data to the on-network aggregation device on the PS side via the RDMA protocol, performs gradient compression and sparsity processing, and the updated parameters are transmitted back to the worker in real time via the broadcast module.

[0013] Preferably, the configurable matching table adopts a multi-stage pipeline structure, including: a protocol-independent parsing unit, a keyword extraction unit, an action matching unit, an aggregator, and a protocol-independent inverse parser. When a message enters the protocol-independent parsing unit, the message header is extracted. The message header contains the task ID and packet sequence number ID, and the message data is temporarily stored in the data buffer. In each stage, the message header enters the keyword extraction unit, which extracts the first N bytes and hashes them into an index. The action matching unit uses the extracted first N bytes as the index to find the corresponding aggregator. The aggregator performs gradient aggregation, judging whether to update or discard the table entry based on the metadata extracted from the message header and the information read from the corresponding position in the aggregation table. After all stages have completed the processing of the message header, the protocol-independent inverse parsing unit concatenates the processed message header with the message data stored in the data buffer.

[0014] The present invention also provides a storage medium storing a computer program that executes an online aggregation method when running.

[0015] This invention employs a configurable matching table architecture, enabling customized protocols and line-rate processing specific to data packets. It also handles exceptional situations that network interface cards (NICs) with limited flexibility cannot handle (such as hash collisions and resource overflows). This invention is well-suited for in-network aggregation scenarios. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0017] Figure 1 This is a flowchart of the network aggregation method according to an embodiment of the present invention; Figure 2 A schematic diagram of the architecture for a configurable matching table; Figure 3 This is a schematic diagram of the network aggregation system architecture according to an embodiment of the present invention. Detailed Implementation

[0018] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0019] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0020] Example 1: like Figure 1 As shown, this embodiment of the invention provides an on-network aggregation method suitable for distributed deep learning training scenarios with hundreds of billions of parameters. It supports multiple parallel modes (data / model / pipeline parallelism), is compatible with network protocols such as RDMA / RoCEv2 / TCP, and is particularly optimized for sparse training and mixed precision scenarios, including: Step S1: The RMT programmable data plane uses a configurable matching table architecture to execute gradient aggregation logic; Step S2: In the execution of gradient aggregation logic, hash collisions, resource overflows, and floating-point accumulation / sparse encoding calculations are handled by the RISC-V computing module.

[0021] As one embodiment of the present invention, such as Figure 2As shown, the configurable matching table adopts a multi-stage pipeline structure, including: a protocol-independent parsing unit, a keyword extraction unit, an action matching unit, an aggregator, and a protocol-independent inverse parser. When a packet enters the protocol-independent parsing unit, the packet header is extracted. The packet header contains the task ID, packet sequence number ID, and metadata fields identifying gradient sparsity and data type. The packet data is temporarily stored in the data buffer. The packet header then enters the multi-stage action matching table stage. In each stage, the packet header enters the keyword extraction unit, extracts the first N bytes, and hashes them into an index. If the traffic is too high at this time, the aggregator of this index may be performing aggregation operations on other packets, resulting in a hash collision; or when the aggregator detects that the current aggregation table entry is short of resources (such as memory overflow or counter saturation), a resource overflow flag is generated. In cases of hash collisions or resource overflows, data packets are rerouted to the RISC-V computing module for processing. The RISC-V module uses software algorithms to resolve collisions and rehash, or dynamically allocates and schedules memory for overflowing aggregation operations. The action matching unit uses the extracted first N bytes as an index to find the corresponding aggregator. The aggregator performs gradient aggregation, judging whether to update or discard entries based on the metadata extracted from the packet header and the information read from the corresponding position in the aggregation table. During aggregation, the gradient vector values ​​are first converted to 32-bit high-precision floating-point numbers, and then converted back to their original precision values ​​after aggregation. It supports mixed precision calculations of FP8 / FP16 / BF16, which can support the processing of floating-point numbers of different precisions and ensure that no precision is lost during aggregation. After all stages have completed the processing of the packet header, the protocol-independent inverse parsing unit concatenates the processed packet header with the packet data stored in the data buffer.

[0022] Specifically, for sparse gradient scenarios, the RMT programmable data plane identifies sparse coding scenarios by parsing sparsity identification fields (such as sparsity thresholds and non-zero element index distributions) in the message header and forwards the sparse gradient data to the RISC-V computing module. The RISC-V computing module has a built-in dedicated instruction set that can identify and decode the non-zero element indices and corresponding values ​​in the sparse gradient matrix. For sparse gradients generated using Top-K or threshold filtering, the RISC-V module efficiently processes non-zero value aggregation through dynamic memory allocation and compressed storage structures (such as CSR / CSC formats), while skipping zero values ​​to reduce computational overhead. In addition, the module supports sparse-dense hybrid operations, automatically triggering a sparse coding optimization process when the gradient tensor sparsity exceeds a set threshold, significantly reducing data transmission volume and computational latency. After processing by the RISC-V module, the aggregation result is written back to the RMT's aggregation table, where the protocol-independent inverse parsing unit completes message reassembly, achieving joint processing.

[0023] Example 2: This invention also provides an on-grid aggregation device, which is a programmable smart network interface card (NIC), comprising: The first processing module is used to execute gradient aggregation logic using a configurable matching table architecture through the RMT programmable data plane. The second processing module is used to handle hash collisions, resource overflows, and floating-point accumulation / sparse coding calculations through the RISC-V computing module during the execution of gradient aggregation logic.

[0024] As one embodiment of the present invention, such as Figure 2 As shown, the configurable matching table adopts a multi-stage pipeline structure, including: a protocol-independent parsing unit, a keyword extraction unit, an action matching unit, an aggregator, and a protocol-independent inverse parser. When a packet enters the protocol-independent parsing unit, the packet header is extracted. The packet header contains the task ID, packet sequence number ID, and metadata fields identifying gradient sparsity and data type. The packet data is temporarily stored in the data buffer. The packet header then enters the multi-stage action matching table stage. In each stage, the packet header enters the keyword extraction unit, extracts the first N bytes, and hashes them into an index. If the traffic is too high at this time, the aggregator of this index may be performing aggregation operations on other packets, resulting in a hash collision; or when the aggregator detects that the current aggregation table entry is short of resources (such as memory overflow or counter saturation), a resource overflow flag is generated. In cases of hash collisions or resource overflows, data packets are rerouted to the RISC-V computing module for processing. The RISC-V module uses software algorithms to resolve collisions and rehash, or dynamically allocates and schedules memory for overflowing aggregation operations. The action matching unit uses the extracted first N bytes as an index to find the corresponding aggregator. The aggregator performs gradient aggregation, judging whether to update or discard entries based on the metadata extracted from the packet header and the information read from the corresponding position in the aggregation table. During aggregation, the gradient vector values ​​are first converted to 32-bit high-precision floating-point numbers, and then converted back to their original precision values ​​after aggregation. It supports mixed precision calculations of FP8 / FP16 / BF16, which can support the processing of floating-point numbers of different precisions and ensure that no precision is lost during aggregation. After all stages have completed the processing of the packet header, the protocol-independent inverse parsing unit concatenates the processed packet header with the packet data stored in the data buffer.

[0025] Specifically, for sparse gradient scenarios, the RMT programmable data plane identifies sparse coding scenarios by parsing sparsity identification fields (such as sparsity thresholds and non-zero element index distributions) in the message header and forwards the sparse gradient data to the RISC-V computing module. The RISC-V computing module has a built-in dedicated instruction set that can identify and decode the non-zero element indices and corresponding values ​​in the sparse gradient matrix. For sparse gradients generated using Top-K or threshold filtering, the RISC-V module efficiently processes non-zero value aggregation through dynamic memory allocation and compressed storage structures (such as CSR / CSC formats), while skipping zero values ​​to reduce computational overhead. In addition, the module supports sparse-dense hybrid operations, automatically triggering a sparse coding optimization process when the gradient tensor sparsity exceeds a set threshold, significantly reducing data transmission volume and computational latency. After processing by the RISC-V module, the aggregation result is written back to the RMT's aggregation table, where the protocol-independent inverse parsing unit completes message reassembly, achieving joint processing.

[0026] Example 3: This invention also provides an on-network aggregation system, which adopts a Worker-Parameter Server (PS) distributed architecture to achieve efficient gradient aggregation and parameter synchronization. The system includes a worker end and a PS end. The PS end includes an on-network aggregation device (which is a programmable smart network interface card) and a broadcast module. The on-network aggregation device includes a first processing module for executing gradient aggregation logic using a configurable matching table architecture; and a second processing module for handling hash collisions, resource overflows, and floating-point accumulation / sparse coding calculations via a RISC-V computing module during the execution of the gradient aggregation logic. The worker end sends message data to the programmable smart network interface card of the PS end via the RDMA protocol to perform gradient compression and sparse processing. The updated parameters are then transmitted back to the worker end in real time via the broadcast module, forming a complete closed-loop process of "computation (forward / backward) → transmission (gradient sending) → aggregation (programmable smart network interface card) → synchronization (parameter broadcast)".

[0027] Furthermore, such as Figure 3 As shown, on the worker side, the deep learning training code written by the user runs on the PyTorch framework. PyTorch is an open-source deep learning framework that provides dynamic computation graphs and automatic differentiation. Users define the model structure, loss function, and optimizer through the PyTorch API to drive the training process. During the backpropagation phase of the training process, PyTorch's automatic differentiation mechanism (Autograd) automatically calculates the local gradient of each parameter (i.e., ...). The gradients (gradients of the loss function with respect to the weight parameters) are stored in GPU memory as tensors. The PyTorch Plug, acting as an interface with the BytePS library, intercepts these gradient tensors at this stage, takes them as input as the raw gradient tensors generated by PyTorch's reverse computation, and outputs the gradient data after preliminary formatting, which is then passed to the BytePS module (BytePS Core).

[0028] The BytePS module is responsible for communication scheduling and gradient preprocessing. Its input is gradient data passed from the PyTorch plugin, and its output is optimized gradient data blocks. The BytePS module performs the following operations: First, it performs chunking processing on the gradients, dividing large gradient tensors into data blocks suitable for network transmission; second, it supports gradient sparsification, reducing the amount of communication data through threshold filtering or Top-K selection; in addition, it coordinates the transmission timing of multiple worker nodes to avoid network congestion. The preprocessed gradient data is then fed into the CUDA computation module, which uses the GPU to perform possible compression and encoding calculations (such as quantization) to generate the final gradient data to be transmitted.

[0029] The gradient sending module transmits gradient data via the Remote Direct Memory Access (RDMA) protocol. RDMA is a networking technology that allows a computer to read or write data directly from memory (or GPU memory) without CPU intervention, thereby significantly reducing latency and CPU overhead. In this invention, combined with NVIDIA's GPU Direct RDMA technology, the gradient sending module can directly read gradient data from GPU memory and transmit it directly to the programmable smart network card on the PS side via an RDMA network interface card (RNIC). Its input is a gradient data block in GPU memory, and its output is a network data stream sent via RDMA.

[0030] On the PS side, the programmable smart NIC receives gradient data sent by all Workers via RDMA. The programmable smart NIC incorporates programmable hardware (such as an FPGA or ASIC) to offload computationally intensive operations. Its input is the gradient data streams from multiple Workers, and its output is the aggregated global gradient. The programmable smart NIC performs gradient aggregation calculations on hardware, while also supporting the decoding of compressed gradient data and the recovery of sparse data. The aggregation operation is entirely performed on the programmable smart NIC without host CPU intervention, significantly improving efficiency.

[0031] The broadcast module is responsible for transmitting the aggregated global gradient (or updated parameters) back to each Worker. Its input is the global parameters generated by the programmable smart NIC aggregation center, and its output is a parameter data stream transmitted via RDMA multicast. The broadcast module employs an efficient multicast strategy, sending the same parameter data to multiple Worker nodes simultaneously, reducing network bandwidth consumption.

[0032] The parameter receiving module on the worker side obtains the updated parameters from the network via RDMA and writes them directly to the GPU memory. PyTorch then uses these parameters to update the local model and starts the next training iteration, thus forming a complete closed-loop process of "computation (forward / backward) → transmission (gradient sending) → aggregation (programmable smart card) → synchronization (parameter broadcasting)".

[0033] In summary, the on-network aggregation system offloads gradient aggregation operations through programmable smart network interface cards, and combines PyTorch's automatic differentiation, BytePS's communication optimization, and RDMA's zero-copy transmission to achieve efficient and low-latency distributed training, significantly improving the throughput and scalability of large-scale deep learning training.

[0034] Example 4: This invention also provides a storage medium storing a computer program that executes an online aggregation method during runtime.

[0035] The embodiments described above are merely preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Various modifications and improvements made to the technical solutions of the present invention by those skilled in the art without departing from the spirit of the present invention should fall within the protection scope defined by the claims of the present invention.

Claims

1. An on-network aggregation method, characterized in that, include: Step S1: Execute gradient aggregation logic using a configurable matching table architecture; Step S2: In the execution of gradient aggregation logic, hash collisions, resource overflows, and floating-point accumulation / sparse encoding calculations are handled by the RISC-V computing module.

2. The on-network aggregation method as described in claim 1, characterized in that, The configurable matching table adopts a multi-stage pipeline structure, including: a protocol-independent parsing unit, a keyword extraction unit, an action matching unit, an aggregator, and a protocol-independent inverse parser. When a packet enters the protocol-independent parsing unit, the packet header is extracted. The packet header contains the task ID and packet sequence number ID, and the packet data is temporarily stored in the data buffer. In each stage, the packet header enters the keyword extraction unit, which extracts the first N bytes and hashes them into an index. The action matching unit uses the extracted first N bytes as the index to find the corresponding aggregator. The aggregator performs gradient aggregation, judging whether to update or discard table entries based on the metadata extracted from the packet header and the information read from the corresponding position in the aggregation table. After all stages have completed the processing of the packet header, the protocol-independent inverse parsing unit concatenates the processed packet header with the packet data stored in the data buffer.

3. An on-grid aggregation device, characterized in that, include: The first processing module is used to execute gradient aggregation logic using a configurable matching table architecture; The second processing module is used to handle hash collisions, resource overflows, and floating-point accumulation / sparse coding calculations through the RISC-V computing module during the execution of gradient aggregation logic.

4. The on-grid aggregation apparatus as described in claim 3, characterized in that, The configurable matching table adopts a multi-stage pipeline structure, including: a protocol-independent parsing unit, a keyword extraction unit, an action matching unit, an aggregator, and a protocol-independent inverse parser. When a packet enters the protocol-independent parsing unit, the packet header is extracted. The packet header contains the task ID and packet sequence number ID, and the packet data is temporarily stored in the data buffer. In each stage, the packet header enters the keyword extraction unit, which extracts the first N bytes and hashes them into an index. The action matching unit uses the extracted first N bytes as the index to find the corresponding aggregator. The aggregator performs gradient aggregation, judging whether to update or discard table entries based on the metadata extracted from the packet header and the information read from the corresponding position in the aggregation table. After all stages have completed the processing of the packet header, the protocol-independent inverse parsing unit concatenates the processed packet header with the packet data stored in the data buffer.

5. An online aggregation system, characterized in that, include: The worker side and the PS side, the PS side including: an on-network aggregation device and a broadcast module; the on-network aggregation device including: a first processing module for executing gradient aggregation logic using a configurable matching table architecture; a second processing module for handling hash collisions, resource overflows, and floating-point accumulation / sparse coding calculations through a RISC-V computing module during the execution of gradient aggregation logic; wherein, The worker sends message data to the on-network aggregation device on the PS side via the RDMA protocol, performs gradient compression and sparsity processing, and the updated parameters are transmitted back to the worker in real time via the broadcast module.

6. The on-network aggregation system as described in claim 5, characterized in that, The configurable matching table adopts a multi-stage pipeline structure, including: a protocol-independent parsing unit, a keyword extraction unit, an action matching unit, an aggregator, and a protocol-independent inverse parser. When a packet enters the protocol-independent parsing unit, the packet header is extracted. The packet header contains the task ID and packet sequence number ID, and the packet data is temporarily stored in the data buffer. In each stage, the packet header enters the keyword extraction unit, which extracts the first N bytes and hashes them into an index. The action matching unit uses the extracted first N bytes as the index to find the corresponding aggregator. The aggregator performs gradient aggregation, judging whether to update or discard table entries based on the metadata extracted from the packet header and the information read from the corresponding position in the aggregation table. After all stages have completed the processing of the packet header, the protocol-independent inverse parsing unit concatenates the processed packet header with the packet data stored in the data buffer.

7. A storage medium, characterized in that, The storage medium stores a computer program that, when executed, performs the online aggregation method as described in any one of claims 1 to 2.

Images