Training data processing method and device, electronic equipment and storage medium

Abstract

The invention provides a training data processing method and device, electronic equipment and a storage medium, and belongs to the technical field of artificial intelligence, and the method comprises the steps: reading training data from a remote distributed file system, and storing the training data in a memory of a DPU; preprocessing the training data in the DPU memory; and directly writing the preprocessed training data from the memory of the DPU into the video memory of each local GPU. According to the method, the training data are read through the DPU, the training data are preprocessed and directly written into the video memory of each GPU, two tasks, namely data reading and data preprocessing, which are traditionally executed by the CPU are completely migrated to the DPU to be executed, and a data shortcut from remote storage to the video memory of the GPU is constructed by utilizing the DPU, so that the bottlenecks of the CPU and the memory of a host are bypassed; and the communication delay of the distributed training data preparation stage is reduced.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a training data processing method, apparatus, electronic device, and storage medium. Background Technology

[0002] With the rapid increase in the parameters of deep learning models and the size of training datasets, distributed parallel training has become the standard mode for model training in cloud data centers. However, in the distributed training process, the massive data flow and communication overhead between nodes have become major performance bottlenecks restricting training efficiency and system scalability.

[0003] In traditional distributed training workflows, data flow typically follows a fixed, high-overhead path. First, during the data preparation phase, each training node needs to retrieve training data from a remote distributed file system (such as a data lake). The data is first loaded into the memory of the node's Central Processing Unit (CPU), where the CPU performs preprocessing operations such as data augmentation. Finally, it is transferred to the GPU's video memory via a high-speed PCIe (Peripheral Component Interconnect Express) bus. This process not only frequently consumes CPU computing resources and repeatedly triggers hardware interrupts, resulting in significant context switching overhead, but also causes severe contention for bus resources within the node due to the multiple data transfers between storage, CPU, and GPU, leading to high communication latency in distributed training. Summary of the Invention

[0004] This invention provides a training data processing method to solve the technical problem of high communication latency in distributed training in the prior art.

[0005] This invention provides a training data processing method, comprising: Training data is read from a remote distributed file system and stored in the DPU's memory; The training data in the DPU memory is preprocessed; The preprocessed training data is written directly from the DPU's memory to the local video memory of each GPU.

[0006] According to a training data processing method provided by the present invention, reading training data from a remote distributed file system includes: The training data is read by actively initiating a data read request to the remote distributed file system using RDMA technology.

[0007] According to a training data processing method provided by the present invention, the step of directly writing the preprocessed training data from the DPU's memory into the local GPU's video memory includes: The training data is written directly from the DPU's memory to the local video memory of each GPU using data pass-through technology.

[0008] According to a training data processing method provided by the present invention, after writing the preprocessed training data directly from the DPU's memory into the local GPU's video memory, the method further includes: Receive local gradient data obtained by each local GPU based on the training data and store it in the memory of the DPU; The local gradient data in the DPU's memory are aggregated to obtain the aggregated gradient; The aggregated gradient is synchronized to the DPUs of the other distributed computing nodes to obtain the global gradient; The global gradient is distributed to each local GPU.

[0009] According to a training data processing method provided by the present invention, synchronizing the aggregated gradient to the DPUs of the other distributed computing nodes includes: The aggregated gradient is synchronized to the DPUs of the other distributed computing nodes using RDMA technology.

[0010] According to a training data processing method provided by the present invention, distributing the global gradient to each local GPU includes: The global gradient is distributed to each local GPU using a data passthrough technique.

[0011] The present invention also provides a training data processing apparatus, characterized in that it comprises: The read module is used to read training data from a remote distributed file system and store it in the DPU's memory; The processing module is used to preprocess the training data in the DPU memory; The write module is used to write the preprocessed training data directly from the DPU's memory to the local video memory of each GPU.

[0012] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the training data processing method as described above.

[0013] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the training data processing method as described above.

[0014] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the training data processing method as described above.

[0015] The training data processing method, apparatus, electronic device, and storage medium provided by this invention read training data through a DPU, preprocess the training data, and write it directly to the video memory of each GPU. This completely migrates the two tasks of data reading and data preprocessing, which are traditionally performed by the CPU, to the DPU. By utilizing the DPU, a data shortcut from remote storage to GPU video memory is constructed, bypassing the bottlenecks of the host CPU and memory, and reducing the communication latency in the distributed training data preparation stage. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0017] Figure 1 This is a schematic diagram of the structure of the distributed training system provided by the present invention.

[0018] Figure 2 This is a flowchart illustrating the training data processing method provided by the present invention.

[0019] Figure 3 This is a schematic diagram illustrating the principle of the gradient data synchronization stage provided by the present invention.

[0020] Figure 4 This is a schematic diagram of the distributed training process provided by the present invention.

[0021] Figure 5 This is a schematic diagram of the training data processing device provided by the present invention.

[0022] Figure 6 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0024] The technical solutions related to this invention will be introduced here first.

[0025] Related technical solution 1 first intercepts network requests through the service mesh data plane within the DPU, then forwards the requests to the hardware acceleration engine within the DPU for encryption processing, and finally returns the encrypted data to the data plane for forwarding. Although this solution demonstrates the DPU's ability to offload network protocol processing and effectively free up host CPU resources, its application scenarios mainly focus on the network communication layer. It does not address the optimization of the entire data preparation chain from remote storage systems to GPU computing units in distributed training scenarios, nor does it solve the core communication bottleneck of gradient data synchronization between multiple GPUs within a node.

[0026] Related technical solution 2 provides a collaborative working mode between CPU and DPU, where the CPU performs the main iterative calculations, while the DPU performs residual calculations for data verification in parallel. This achieves asynchronous parallelism between CPU computing tasks and DPU data processing tasks, reducing synchronization wait times during computation. Although this solution separates computation from some data processing tasks, its data interaction model is limited to the CPU main memory and DPU, and it does not solve the problem of excessively long end-to-end data flow paths and multiple data copies in distributed training. Therefore, its optimization model is difficult to directly apply to AI training scenarios where GPUs are the computing core.

[0027] Related technical solution 3 provides an internal communication architecture for a DPU chip, connecting multiple acceleration units with different functions within the chip through an on-chip network, achieving efficient and flexible forwarding of internal data streams. This design enhances the DPU's ability to handle complex data streams and multi-task scenarios. However, this solution only focuses on optimizing the internal architecture of the DPU chip itself and does not provide a system-level application method to explain how to utilize this DPU to solve system-level data stream bottlenecks across storage, network, and computing nodes in distributed training clusters, especially the specific offloading processes in the two key stages of data preprocessing and gradient transmission.

[0028] The following is combined Figures 1-6 This invention describes the training data processing method, apparatus, electronic device, and storage medium provided by the present invention.

[0029] Figure 1 The distributed training system provided by the present invention includes a remote distributed file system (e.g., a data lake) and multiple distributed computing nodes. Each computing node includes a DPU (Data Processing Unit), multiple GPUs, a host CPU, and host memory.

[0030] Figure 2This is a flowchart illustrating the training data processing method provided by the present invention, as shown below. Figure 2 As shown, steps S1, S2 and S3 are included but are not limited to.

[0031] Step S1: Read the training data from the remote distributed file system and store it in the DPU's memory.

[0032] When a distributed training task starts, the central controller or master compute node logically partitions the entire training dataset according to the global configuration. This partitioning strategy aims to achieve data parallelism, ensuring that each GPU participating in training processes a different subset of data within the same training cycle. Partition information (such as file paths and data index ranges) is distributed to the DPUs of each compute node in the cluster, serving as the basis for their subsequent data fetching tasks. The DPUs then establish a data task queue in memory based on this information.

[0033] Each compute node's DPU will transfer the partitioned training data directly from the memory of the remote distributed file system to the DPU's memory, such as DRAM (Dynamic Random Access Memory), according to its task queue. The entire process does not require the intervention of the host CPU, thereby eliminating the huge overhead caused by system calls, interrupt handling and context switching in traditional I / O operations and greatly reducing data acquisition latency.

[0034] Step S2: Preprocess the training data in the DPU memory.

[0035] After the raw training data blocks are loaded into the DPU's memory at high speed, the multi-core programmable ARM processor on the DPU immediately initiates the preset data augmentation process. Users can predefine a series of data augmentation functions and compile these functions into programs that can run on the DPU. The multiple ARM processors of the DPU can process different data samples in parallel or perform pipelined augmentation operations on the same data sample, thereby achieving high-throughput data preprocessing. This design shifts computationally intensive tasks that originally consumed a lot of CPU cycles to a dedicated DPU, freeing up the host CPU to perform other more critical scheduling or computational tasks.

[0036] Step S3: Write the preprocessed training data directly from the DPU's memory to the local GPU's video memory.

[0037] After data augmentation is complete, the DPU constructs a high-speed data channel from DPU memory to the memory of each GPU. This high-speed channel directly writes the processed, training-ready data into the pre-allocated GPU memory buffer. This step avoids the extra step of copying data from host memory to GPU memory in the traditional process, further reducing data copying and PCIe bus transactions.

[0038] Then each GPU can perform training processes such as forward propagation and backward propagation based on the training data it has.

[0039] As can be seen from the above, the training data processing method of the present invention reads training data through DPU, preprocesses the training data, and writes it directly to the video memory of each GPU. The two tasks of data reading and data preprocessing, which are traditionally performed by the CPU, are completely migrated to the DPU. The DPU is used to build a data shortcut from remote storage to GPU video memory, bypassing the bottleneck of host CPU and memory, and reducing the communication latency in the distributed training data preparation stage.

[0040] By completely removing the host CPU and host memory from the critical path of the data preparation phase, hardware offloading of data I / O and data processing is achieved. This allows data preprocessing to overlap with the training computation process on the GPU to the greatest extent, thereby significantly reducing the waiting time for each training iteration.

[0041] In one embodiment, step S1 of the present invention, reading training data from a remote distributed file system, may specifically include: The training data is read by actively initiating a data read request to a remote distributed file system using RDMA technology.

[0042] The DPU of this invention can utilize its built-in high-performance RDMA (remote direct memory access) engine to actively initiate data read requests to a remote distributed file system to read training data.

[0043] RDMA technology allows the DPU to directly access the memory of a remote distributed file system without operating system intervention, enabling high-bandwidth and low-latency training data acquisition.

[0044] In one embodiment, step S3 of the present invention may specifically include: The training data is written directly from the DPU's memory to the local video memory of each GPU using data pass-through technology.

[0045] After data augmentation is complete, the DPU can utilize data passthrough technologies such as GPUDirect to build a high-speed data channel from the DPU memory to the various GPU memory locations. Through data passthrough technology, the DPU's DMA (direct memory access) engine can directly write the processed, training-ready data to the pre-allocated GPU memory buffer via the PCIe bus.

[0046] Data passthrough technology allows the DPU to bypass CPU memory and exchange data directly with GPU memory, which can significantly reduce latency.

[0047] In existing technologies, communication latency issues exist not only during the data preparation phase but also during the gradient data synchronization phase within computing nodes. After multiple GPUs within a computing node complete a round of backpropagation computation, their generated gradients need to be aggregated before global synchronization between nodes can occur. Existing solutions typically employ a complex internal communication model: first, each GPU within the node uses a round of ring-allreduce communication to aggregate some of the aggregation results onto each GPU; then, the master GPU sends data to the network interface card (NIC) for cross-node communication; finally, after receiving the global aggregation result, it distributes the final result back to each GPU within the computing node using a broadcast algorithm. Throughout this process, gradient data needs to be transmitted multiple times between GPUs and between GPUs and the host CPU / NIC, resulting in lengthy data paths and unnecessary communication latency and data preparation overhead.

[0048] To reduce communication latency during the gradient data synchronization phase, in one embodiment, after step S3, the training data processing method of the present invention may further include: Receive local gradient data obtained by each GPU based on the training data and store it in the DPU's memory; The aggregated gradient is obtained by aggregating the local gradient data in the memory of the DPU. The aggregated gradient is synchronized to the DPUs of other distributed computing nodes to obtain the global gradient; Distribute the global gradient to each local GPU.

[0049] The gradient data synchronization stage of the present invention is as follows: Figure 3As shown, scatter-reduce is a scatter-reduce algorithm, and Allgather is a full collection algorithm. After the backpropagation phase of a training iteration, each GPU within the computing node computes its local gradient data. In this invention, all GPUs no longer communicate point-to-point with each other, but instead send their gradient data directly to the DPU on the same node via the PCIe bus. This process constitutes a "many-to-one" data aggregation. The DPU allocates a dedicated aggregation buffer for the global gradient in its memory. When it receives gradient data blocks from different GPUs, the DPU's ARM processor immediately performs efficient aggregation operations. This is typically an element-wise summation operation. Since all local gradient data is concentrated in the DPU's memory, aggregation computation can be performed efficiently, avoiding the additional synchronization overhead and latency caused by step-by-step aggregation across multiple devices. After aggregation is completed, the DPU's memory holds the aggregated gradient representing the overall gradient update direction of the local computing node.

[0050] The DPU holding the aggregated gradient of a computing node acts as the sole network communication endpoint for that node, synchronizing global gradients with DPUs on other computing nodes. This synchronization typically employs efficient aggregate communication algorithms, such as Ring-Allreduce. Since the network communication task is entirely handled by the DPU, the host CPU does not need to participate in the packaging, sending, and receiving of network data, thus avoiding the occupation and interference of network I / O on CPU resources.

[0051] Through cross-compute node communication, each compute node's DPU obtains the final, globally synchronized gradient tensor. The DPU then performs a distribution task, broadcasting the complete global gradient simultaneously or sending it separately to the memory of each GPU within the local compute node. Upon receiving the global gradient, the GPU can immediately use it to update its local model parameters, preparing for the next iteration.

[0052] In this way, the present invention uses the DPU as the data aggregation and distribution center within the node and designs a "GPU-DPU" direct connection data exchange model, which simplifies the data flow of gradient synchronization and replaces the complex path of transmission between the CPU and multiple GPUs in the traditional solution, effectively reducing the communication latency in the gradient data synchronization stage.

[0053] In one embodiment, synchronizing the aggregated gradient to the DPUs of other distributed computing nodes may further include: The aggregated gradient is synchronized to the DPUs of other distributed computing nodes using RDMA technology.

[0054] Communication between DPUs can also utilize RDMA technology, ensuring high bandwidth and low latency for cross-node gradient exchange.

[0055] In one embodiment, the present invention distributes global gradients to local GPUs, which may further include: The global gradient is distributed to each local GPU through data passthrough technology.

[0056] Data passthrough technology allows the DPU to bypass CPU memory and exchange data directly with GPU memory, which can significantly reduce the communication latency of global gradients.

[0057] This invention was validated in a distributed training system consisting of four computing nodes, each of which is equipped with a host CPU, an NVIDIA RTX 4090 GPU, and an NVIDIA BlueField-2 DPU.

[0058] The specific implementation process is as follows: Figure 4 As shown. First, during the training preparation phase, the cluster's central controller issues training task instructions to each compute node. These instructions include metadata such as model architecture and training parameters. Based on this metadata, the host CPU instructs the GPU to pre-allocate buffers in its video memory to store training data, model parameters, and gradients. The DPU receives the data task configuration for the CIFAR-10 dataset and uses its RDMA engine to pull image data from a remote Lustre distributed file system. Once the data arrives in the DPU's onboard memory, its ARM processor immediately performs data augmentation operations such as random cropping, flipping, and normalization of the images. After processing, the DPU uses GPUDirect technology to directly write one or more batches of augmented data into the pre-allocated data buffers in the GPU's video memory, making them readily available to the GPU during the training phase.

[0059] Secondly, during the training phase, the GPU reads a data slice from its memory buffer, performs forward and backward propagation calculations of the model, and generates local gradient data. After backpropagation is complete, multiple GPUs within a compute node send their respective local gradient data directly to the DPU for aggregation. The DPU internally performs a summation and averaging operation on multiple local gradient data sets to generate the node aggregated gradient. Subsequently, this DPU, acting as the node representative, performs an efficient Ring-Allreduce ensemble communication with the DPUs of the other three compute nodes to compute the global gradient. After communication, the DPU then directly distributes the global gradient back to its local GPUs. Upon receiving the global gradient, the GPU immediately uses it to update its local model parameters and prepares to start the next iteration until the current training epoch is completed.

[0060] Finally, after a training epoch, the validation phase begins. Once an epoch of training is complete, the GPU sends the currently trained model parameters to the host CPU. Because this invention offloads heavy tasks such as data preprocessing and gradient communication management from the CPU, the host CPU has sufficient computing power to immediately calculate the accuracy and loss values ​​of the received model for model validation. Crucially, the GPU does not need to wait for validation to complete; it can continue using the data preloaded by the DPU for the next epoch and begin a new training iteration. This invention's pipelined parallel operation mode for CPU and GPU avoids the serial waiting caused by CPU resource contention in traditional architectures, further improving the overall utilization of system resources and end-to-end training efficiency.

[0061] This invention successfully offloads two major bottlenecks in the traditional training process—data preprocessing and intra-node gradient communication—to the DPU, effectively freeing up host CPU resources and constructing a low-latency data path from storage to the GPU and between GPUs for gradient exchange. This invention solves the performance limitations caused by CPU bottlenecks and the complexity of data flow in traditional architectures, significantly accelerating large-scale distributed training processes and improving the parallel utilization efficiency of heterogeneous computing resources in practical applications.

[0062] like Figure 5 As shown, the training data processing apparatus provided by the present invention includes, but is not limited to: The read module is used to read training data from a remote distributed file system and store it in the DPU's memory; The processing module is used to preprocess the training data in the DPU memory; The write module is used to write the preprocessed training data directly from the DPU's memory to the local video memory of each GPU.

[0063] It should be noted that the training data processing device provided by the present invention can execute the training data processing method described in any of the above embodiments during specific operation, and this embodiment will not elaborate on this.

[0064] Figure 6This is a schematic diagram of the electronic device provided by the present invention. The electronic device may include: a processor, a communication interface, memory, and a communication bus, wherein the processor, communication interface, and memory communicate with each other through the communication bus. The processor can call logical instructions in the memory to execute a training data processing method, which includes: reading training data from a remote distributed file system and storing it in the memory of the DPU; preprocessing the training data in the DPU memory; and writing the preprocessed training data directly from the DPU memory to the video memory of each local GPU.

[0065] Furthermore, the logical instructions in the aforementioned memory can be implemented as software functional units and sold or used as independent products, and can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0066] On the other hand, the present invention also provides a computer program product, the computer program product including a computer program stored on a non-transitory computer-readable storage medium, the computer program including program instructions, and when the program instructions are executed by a computer, the computer is able to execute the training data processing method provided in the above embodiments, the method including: reading training data from a remote distributed file system and storing it in the memory of the DPU; preprocessing the training data in the DPU memory; and writing the preprocessed training data directly from the DPU memory to the video memory of each local GPU.

[0067] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, it is implemented to perform the training data processing method provided in the above embodiments. The method includes: reading training data from a remote distributed file system and storing it in the memory of the DPU; preprocessing the training data in the DPU memory; and writing the preprocessed training data directly from the DPU memory to the video memory of each local GPU.

[0068] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0069] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0070] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A training data processing method, characterized in that, include: Training data is read from a remote distributed file system and stored in the DPU's memory; The training data in the DPU memory is preprocessed; The preprocessed training data is written directly from the DPU's memory to the local video memory of each GPU.

2. The training data processing method according to claim 1, characterized in that, The step of reading training data from a remote distributed file system includes: The training data is read by actively initiating a data read request to the remote distributed file system using RDMA technology.

3. The training data processing method according to claim 1, characterized in that, The step of writing the preprocessed training data directly from the DPU's memory to the local GPU's video memory includes: The training data is written directly from the DPU's memory to the local video memory of each GPU using data pass-through technology.

4. The training data processing method according to claim 1, characterized in that, After writing the preprocessed training data directly from the DPU's memory to the local GPU's video memory, the process further includes: Receive local gradient data obtained by each local GPU based on the training data and store it in the memory of the DPU; The local gradient data in the DPU's memory are aggregated to obtain the aggregated gradient; The aggregated gradient is synchronized to the DPUs of the other distributed computing nodes to obtain the global gradient; The global gradient is distributed to each local GPU.

5. The training data processing method according to claim 4, characterized in that, The step of synchronizing the aggregated gradient to the DPUs of the other distributed computing nodes includes: The aggregated gradient is synchronized to the DPUs of the other distributed computing nodes using RDMA technology.

6. The training data processing method according to claim 4, characterized in that, The step of distributing the global gradient to each local GPU includes: The global gradient is distributed to each local GPU using a data passthrough technique.

7. A training data processing apparatus, characterized in that, include: The read module is used to read training data from a remote distributed file system and store it in the DPU's memory; The processing module is used to preprocess the training data in the DPU memory; The write module is used to write the preprocessed training data directly from the DPU's memory to the local video memory of each GPU.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the training data processing method as described in any one of claims 1 to 6.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the training data processing method as described in any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the training data processing method as described in any one of claims 1 to 6.

Images