Lightweight model multi-GPU distribution training method and system based on autonomous controllable framework

Abstract

The embodiment of the invention provides a lightweight model multi-GPU distribution training method and system based on an autonomous controllable framework. The method comprises the following steps: constructing a full stack type localized distributed training framework, wherein the distributed training framework comprises a communication layer, a task scheduling layer, a resource management layer and a model compiler; model parameter dynamic division is carried out on the trained lightweight model, and a gradient compression and quantitative transmission strategy is adopted to reduce the communication data volume; when model training is executed, a calculation-communication overlapping mechanism is adopted, and forward calculation and gradient synchronization operation are executed in parallel; based on the GPU utilization rate and the video memory occupation data monitored in real time, distributing training tasks for the plurality of GPUs through a dynamic load balancing algorithm; a hardware-perceived energy efficiency optimization strategy is utilized to carry out optimization configuration on a memory and a PCIe bus of a domestic GPU so as to reduce training power consumption. The method and the system are higher in safety and better in flexibility, and the lightweight model training efficiency is greatly improved.

Description

Technical Field

[0001] This invention relates to the field of deep learning model training technology in the field of artificial intelligence, and more specifically, to a lightweight multi-GPU distributed training method and system for models based on an autonomous and controllable framework. Background Technology

[0002] With the widespread application of deep learning models, model size and data volume are growing exponentially, making single-GPU training insufficient to meet real-time and computational efficiency requirements. However, current mainstream multi-GPU distributed training frameworks (such as PyTorch's DistributedDataParallel and TensorFlow's MirroredStrategy) suffer from the following problems: 1. Dependence on external frameworks: Mainstream frameworks rely on foreign open-source technology stacks (such as CUDA and NCCL), which poses risks to code auditing and supply chain security. 2. Poor adaptability of lightweight models: Existing frameworks are optimized for large models (such as Transformer and ResNet-152), but have not been specifically optimized for lightweight models (such as MobileNetV3 and ShuffleNetV2) in terms of parameter distribution and communication overhead, resulting in wasted resources; 3. Insufficient communication and computing coordination: The traditional AllReduce algorithm is prone to high communication latency due to small batches of data in lightweight models, and lacks a dynamic load balancing mechanism; 4. Low energy efficiency: Energy efficiency optimization is not fully integrated with hardware characteristics (such as GPU memory bandwidth and PCIe 5.0 protocol) in edge devices or server clusters; In summary, existing multi-GPU distributed training frameworks rely on foreign frameworks, posing potential vulnerability risks. During lightweight model training, GPU utilization is less than 60%, and communication latency accounts for over 30%. Furthermore, they lack parameter partitioning strategies and dynamic resource scheduling mechanisms specifically for lightweight models. Summary of the Invention

[0003] The purpose of this invention is to provide a lightweight model multi-GPU distributed training method and system based on an autonomous and controllable framework. This method and system have higher security, better flexibility, and greatly improve the training efficiency of lightweight models.

[0004] To achieve the above objectives, embodiments of the present invention provide a lightweight multi-GPU distributed training method for models based on an autonomous and controllable framework, the method comprising: A full-stack domestic distributed training framework is constructed, which includes a communication layer, a task scheduling layer, a resource management layer, and a model compiler. The model parameters are dynamically partitioned for the trained lightweight model, and gradient compression and quantization transmission strategies are used to reduce the amount of communication data. During model training, a computation-communication overlap mechanism is used to perform forward computation and gradient synchronization operations in parallel. Based on real-time monitoring of GPU utilization and memory usage data, training tasks are allocated to multiple GPUs using a dynamic load balancing algorithm. By employing hardware-aware energy efficiency optimization strategies, the memory and PCIe bus of domestically produced GPUs are optimized to reduce training power consumption.

[0005] Preferably, the dynamic partitioning of model parameters includes automatically adjusting the partitioning granularity of the model on different GPUs based on the hierarchical structure type and number of parameters of the lightweight model, as well as the memory capacity of each GPU.

[0006] Preferably, the gradient compression and quantization transmission strategy includes quantizing the gradient tensor with FP16 or INT8 precision and applying the Top-K sparsification method to transmit only a portion of the important gradients.

[0007] Preferably, the computation-communication overlap mechanism includes asynchronously initiating gradient synchronization operations on GPU nodes that have completed computation while the GPU is performing backpropagation computation on the current batch of data.

[0008] Preferably, the dynamic load balancing algorithm includes dynamically adjusting the priority and target GPU of the micro-batch tasks to be allocated based on the real-time collected GPU utilization and memory usage indicators.

[0009] Preferably, the communication layer employs a hybrid AllReduce algorithm, dynamically selecting either a ring topology or a tree topology for gradient aggregation based on the number of GPUs in the cluster.

[0010] Preferably, the hardware-aware energy efficiency optimization strategy includes a unified memory architecture for domestically produced GPUs, adopting a block storage and data prefetching strategy, and making full use of the high bandwidth of the PCIe 5.0 protocol to improve the data transfer rate between GPUs.

[0011] On the other hand, the present invention provides a system for implementing the above-mentioned lightweight model multi-GPU distributed training method based on an autonomous and controllable framework, the system comprising: The framework building module is used to deploy the full-stack domestically produced distributed training framework. The model processing module is used to perform dynamic partitioning, gradient compression, and quantization of model parameters; The parallel training module is used to schedule and manage the multi-GPU collaborative training process, enabling computation-communication overlap. The resource scheduling module is used to monitor GPU status and execute dynamic load balancing algorithms; The energy efficiency optimization module is used to implement hardware-aware memory and bus optimization strategies.

[0012] According to the above technical solution, this invention constructs a full-stack domestic framework, avoiding reliance on foreign open-source code libraries, reducing supply chain risks, and improving security. By optimizing communication protocols, load balancing algorithms, and model segmentation strategies, it improves the training efficiency of lightweight models in multi-GPU environments, solving the efficiency problem. Simultaneously, by combining the characteristics of domestic GPU hardware (such as memory bandwidth and PCIe 5.0 protocol) for energy efficiency optimization, it reduces training power consumption, solving the energy efficiency problem. Furthermore, this invention supports rapid adaptation to various lightweight models and provides API interfaces for custom partitioning strategies and scheduling rules, improving flexibility.

[0013] Other features and advantages of the embodiments of the present invention will be described in detail in the following detailed description section. Attached Figure Description

[0014] The accompanying drawings are provided to further illustrate embodiments of the present invention and form part of the specification. They are used together with the following detailed description to explain the embodiments of the present invention, but do not constitute a limitation thereof. In the drawings: Figure 1 This is a schematic diagram of the structure of the ACDT Framework, a full-stack domestic framework in a lightweight model multi-GPU distributed training system based on an independently controllable framework, provided by the present invention. Detailed Implementation

[0015] The specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are for illustration and explanation only and are not intended to limit the scope of the present invention.

[0016] It should be noted that the acquisition, transmission, storage, use, and processing of data in the technical solution of this application all comply with relevant laws and regulations. In the embodiments of this application, certain existing industry solutions such as software, components, and models may be mentioned. These should be considered exemplary, intended only to illustrate the feasibility of implementing the technical solution of this application, and do not imply that the applicant has already used or necessarily used such solutions.

[0017] This invention provides a lightweight multi-GPU distributed training method for models based on an autonomous and controllable framework, the method comprising: A full-stack domestic distributed training framework is constructed, which includes a communication layer, a task scheduling layer, a resource management layer, and a model compiler. The model parameters are dynamically partitioned for the trained lightweight model, and gradient compression and quantization transmission strategies are used to reduce the amount of communication data. During model training, a computation-communication overlap mechanism is used to perform forward computation and gradient synchronization operations in parallel. Based on real-time monitoring of GPU utilization and memory usage data, training tasks are allocated to multiple GPUs using a dynamic load balancing algorithm. By employing hardware-aware energy efficiency optimization strategies, the memory and PCIe bus of domestically produced GPUs are optimized to reduce training power consumption.

[0018] In this embodiment, preferably, the communication layer supports the hybrid deployment of domestic GPUs (such as Huawei Ascend and Cambricon MLU) and x86 architecture, and is compatible with heterogeneous interfaces such as CUDA, ROCM and OpenCL; the task scheduling layer dynamically adjusts task allocation based on real-time GPU utilization and memory usage; the resource management layer provides hardware-aware resource allocation strategies and supports the PCIe 5.0 high-speed bus protocol; and the model compiler is used to optimize operator fusion and memory reuse mechanisms.

[0019] In terms of security features, it has passed the Level 3 certification of the National Information Security Protection System and supports encrypted communication using national cryptographic algorithms (SM2 / SM4); it provides code auditing tools to ensure that the framework has no external dependency vulnerabilities.

[0020] In terms of lightweight model optimization strategies, the specific strategies include: 1. Dynamic partitioning of model parameters: Layered partitioning strategy: Design a layered partitioning strategy based on the model structure (such as the convolutional layer of CNN, the linear layer of MLP) to reduce the amount of cross-GPU communication data; Dynamic partitioning algorithm: Automatically adjusts the partitioning granularity based on the number of model parameters and the GPU memory capacity.

[0021] 2. Gradient compression and quantization transmission: Hybrid precision quantization: Employs FP16 / INT8 quantization and sparse gradient updates (such as Top-K sparsification) to reduce communication bandwidth usage; Compression ratio: During MobileNetV3 training, the amount of gradient transfer data is reduced by 80%.

[0022] 3. Computation-communication overlap mechanism: Pipeline scheduling: Forward computation and gradients are executed synchronously and in parallel, utilizing GPU idle cycles to reduce waiting time. For example, in ShuffleNetV2 training, the computation-communication overlap rate reaches 90%, and the communication latency is reduced to 12%.

[0023] The multi-GPU collaborative scheduling algorithm provided by this invention includes: 1. Dynamic load balancing algorithm: Improved Round-Robin strategy: Dynamically adjust task allocation based on real-time GPU utilization and memory usage; Heterogeneous hardware-aware scheduling: Automatically adjusts the number of threads and memory allocation strategies for GPUs with different architectures (such as NVIDIA A100 vs Huawei Ascend 910).

[0024] 2. Asynchronous gradient aggregation optimization: Latency tolerance mechanism: Allows some nodes to participate in gradient updates later, reducing waiting time. For example, in a 16-GPU cluster, asynchronous aggregation reduces training time by 15%.

[0025] This invention also includes hardware-aware energy efficiency optimization, specifically including: 1. Domestic GPU memory optimization: Block storage and prefetching strategy: For Huawei Ascend's Unified Memory, a block storage mechanism is designed to maximize memory bandwidth utilization. For example, on the Atlas 300 device, memory bandwidth utilization is increased from 65% to 92%.

[0026] 2. PCIe 5.0 protocol utilization: High-speed data transfer: The data transfer rate between GPUs is increased to 4GB / s through the PCIe 5.0 protocol, and the communication latency is reduced to <2ms.

[0027] Furthermore, the present invention optimizes the communication layer, specifically including: 1. Hybrid AllReduce algorithm: Topology switching logic: Dynamically switch between ring topology (Ring AllReduce) and tree topology (Tree AllReduce) based on the number of GPUs. Code example is as follows: Python def hybrid_allreduce(gpus, data): if len(gpus) <= 8: return ring_allreduce(data) else: return tree_allreduce(data) 2. Gradient compression module: Quantization + Sparsification Process: Convert the gradient tensor to INT8 format (FP16 quantization). Apply Top-K sparsity (preserving the top K% maximum values). Transmit the sparsified gradient tensor.

[0028] The compression ratio is calculated using the following formula: 3. Task scheduling algorithm: Dynamic task queue: A FIFO priority queue that monitors GPU idle status in real time and allocates micro-batch tasks. The scheduling pseudocode is as follows: Python while not all_tasks_complete: For GPU in GPUs: if gpu.is_idle(): task = get_next_task() assign_task(gpu, task) Heterogeneous hardware-aware scheduling: The configuration file code is as follows: Json {"NVIDIA_A100": {"threads": 256, "memory_limit": 40}, "Huawei_Shengteng 910": {"threads": 128, "memory_limit": 32} } 4. Model compiler optimization: Operator fusion: Rule: Combine consecutive convolutional layers and BatchNorm layers into a single operator to reduce kernel call overhead. For example, in MobileNetV3, the number of convolution-BatchNorm combined operators is reduced by 40%.

[0029] Memory reuse mechanism: Static analysis model computation graph: Reusable intermediate results are identified through graph traversal algorithms. For example, in ShuffleNetV2, memory reuse rate was improved from 50% to 85%.

[0030] In conjunction with the above specific implementation methods, this invention enhances security by utilizing a full-stack, independently controllable framework, achieving Level 3 certification under the National Information Security Protection Scheme, and supporting encrypted communication using national cryptographic algorithms. When deployed on edge devices, the risk of model parameter leakage is reduced by 90%. Regarding efficiency improvements, a 14.8× speedup is achieved when training MobileNetV3 with a 16-GPU cluster; GPU utilization increases from 60% to 85% during lightweight model training; and gradient synchronization time decreases from 38% to 12%. In terms of energy efficiency optimization, through the deployment of domestically produced GPUs, specifically on Huawei Atlas 300 devices, training power consumption is reduced to 15W; and the energy efficiency ratio increases from 18 to 25.6 on Huawei Ascend 910 devices. Furthermore, in terms of enhanced flexibility, it supports rapid adaptation of lightweight models such as YOLOv4-tiny and EfficientNet-B0, enabling multi-model adaptation; and provides APIs for custom partitioning strategies and scheduling rules, allowing users to flexibly adjust configurations.

[0031] In addition, such as Figure 1 As shown, the present invention also provides a system for implementing the above-described lightweight model multi-GPU distributed training method based on an autonomous and controllable framework, the system comprising: The framework building module is used to deploy the full-stack domestically produced distributed training framework. The model processing module is used to perform dynamic partitioning, gradient compression, and quantization of model parameters; The parallel training module is used to schedule and manage the multi-GPU collaborative training process, enabling computation-communication overlap. The resource scheduling module is used to monitor GPU status and execute dynamic load balancing algorithms; The energy efficiency optimization module is used to implement hardware-aware memory and bus optimization strategies.

[0032] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0033] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0034] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0035] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0036] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0037] Memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0038] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0039] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0040] The above are merely embodiments of this application and are not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A lightweight multi-GPU distributed training method for models based on an autonomous and controllable framework, characterized in that, The method includes: A full-stack domestic distributed training framework is constructed, which includes a communication layer, a task scheduling layer, a resource management layer, and a model compiler. The model parameters are dynamically partitioned for the trained lightweight model, and gradient compression and quantization transmission strategies are used to reduce the amount of communication data. During model training, a computation-communication overlap mechanism is used to perform forward computation and gradient synchronization operations in parallel. Based on real-time monitoring of GPU utilization and memory usage data, training tasks are allocated to multiple GPUs using a dynamic load balancing algorithm. By employing hardware-aware energy efficiency optimization strategies, the memory and PCIe bus of domestically produced GPUs are optimized to reduce training power consumption.

2. The lightweight model multi-GPU distributed training method based on an autonomous and controllable framework according to claim 1, characterized in that, The dynamic partitioning of model parameters includes automatically adjusting the partitioning granularity of the model on different GPUs based on the hierarchical structure type and number of parameters of the lightweight model, as well as the memory capacity of each GPU.

3. The lightweight model multi-GPU distributed training method based on an autonomous and controllable framework according to claim 1, characterized in that, The gradient compression and quantization transmission strategy includes quantizing the gradient tensor with FP16 or INT8 precision and applying the Top-K sparsification method to transmit only a portion of the important gradients.

4. The lightweight model multi-GPU distributed training method based on an autonomous and controllable framework according to claim 1, characterized in that, The computation-communication overlap mechanism includes asynchronously initiating gradient synchronization operations on GPU nodes that have already completed computations while the GPU is performing backpropagation computations on the current batch of data.

5. The lightweight model multi-GPU distributed training method based on an autonomous and controllable framework according to claim 1, characterized in that, The dynamic load balancing algorithm includes dynamically adjusting the priority and target GPU of the micro-batch tasks to be allocated based on the real-time collected GPU utilization and memory usage indicators.

6. The lightweight model multi-GPU distributed training method based on an autonomous and controllable framework according to claim 1, characterized in that, The communication layer employs a hybrid AllReduce algorithm, dynamically selecting either a ring topology or a tree topology for gradient aggregation based on the number of GPUs in the cluster.

7. The lightweight model multi-GPU distributed training method based on an autonomous and controllable framework according to claim 1, characterized in that, The hardware-aware energy efficiency optimization strategy includes a unified memory architecture for domestic GPUs, adopting block storage and data prefetching strategies, and making full use of the high bandwidth of the PCIe 5.0 protocol to improve the data transfer rate between GPUs.

8. A system for implementing the lightweight multi-GPU distributed training method for models based on an autonomous and controllable framework as described in any one of claims 1-7, characterized in that, The system includes: The framework building module is used to deploy the full-stack domestically produced distributed training framework. The model processing module is used to perform dynamic partitioning, gradient compression, and quantization of model parameters; The parallel training module is used to schedule and manage the multi-GPU collaborative training process, enabling computation-communication overlap. The resource scheduling module is used to monitor GPU status and execute dynamic load balancing algorithms; The energy efficiency optimization module is used to implement hardware-aware memory and bus optimization strategies.

Images