A distributed training method, device and equipment for large-scale models

Abstract

The present invention discloses a distributed training method, device, and equipment for large-scale models, comprising: configuring multiple computing nodes on a current device according to the scale of the large model to be trained and the training task, and configuring the hardware and software resources of each computing node to obtain a distributed training environment; using parallel strategies of different dimensions to divide the training data set and the large model to be trained to obtain corresponding multiple data blocks and multiple groups of model layers; loading model parameters, multiple data blocks, and multiple groups of model layers onto corresponding computing nodes in the distributed training environment to start the training process; during the training process, monitoring the operating status of each computing node, dynamically adjusting the training task of the corresponding computing node by introducing an adaptive scheduling mechanism, and saving the trained target large model when a preset training termination condition is met. This method can meet the requirements of efficient, stable, and low-cost training.

Description

Technical Field

[0001] The present invention relates to the field of model training technology, and in particular to a distributed training method, apparatus and equipment for large-scale models. Background Art

[0002] With the rapid development of deep learning technology, large language models (LLMs) have demonstrated tremendous application potential in numerous fields. Their parameter size continues to expand, from billions to trillions, posing unprecedented challenges to model training. On the one hand, the time and computing resources required for training have skyrocketed, placing a heavy burden on hardware infrastructure. On the other hand, existing distributed training methods have increasingly exposed numerous challenges when dealing with such ultra-large models. For example, some existing distributed training strategies are often limited by the memory capacity of a single device when processing large models, resulting in incomplete model deployment. Meanwhile, simple multi-device expansion can lead to excessive communication overhead and load imbalance, severely hindering training efficiency. Furthermore, existing methods lack sophisticated utilization of input data features, making it difficult to achieve efficient and accurate parallel processing and model optimization for long texts or complex contexts. This makes it difficult to ensure the stability and consistency of model generation results. In summary, existing methods have numerous shortcomings in meeting the requirements for efficient training of large language models, making them difficult to meet the current high-efficiency requirements for large-scale model training. Summary of the Invention

[0003] In view of this, the purpose of the present invention is to propose a distributed training method, device and equipment for large-scale models, aiming to solve the problems of existing distributed training methods for large-scale models, such as high hardware resource cost, unbalanced load, low efficiency, etc., which cannot meet the requirements of efficient, stable and low-cost training.

[0004] To achieve the above objectives, the present invention provides a distributed training method for large-scale models, the method comprising:

[0005] Configure multiple computing nodes on the current device based on the scale of the large model to be trained and the training task, and configure the hardware and software resources of each computing node to obtain a distributed training environment.

[0006] Using parallel strategies of different dimensions to divide the training data set and the large model to be trained to obtain corresponding multiple data blocks and multiple groups of model layers, wherein the parallel strategies include data parallelism, model parallelism, pipeline parallelism, and context parallelism;

[0007] Initiate a training process by loading model parameters, multiple data blocks, and multiple sets of model layers onto corresponding computing nodes in the distributed training environment;

[0008] During the training process, the operating status of each computing node is monitored, and the training tasks of the corresponding computing nodes are dynamically adjusted by introducing an adaptive scheduling mechanism. When the preset training termination conditions are met, the trained target large model is saved.

[0009] Preferably, configuring the hardware resources and software resources of each computing node includes:

[0010] The configuration of the hardware resources includes configuring the number of GPUs / TPUs and the memory capacity of each computing node;

[0011] The configuration of the software resources includes installing a compatible operating system, corresponding GPU / TPU drivers, a distributed training framework, and compatible dependent libraries on each computing node.

[0012] Preferably, the use of parallel strategies of different dimensions to divide the training data set and the large model to be trained includes:

[0013] A data parallel strategy is used to divide the training data set into multiple data blocks according to the number of computing nodes, and the size of the data blocks is dynamically adjusted according to the computing power and storage capacity of each computing node;

[0014] The large model to be trained is divided into layers by a model parallel strategy to obtain multiple groups of model layers, and each group of model layers is assigned to a corresponding computing node;

[0015] A pipeline parallel strategy is used to divide the large model to be trained into multiple consecutive stages, and the stages are assigned to the corresponding computing nodes for sequential execution.

[0016] A context-parallel parallel strategy is adopted to segment the input sequence according to its contextual features, assign each segment to a matching computing node, and dynamically adjust the attention mechanism.

[0017] Preferably, the large model to be trained is divided into layers by a parallel strategy of model parallelism to obtain multiple groups of model layers, and each group of model layers is assigned to a corresponding computing node, including:

[0018] Determine the number of segments of the large model to be trained according to the number of computing nodes, obtain multiple stages, and assign the continuous model layers of each stage to the corresponding computing nodes;

[0019] During forward propagation, the input sequence is passed through each computing node in sequence;

[0020] During back propagation, the gradient is transferred layer by layer from the last calculation node to the previous calculation node and the weights are updated.

[0021] Preferably, the context-parallel parallel strategy is adopted to segment the input sequence according to the context features of the input sequence, assign each segment to a matching computing node and dynamically adjust the attention mechanism, including:

[0022] Extract the input sequence according to its length and attention weight distribution to obtain context features;

[0023] Based on context features, the input sequence is segmented according to the preset window size, and segments corresponding to different context complexities are assigned to matching computing nodes based on the context complexity of each segment and the load status of the current corresponding computing node;

[0024] The attention mechanism is dynamically adjusted according to the context complexity corresponding to different fragments. After each computing node completes processing of the current fragment, the context complexity information is fed back to the scheduler, and the scheduler optimizes the task division and resource scheduling of the next micro-batch based on the context complexity information.

[0025] Preferably, the monitoring of the operating status of each computing node and the dynamic adjustment of the training tasks of the corresponding computing node by introducing an adaptive scheduling mechanism include:

[0026] Obtain the load status of each computing node by monitoring the GPU utilization, memory usage, and task completion time of each computing node;

[0027] The training task allocation of each stage in the pipeline parallelism is dynamically adjusted according to the load status of the corresponding computing node, and micro-batch data processing at different stages is performed asynchronously.

[0028] Preferably, the training termination condition includes reaching a preset training round or the loss value change of the validation set is less than a preset threshold.

[0029] To achieve the above object, the present invention further provides a distributed training device for large-scale models, the device comprising:

[0030] The configuration unit is used to configure multiple computing nodes on the current device according to the scale of the large model to be trained and the training task, and configure the hardware and software resources of each computing node to obtain the established distributed training environment;

[0031] A partitioning unit is configured to partition the training data set and the large model to be trained using parallel strategies of different dimensions to obtain corresponding multiple data blocks and multiple groups of model layers, wherein the parallel strategies include data parallelism, model parallelism, pipeline parallelism, and context parallelism;

[0032] A training unit, configured to start a training process by loading model parameters, multiple data blocks, and multiple groups of model layers onto corresponding computing nodes in the distributed training environment;

[0033] The monitoring unit is used to monitor the operating status of each computing node during the training process, dynamically adjust the training tasks of the corresponding computing nodes by introducing an adaptive scheduling mechanism, and save the completed target large model when the preset training termination conditions are met.

[0034] In order to achieve the above-mentioned objectives, the present invention also proposes a distributed training device for large-scale models, comprising a processor, a memory, and a computer program stored in the memory, wherein the computer program is executed by the processor to implement the steps of a distributed training method for large-scale models as described in the above-mentioned embodiment.

[0035] In order to achieve the above-mentioned objectives, the present invention also proposes a computer-readable storage medium, on which a computer program is stored. The computer program is executed by a processor to implement the steps of a distributed training method for a large-scale model as described in the above-mentioned embodiment.

[0036] In order to achieve the above objectives, the present invention also proposes a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps of a distributed training method for a large-scale model as described in the above embodiment.

[0037] Beneficial effects:

[0038] The above solution builds a distributed training environment by configuring multiple computing nodes, and comprehensively applies multiple parallel strategies to divide and train training data sets and large models. At the same time, an adaptive scheduling mechanism is introduced during training to dynamically adjust tasks, thereby achieving efficient integration and utilization of hardware resources, effectively reducing dependence on hardware resources and reducing training costs. At the same time, it also significantly improves the efficiency and stability of distributed training of large models, which can meet the needs of efficient training of ultra-large-scale language models and ensure high-quality model training with limited hardware resources.

[0039] By clarifying the hardware resources of each computing node (such as the number of GPUs / TPUs and video memory capacity) and installing compatible software resources such as operating systems, drivers, distributed training frameworks, and dependent libraries on the computing nodes, a precise and complete operating environment guarantee is provided for large-scale distributed model training. This ensures that each computing node has sufficient computing power and storage capacity to handle the assigned model training tasks, while avoiding training failures caused by software version incompatibility, driver missing, and other problems. This greatly improves the efficiency and reliability of setting up the training environment, laying a solid foundation for the smooth progress of the entire distributed training.

[0040] By leveraging the strengths of various parallel strategies, efficient multi-dimensional parallel training is achieved. Data parallelism, combined with dynamic adjustment of data block size, ensures load balancing across nodes. Model parallelism and pipeline parallelism address the issue of insufficient memory on a single device and improve device utilization by rationally dividing and allocating model layers and stages. Context parallelism dynamically adjusts the attention mechanism based on contextual features to optimize model output quality. Overall, the adoption of multi-dimensional parallel strategies effectively overcomes the shortcomings of traditional parallel methods in load balancing, communication overhead, model update synchronization, and insufficient utilization of input data features. This significantly improves training speed, resource utilization, and model performance, enhances the system's adaptability and fault tolerance to large-scale model training, and makes the training process more efficient, stable, and flexible, enabling it to better adapt to the training needs of models of different scales and dynamic workloads.

[0041] By real-time monitoring of metrics such as GPU utilization, memory usage, and task completion time on compute nodes, we accurately capture the load status of each node and dynamically adjust task allocation within the parallel pipeline accordingly. Furthermore, we combine asynchronous execution of micro-batch data processing at different stages to further optimize the scheduling and execution of training tasks. This effectively avoids bottlenecks or waiting times on some nodes due to task overload, fully exploits the parallel processing capabilities of hardware resources, and minimizes idle and wasted resources, thereby further improving the efficiency and performance of the entire distributed training system and ensuring efficient training progress.

[0042] The set training termination conditions provide clear and flexible stopping criteria for the training process. Stopping training at the preset number of training rounds ensures that the model is fully trained to achieve good performance. Terminating training when the change in validation set loss falls below a preset threshold effectively avoids overtraining, preventing the model from overfitting the training set while saving unnecessary computing resources. This flexible termination condition setting helps control training costs while ensuring that the trained model has good generalization capabilities and meets the performance requirements of real-world applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0044] Figure 1 A flowchart of a distributed training method for a large-scale model provided by one embodiment of the present invention.

[0045] Figure 2 A schematic diagram of the structure of a distributed training device for large-scale models provided by one embodiment of the present invention.

[0046] The realization of the objectives of the invention, the functional features and advantages will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0047] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the invention for which protection is sought, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0048] The present invention is described in detail below with reference to the embodiments.

[0049] Reference Figure 1 The figure shows a flow chart of a distributed training method for a large-scale model provided by one embodiment of the present invention.

[0050] In this embodiment, the method includes:

[0051] S11: Configure multiple computing nodes on the current device based on the scale of the large model to be trained and the training task, and configure the hardware and software resources of each computing node to obtain a distributed training environment.

[0052] S12, using parallel strategies of different dimensions to divide the training data set and the large model to be trained to obtain corresponding multiple data blocks and multiple groups of model layers, wherein the parallel strategies include data parallelism, model parallelism, pipeline parallelism, and context parallelism;

[0053] S13, starting a training process by loading model parameters, multiple data blocks, and multiple groups of model layers onto corresponding computing nodes in the distributed training environment;

[0054] S14, during the training process, monitor the operating status of each computing node, dynamically adjust the training tasks of the corresponding computing node by introducing an adaptive scheduling mechanism, and save the completed target large model when the preset training termination conditions are met.

[0055] This embodiment effectively integrates the advantages of multi-dimensional parallelism to achieve efficient and stable model training under limited hardware resources, significantly improving training efficiency and resource utilization, while reducing manual intervention and scheduling complexity. It is suitable for distributed training scenarios of ultra-large-scale language models.

[0056] Furthermore, in step S12, a parallel strategy is dynamically selected and combined based on various factors (including the scale of the large model, hardware resource configuration, task type, and input features), including:

[0057] Select a parallel strategy based on the size of the large model to be trained. This includes: when the number of parameters of the large model to be trained is less than 1 billion, only enable data parallelism; when the number of parameters is between 1 billion and 10 billion, enable data parallelism and model parallelism; when the number of parameters exceeds 10 billion, enable data parallelism, model parallelism, and pipeline parallelism; when the input sequence length is long and there is a significant difference in context complexity, further enable context parallelism;

[0058] Select parallel strategies based on hardware resource configuration, including: When a large number of available GPUs / TPUs are available and sufficient video memory is available, prioritize increasing the number of data parallel groups; when video memory on a single card is limited, prioritize enabling model parallelism or pipeline parallelism to reduce single-device memory usage; when communication bandwidth between computing nodes is high, increase the number of data parallel groups; for heterogeneous computing environments (such as mixed deployments of different GPU models), combine context parallelism to assign high-complexity tasks to high-performance GPUs for load balancing;

[0059] The parallel strategy is selected based on the task type and the characteristics of the input data (the characteristics of the input data include the length of the input data, the number of tokens contained, the semantic complexity, the amount of information, etc.). For example: for tasks such as short-sequence text classification, if the context complexity is low, the context parallelism can be disabled or the granularity can be reduced; for long text generation tasks, due to the strong context dependence and high information density, context parallelism is enabled and the attention mechanism is dynamically adjusted; in multimodal tasks, the context parallel logic can be extended to perform complexity evaluation and scheduling optimization for modal segments such as images and audio.

[0060] Furthermore, in step S12, each parallel strategy realizes the integration and coordination of the results of the divisions in different dimensions according to the hierarchical division and the existing communication mechanism, including:

[0061] Layered division: The outermost layer of the large model uses data parallelism to copy training tasks to multiple device groups (computing nodes), with each device group processing different micro-batches of data. Based on data parallelism, the middle layer uses pipeline parallelism to divide the large model to be trained into multiple stages that can be executed continuously. Each stage is completed collaboratively by a group of GPUs, and parallel computing between different stages is achieved through a micro-batch scheduling mechanism. Within each pipeline stage, model parallelism is further adopted to assign different layers of the large model to different GPUs for execution, thereby adapting to the memory capacity limitations of a single graphics card and improving the scalability of the large model. In addition, based on the above parallel strategy, context parallelism is introduced to dynamically adjust the attention mechanism according to the complexity of the input sequence. In combination with the current load status of each GPU, high-complexity tasks are assigned to GPUs with stronger computing power, and low-complexity tasks are assigned to ordinary GPUs, thereby achieving efficient resource utilization.

[0062] The coordination mechanism includes: ensuring the coordinated operation of the above-mentioned parallel strategies at all levels through existing communication mechanisms. For example, at the data parallel level, task distribution is completed by using data synchronization operations across device groups; at the pipeline parallel level, efficient transmission of intermediate results is achieved through point-to-point communication or inter-stage transmission mechanisms; at the model parallel level, parameter sharding and gradient aggregation techniques are used to ensure computational consistency between GPUs; for context parallelism, task allocation is dynamically adjusted in combination with the local load feedback mechanism, and the fusion and execution of multi-dimensional partitioning results can be achieved without introducing additional scheduling modules.

[0063] Furthermore, in step S11, configuring the hardware resources and software resources of each computing node includes:

[0064] The configuration of the hardware resources includes configuring the number of GPUs / TPUs and the memory capacity of each computing node;

[0065] The configuration of the software resources includes installing a compatible operating system, corresponding GPU / TPU drivers, a distributed training framework, and compatible dependent libraries on each computing node.

[0066] In this example, the number of computing nodes required and the hardware configuration of each computing node (such as the number of GPUs / TPUs, memory size, etc.) are determined based on the scale of the large model and the requirements of the training task. For example, for an ultra-large-scale language model, hundreds of computing nodes equipped with multiple high-end GPUs may be required. The configuration of software resources includes:

[0067] Operating system and driver installation: Install a compatible operating system (such as Linux) on each computing node and install the corresponding GPU / TPU drivers (such as CUDA, cuDNN);

[0068] Deep learning framework deployment: Select and install a suitable large-scale distributed training framework, such as TensorFlow, PyTorch, or a specialized distributed training library such as DeepSpeed;

[0069] Dependency library installation: Install all necessary dependency libraries with compatible versions, such as NumPy, SciPy, Matplotlib and other scientific computing libraries.

[0070] Furthermore, in step S12, the parallel strategies of different dimensions are used to divide the training data set and the large model to be trained, including:

[0071] S12-1, using a data parallel strategy to divide the training data set into multiple data blocks according to the number of computing nodes, and dynamically adjusting the size of the data blocks according to the computing power and storage capacity of each computing node;

[0072] S12-2, dividing the large model to be trained into layers using a model parallel strategy to obtain multiple groups of model layers, and assigning each group of model layers to a corresponding computing node;

[0073] S12-3, using a pipeline parallel strategy to divide the large model to be trained into multiple consecutive stages, and assigning them to corresponding computing nodes for sequential execution;

[0074] S12-4 adopts a context-parallel parallel strategy to segment the input sequence according to its contextual features, assigns each segment to a matching computing node, and dynamically adjusts the attention mechanism.

[0075] In this embodiment, the parallel strategies of different dimensions used for partitioning the training data set and the large model to be trained are specifically as follows:

[0076] Data parallelism: Intelligently divides the training dataset according to the number of computing nodes, ensuring that each computing node can obtain appropriate data blocks. The size of the data blocks is dynamically adjusted according to the computing power and storage capacity of each computing node to ensure load balancing.

[0077] Model parallelism: Based on the structural characteristics of the large model to be trained, multiple groups of model layers are obtained, allowing even very large models to be trained on limited hardware resources. For example, for models with hierarchical structures (such as large Transformer models), simple layer-by-layer segmentation is used.

[0078] Pipeline parallelism: The large model to be trained is divided into multiple consecutive stages and executed sequentially on different computing nodes. Micro-batch processing is used to enable different stages to be executed in parallel. An asynchronous update mechanism is introduced to allow some stages to enter the calculation of the next micro-batch in advance, thereby reducing "bubble" idle time and improving hardware utilization.

[0079] Contextual Parallelism: Dynamically adjusts the attention mechanism and generation strategy (a strategy that dynamically adjusts the model processing method based on the contextual features of the input sequence) based on the contextual features of the input sequence (referring to the data blocks after the training dataset is divided; that is, the token sequence with contextual semantic structure actually received and processed by the model during training or inference, which is the result of further segmentation and encoding of the input data). Combined with the evaluation feedback mechanism of the model to be trained (such as the preference alignment capability of Llama-3.1), it optimizes output quality and improves the consistency and accuracy of reasoning and generation.

[0080] Furthermore, in step S12-2, the large model to be trained is divided into layers by the parallel strategy of model parallelism to obtain multiple groups of model layers, and each group of model layers is assigned to a corresponding computing node, including:

[0081] S12-2-1, determining the number of segments of the large model to be trained according to the number of computing nodes, obtaining multiple stages, and allocating the continuous model layers of each stage to the corresponding computing nodes;

[0082] S12-2-2, during forward propagation, the input sequence is passed through each computing node in sequence;

[0083] S12-2-3, during back propagation, the gradient is transferred layer by layer from the last calculation node to the previous calculation node and the weight is updated.

[0084] In this embodiment, the large model to be trained is a Transformer model. The process of layering the large model to be trained is to assign different layers in the Transformer model to corresponding computing nodes to achieve model parallelism, specifically:

[0085] Determine the splitting strategy: Based on the number of available compute nodes, determine how many parts the Transformer model should be divided into; for example, if there are 4 GPUs, the model can be divided into 4 stages;

[0086] Divide the model layers: assign consecutive layers of a stage to a specific GPU. For example, if the Transformer model has 12 layers, it can be divided into 4 stages, each containing 3 layers. Then, GPU1 is assigned to layers 1-3, GPU2 is assigned to layers 4-6, GPU3 is assigned to layers 7-9, and GPU4 is assigned to layers 10-12. Each GPU is responsible for processing its assigned layers.

[0087] Forward propagation process: During training, the input sequence is first sent to the first GPU. After the first GPU processes its part, it passes the output to the next GPU, and so on, until the last GPU completes the entire forward propagation process;

[0088] During backpropagation, gradients are calculated starting from the last GPU and passed forward layer by layer. Each GPU updates its weights based on the received gradients. For example, if there are four GPUs, gradients are calculated starting from GPU4 and passed back to GPU3. GPU3 uses the received gradients to update its weights and continues to pass the gradients forward.

[0089] Furthermore, in step S12-4, the context-parallel strategy is adopted to segment the input sequence according to the context features of the input sequence, assign each segment to a matching computing node, and dynamically adjust the attention mechanism, including:

[0090] S12-4-1, extract the input sequence according to the length of the input sequence and the attention weight distribution to obtain context features;

[0091] S12-4-2, based on the context features, the input sequence is segmented according to the preset window size, and segments corresponding to different context complexities are assigned to matching computing nodes according to the context complexity of each segment and the load status of the current corresponding computing node;

[0092] S12-4-3, dynamically adjust the attention mechanism according to the context complexity corresponding to different fragments. After each computing node completes the processing of the current fragment, the context complexity information is fed back to the scheduler, and the scheduler optimizes the task division and resource scheduling of the next micro-batch based on the context complexity information.

[0093] In this embodiment, a context parallel approach is adopted, specifically including:

[0094] Extraction and analysis of contextual features: Using an evaluation method based on input sequence length and attention weight distribution, we extract contextual features, including indicators of contextual features such as input sequence length, attention weight concentration, and context complexity.

[0095] Context Segmentation and GPU Allocation: The input sequence is segmented into fixed windows and allocated based on the context complexity of each segment (evaluated by the input sequence length and attention weight distribution) and the current load status of each computing node (such as video memory usage and computing power). That is, the high context complexity is allocated to the GPU with strong computing power.

[0096] Dynamic adjustment of the attention mechanism: Based on the evaluation results of context complexity, the attention mechanism configuration within the model is dynamically adjusted. For fragments with high context complexity, the number of attention heads is increased or the attention window is expanded to improve the focus on key information; for fragments with low context complexity, the number of attention heads is reduced or the window range is narrowed to reduce computational overhead. Each GPU independently executes the attention configuration that matches its assigned content, improving overall parallel efficiency.

[0097] Asynchronous feedback and rescheduling mechanism: After completing the current batch processing, each GPU feeds back context complexity information to the scheduler, which then optimizes the task division and resource scheduling for the next micro-batch. By introducing an asynchronous update mechanism, some GPUs are allowed to enter the next micro-batch calculation in advance, reducing pipeline idle time and improving hardware utilization.

[0098] Furthermore, in step S14, the operating status of each computing node is monitored, and the training task of the corresponding computing node is dynamically adjusted by introducing an adaptive scheduling mechanism, including:

[0099] S14-1, obtain the load status of each computing node by monitoring the GPU utilization, memory usage, and task completion time of each computing node;

[0100] S14-2 dynamically adjusts the training task allocation of each stage in the pipeline parallelism according to the load status of the corresponding computing node, and performs micro-batch data processing at different stages in an asynchronous manner.

[0101] In this embodiment, an adaptive scheduling mechanism is introduced to dynamically adjust the training tasks of the corresponding computing nodes. Specifically, the load status of each node is obtained by monitoring the GPU utilization, memory usage, task completion time and other indicators of each computing node. When it is detected that a computing node is overloaded for a long time, some training tasks are migrated to nodes with lighter loads to achieve global load balancing. The allocation of training tasks in each stage of the parallel pipeline is dynamically adjusted according to the load status of the computing nodes to avoid some nodes being idle or overloaded for a long time. Micro-batch data can be processed asynchronously at different stages to further improve hardware utilization and reduce performance loss caused by the "bubble" effect.

[0102] Furthermore, the training termination condition includes reaching a preset training round or the loss value change of the validation set is less than a preset threshold.

[0103] In this embodiment, the training termination condition includes the configured number of training rounds (Epoch) or the convergence index on the validation set (such as the Loss change value). If the termination condition is not met, return to step S12, re-divide the training data set and the large model to be trained, and start a new cycle of training process.

[0104] This embodiment further illustrates the implementation of distributed training of the Qwen2.5-72B-Instruct model of the Transformer architecture by taking an example:

[0105] (1) Build a distributed training environment, including preparing computing nodes and configuring necessary software and hardware resources. Specifically, we configured 100 computing nodes, each equipped with 8 NVIDIA A100 GPUs (each with 80GB of video memory), for a total of 800 GPUs, to train a Transformer-based language model with 72 billion parameters. This model consists of 60 Transformer blocks, each containing a Self-Attention layer, a Feed Forward layer, and a Layer Normalization layer. Because a single GPU cannot support such a large-scale model, a multi-GPU collaborative model parallelization strategy must be adopted. Ubuntu 20.04 LTS was installed on each computing node, along with necessary GPU drivers such as CUDA 11.8 and cuDNN 8.6. PyTorch was selected as the primary deep learning training framework, and the DeepSpeed ​​library was integrated to optimize distributed training performance. Ensure version compatibility of scientific computing libraries such as NumPy 1.23.0, SciPy 1.9.0, and other libraries that may be required, such as Transformers 4.20.0.

[0106] (2) The training dataset Wikipedia and the large model to be trained Qwen2.5-72B-Instruct are divided using a multi-dimensional parallel strategy. Specifically:

[0107] (2-1) Data Parallelism: Data cleaning of the Wikipedia corpus was performed. For 800 GPUs, the entire Wikipedia corpus was intelligently partitioned according to the number of computing nodes, with each GPU processing approximately 0.125% of the data volume. The data block size was dynamically adjusted based on the actual computing power and storage capacity of each computing node to ensure load balancing.

[0108] (2-2) Model Parallelism: The Qwen2.5-72B-Instruct model is a Transformer-based language model with 72 billion parameters. Since the Transformer model has a distinct hierarchical structure, a simple hierarchical segmentation method is used, including:

[0109] Determine the segmentation strategy: Divide the model into multiple stages to deploy on different GPUs of computing nodes. For example, based on the available 800 GPUs and 60 Transformer Blocks, divide the entire model into 60 segments. Each stage corresponds to a Transformer Block and is assigned to a separate GPU for processing.

[0110] Divide the model layers and assign GPUs: A set of consecutive model layers is packaged into a stage and assigned to a specific GPU for execution. According to the partitioning strategy, GPU1 is assigned Transformer Block 1, GPU2 is assigned Transformer Block 2, and so on. GPU60 is assigned Transformer Block 60. Each GPU is responsible for processing its assigned layers.

[0111] Forward propagation process: During training, the input sequence is first sent to GPU1. After GPU1 processes Transformer Block 1, it passes the output to GPU2, and so on, until the last GPU60 completes all forward calculations;

[0112] Backward propagation process: The gradient is calculated starting from GPU60 and passed forward layer by layer to GPU1; after receiving the reverse gradient, each GPU updates the model parameters it manages and continues to pass the gradient information forward to complete the complete parameter update process.

[0113] (2-3) Pipeline Parallelism: After completing the model layer segmentation, the Qwen2.5-72B-Instruct model is divided into 60 Transformer Blocks and assigned to GPU1 to GPU60 for processing. To further improve training throughput and hardware utilization, this embodiment uses the remaining 740 GPUs (GPU61 to GPU800) for pipeline parallelism, specifically:

[0114] The 60 Transformer blocks are divided into 10 stages, each stage contains 6 blocks, that is, stage 1 contains blocks 1 to 6, stage 2 contains blocks 7 to 12, and so on. Stage 10 contains blocks 55 to 60.

[0115] The input data (i.e., a single training batch) is further divided into multiple micro-batches for processing. Each micro-batch enters each pipeline stage in turn as an input sequence. Each stage is executed in parallel by a group of GPUs, i.e., GPU61 to GPU70 process stage 1, GPU71 to GPU80 process stage 2, and so on. GPU151 to GPU160 process stage 10.

[0116] After completing the forward calculation of the current batch in a certain stage, the processing of the next batch begins immediately without waiting for the completion of other stages. In the reverse propagation stage, the same asynchronous execution method is adopted. Once a stage receives the gradient from the previous layer, the reverse calculation of the current stage can be started immediately. This asynchronous update mechanism can reduce idle time.

[0117] (2-4) Contextual parallelism: Dynamically adjust the calculation strategy of attention weights according to the length of the input data to optimize the output quality; for example, when processing an input sequence (a token sequence with a contextual semantic structure), give priority to processing the parts with high attention weights to improve efficiency.

[0118] (3) Load the corresponding data blocks and model layers on all computing nodes and start the forward propagation and backpropagation processes to start the training process.

[0119] (4) During the training process, the operating status of each computing node (such as GPU utilization and memory usage) is monitored, and the training tasks of the corresponding computing node are dynamically adjusted. The training is considered to have converged by configuring the number of training epochs to 30, or when the change in loss on the validation set is less than a preset threshold (such as 0.0001). At this time, when the training reaches the termination condition, the trained target model is saved.

[0120] This embodiment effectively reduces the memory pressure of a single device by combining the configuration of multiple computing nodes, the division of data sets and models, and multiple parallel strategies. It also optimizes task allocation through an adaptive scheduling mechanism, avoids synchronous waiting of computing nodes, and improves overall computing efficiency. Compared with traditional data parallel methods, it significantly reduces dependence on the number of GPUs / TPUs, reduces training costs, and solves problems such as high memory usage and large communication overhead. However, by adopting multiple parallel strategies, combined with asynchronous micro-batch processing and hierarchical communication management, it overcomes the problems of uneven parallel load and pipeline bubble effect of traditional models; effectively improves the scalability and fault tolerance of the system, ensuring efficient and stable operation under models of different scales and dynamic workloads. In addition, through the contextual parallel mechanism, the attention mechanism and generation strategy are dynamically adjusted, and the output quality is optimized in combination with model evaluation feedback. This not only improves the understanding and generation capabilities of the model, but also enhances the consistency of the output during training, improves the expressiveness of the final model, and solves the problem that the existing technology lacks effective utilization of input context features, resulting in unstable or inconsistent model generation results.

[0121] Reference Figure 2 Shown is a structural diagram of a distributed training device for a large-scale model provided by an embodiment of the present invention.

[0122] In this embodiment, the device 20 includes:

[0123] Configuration unit 21, configured to configure multiple computing nodes on the current device according to the scale of the large model to be trained and the training task, and configure the hardware and software resources of each computing node to obtain a distributed training environment;

[0124] A partitioning unit 22 is configured to partition the training data set and the large model to be trained using parallel strategies of different dimensions to obtain corresponding multiple data blocks and multiple groups of model layers, wherein the parallel strategies include data parallelism, model parallelism, pipeline parallelism, and context parallelism;

[0125] A training unit 23 is configured to start a training process by loading model parameters, multiple data blocks, and multiple sets of model layers onto corresponding computing nodes in the distributed training environment;

[0126] The monitoring unit 24 is used to monitor the operating status of each computing node during the training process, dynamically adjust the training tasks of the corresponding computing nodes by introducing an adaptive scheduling mechanism, and save the trained target large model when the preset training termination conditions are met.

[0127] Each unit module of the device 20 can respectively execute the corresponding steps in the above method embodiment, so each unit module will not be described in detail here. Please refer to the description of the corresponding steps above for details.

[0128] The embodiment of the present invention also provides a large-scale model distributed training device, which includes the large-scale model distributed training device as described above, wherein the large-scale model distributed training device can adopt Figure 2 The structure of the embodiment can be executed accordingly. Figure 1 The technical solution of the method embodiment shown has similar implementation principles and technical effects. For details, please refer to the relevant records in the above embodiments and will not be repeated here.

[0129] The device includes: a mobile phone, digital camera, tablet computer, or other device with a camera function, or a device with an image processing function, or a device with an image display function. The device may include components such as a memory, a processor, an input unit, a display unit, and a power supply.

[0130] Among them, the memory can be used to store software programs and modules, and the processor executes various functional applications and data processing by running the software programs and modules stored in the memory. The memory may mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, an application required for at least one function (such as an image playback function, etc.), etc.; the data storage area can store data created according to the use of the device, etc. In addition, the memory may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage device. Accordingly, the memory may also include a memory controller to provide the processor and the input unit with access to the memory.

[0131] The input unit can be used to receive input digital, character, or image information, and generate keyboard, mouse, joystick, optical, or trackball signal input related to user settings and function control. Specifically, the input unit of this embodiment can include not only a camera, but also a touch-sensitive surface (such as a touch display) and other input devices.

[0132] The display unit can be used to display information input by the user or information provided to the user and various graphical user interfaces of the device, which can be composed of graphics, text, icons, videos and any combination thereof. The display unit may include a display panel. Optionally, the display panel can be configured in the form of an LCD (Liquid Crystal Display), an OLED (Organic Light-Emitting Diode), etc. Furthermore, the touch-sensitive surface can cover the display panel. When the touch-sensitive surface detects a touch operation on or near it, it is transmitted to the processor to determine the type of touch event. The processor then provides a corresponding visual output on the display panel based on the type of touch event.

[0133] The embodiment of the present invention further provides a computer-readable storage medium, which may be a computer-readable storage medium included in the memory in the above embodiment; or a computer-readable storage medium that exists independently and is not assembled into a device. The computer-readable storage medium stores at least one instruction, which is loaded and executed by a processor to implement Figure 1 The distributed training method of the large-scale model shown. The computer-readable storage medium can be a read-only memory, a disk or an optical disk, etc.

[0134] The embodiment of the present invention further provides a computer program product, including a computer program / instruction, which is loaded and executed by a processor to implement Figure 1 A distributed training method for large-scale models is shown.

[0135] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. For similar or identical parts between the various embodiments, reference can be made to each other. For the apparatus embodiments, device embodiments, and storage medium embodiments, since they are generally similar to the method embodiments, their descriptions are relatively simple. For relevant parts, reference can be made to the descriptions of the method embodiments.

[0136] Furthermore, in this document, the terms "comprises," "comprising," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or apparatus that includes a list of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not preclude the presence of additional identical elements in the process, method, article, or apparatus that includes the element.

[0137] While the foregoing description shows and describes preferred embodiments of the present invention, it should be understood that the present invention is not limited to the forms disclosed herein and should not be construed as excluding other embodiments. Rather, the present invention can be used in various other combinations, modifications, and environments, and can be modified within the scope of the present invention by the teachings herein or by techniques or knowledge in the relevant art. Modifications and variations made by those skilled in the art without departing from the spirit and scope of the present invention are intended to be within the scope of the appended claims.

Claims

1. A distributed training method for large-scale models, characterized in that: The method comprises: Configure multiple computing nodes on the current device based on the scale of the large model to be trained and the training task, and configure the hardware and software resources of each computing node to obtain a distributed training environment. Using parallel strategies of different dimensions to divide the training data set and the large model to be trained to obtain corresponding multiple data blocks and multiple groups of model layers, wherein the parallel strategies include data parallelism, model parallelism, pipeline parallelism, and context parallelism; The method of dividing the training data set and the large model to be trained by adopting parallel strategies of different dimensions includes: A data parallel strategy is used to divide the training data set into multiple data blocks according to the number of computing nodes, and the size of the data blocks is dynamically adjusted according to the computing power and storage capacity of each computing node; The large model to be trained is divided into layers by a model parallel strategy to obtain multiple groups of model layers, and each group of model layers is assigned to a corresponding computing node; A pipeline parallel strategy is used to divide the large model to be trained into multiple consecutive stages, and the stages are assigned to the corresponding computing nodes for sequential execution. Adopting a contextual parallel strategy, the input sequence is segmented according to its contextual features, each segment is assigned to a matching computation node, and the attention mechanism is dynamically adjusted. The context-parallel strategy is to segment the input sequence according to its contextual features, assign each segment to a matching computing node, and dynamically adjust the attention mechanism, including: Extract the input sequence according to its length and attention weight distribution to obtain contextual features; Based on context features, the input sequence is segmented according to the preset window size, and segments corresponding to different context complexities are assigned to matching computing nodes based on the context complexity of each segment and the load status of the current corresponding computing node; The attention mechanism is dynamically adjusted based on the contextual complexity of different segments. After each computing node completes processing the current segment, the contextual complexity information is fed back to the scheduler, which then optimizes the task division and resource scheduling for the next micro-batch based on the contextual complexity information. Initiate a training process by loading model parameters, multiple data blocks, and multiple sets of model layers onto corresponding computing nodes in the distributed training environment; During the training process, the operating status of each computing node is monitored, and the training tasks of the corresponding computing nodes are dynamically adjusted by introducing an adaptive scheduling mechanism. When the preset training termination conditions are met, the trained target large model is saved.

2. The distributed training method for a large-scale model according to claim 1, characterized in that: The configuration of the hardware resources and software resources of each computing node includes: The configuration of the hardware resources includes configuring the number of GPUs / TPUs and the memory capacity of each computing node; The configuration of the software resources includes installing a compatible operating system, corresponding GPU / TPU drivers, a distributed training framework, and compatible dependent libraries on each computing node.

3. The distributed training method for a large-scale model according to claim 1, characterized in that: The method of dividing the large model to be trained into layers by the parallel strategy of model parallelism to obtain multiple groups of model layers, and allocating each group of model layers to a corresponding computing node includes: Determine the number of segments of the large model to be trained according to the number of computing nodes, obtain multiple stages, and assign the continuous model layers of each stage to the corresponding computing nodes; During forward propagation, the input sequence is passed through each computing node in sequence; During back propagation, the gradient is transferred layer by layer from the last calculation node to the previous calculation node and the weights are updated.

4. The distributed training method for a large-scale model according to claim 1, characterized in that: The operating status of each computing node is monitored, and the training tasks of the corresponding computing node are dynamically adjusted by introducing an adaptive scheduling mechanism, including: Obtain the load status of each computing node by monitoring the GPU utilization, memory usage, and task completion time of each computing node; The training task allocation of each stage in the pipeline parallelism is dynamically adjusted according to the load status of the corresponding computing node, and micro-batch data processing at different stages is performed asynchronously.

5. The distributed training method for a large-scale model according to claim 1, characterized in that: The training termination conditions include reaching a preset training round or the change in the loss value of the validation set is less than a preset threshold.

6. A distributed training device for a large-scale model, used to execute a distributed training method for a large-scale model as claimed in any one of claims 1 to 5; characterized in that: The device comprises: The configuration unit is used to configure multiple computing nodes on the current device according to the scale of the large model to be trained and the training task, and configure the hardware and software resources of each computing node to obtain the established distributed training environment; A partitioning unit is configured to partition the training data set and the large model to be trained using parallel strategies of different dimensions to obtain corresponding multiple data blocks and multiple groups of model layers, wherein the parallel strategies include data parallelism, model parallelism, pipeline parallelism, and context parallelism; A training unit, configured to start a training process by loading model parameters, multiple data blocks, and multiple groups of model layers onto corresponding computing nodes in the distributed training environment; The monitoring unit is used to monitor the operating status of each computing node during the training process, dynamically adjust the training tasks of the corresponding computing nodes by introducing an adaptive scheduling mechanism, and save the completed target large model when the preset training termination conditions are met.

7. A distributed training device for large-scale models, characterized in that: It includes a processor, a memory and a computer program stored in the memory, and the computer program is executed by the processor to implement the steps of a distributed training method for a large-scale model as described in any one of claims 1 to 5.

8. A computer program product, characterized in that It includes a computer program / instruction, which, when executed by a processor, implements the steps of a distributed training method for a large-scale model as described in any one of claims 1 to 5.

Images