Distributed training method, device and equipment for large-scale model

Abstract

The invention discloses a distributed training method, device and equipment for a large-scale model, and the method comprises the steps: configuring a plurality of calculation nodes on current equipment according to the scale of a to-be-trained large model and a training task, and configuring the hardware resources and software resources of each calculation node, and obtaining a constructed distributed training environment; dividing a training data set and the to-be-trained large model by adopting parallel strategies of different dimensions to obtain a plurality of corresponding data blocks and a plurality of groups of model layers; starting a training process by loading model parameters, a plurality of data blocks and a plurality of groups of model layers to corresponding computing nodes in the distributed training environment; in the training process, the running state of each computing node is monitored, the training task of the corresponding computing node is dynamically adjusted by introducing an adaptive scheduling mechanism, and when a preset training termination condition is met, the trained target large model is stored. And the training requirements of high efficiency, stability and low cost can be met.

Description

Technical Field

[0001] The present invention relates to the technical field of model training, and in particular, to a distributed training method, device, and equipment for large-scale models. Background Art

[0002] With the rapid development of deep learning technology, large language models (LLMs) have shown great application potential in many fields. Their parameter scale has been continuously expanding, moving from billions to trillions of levels, which poses unprecedented challenges to model training. On the one hand, the time and computing resources required for training have increased sharply, bringing a heavy burden to hardware facilities; on the other hand, existing distributed training methods gradually expose many irreconcilable problems when dealing with such ultra-large-scale models. For example, some existing distributed training strategies are often limited by the memory capacity of a single device when dealing with large models, resulting in incomplete model deployment. And simple multi-device expansion will cause problems such as excessive communication overhead and load imbalance, seriously restricting the improvement of training efficiency. In addition, the existing methods do not utilize the input data features finely enough, and it is difficult to achieve efficient and accurate parallel processing and model optimization in the face of long texts or complex context scenarios, resulting in difficulty in ensuring the stability and consistency of model generation results. In summary, the existing methods have many deficiencies in meeting the efficient training requirements of large language models and are difficult to adapt to the high-efficiency requirements of current large-scale model training. Summary of the Invention

[0003] In view of this, the purpose of the present invention is to propose a distributed training method, device, and equipment for large-scale models, aiming to solve the problems of high hardware resource costs, load imbalance, low efficiency, etc. in the existing distributed training methods for large-scale models, which cannot meet the training requirements of high efficiency, stability, and low cost.

[0004] To achieve the above purpose, the present invention provides a distributed training method for large-scale models, and the method includes: Configure multiple computing nodes on the current device according to the scale of the large model to be trained and the training task, and configure the hardware resources and software resources of each computing node to obtain the built distributed training environment; Divide the training data set and the large model to be trained using parallel strategies in different dimensions to obtain corresponding multiple data blocks and multiple groups of model layers, where the parallel strategies include data parallelism, model parallelism, pipeline parallelism, and context parallelism; Start the training process by loading the model parameters, multiple data blocks, and multiple groups of model layers onto the corresponding computing nodes in the distributed training environment; During the training process, monitor the running status of each computing node, dynamically adjust the training tasks of the corresponding computing node by introducing an adaptive scheduling mechanism, and save the trained target large model when the preset training termination condition is met.

[0005] Preferably, the configuration of the hardware resources and software resources for each computing node includes: The configuration of the hardware resources includes configuring the number of GPUs / TPUs and the video memory capacity for each computing node; The configuration of the software resources includes installing a compatible operating system, the corresponding GPU / TPU driver, a distributed training framework, and compatible dependency libraries on each computing node.

[0006] Preferably, the use of parallel strategies in different dimensions to partition the training dataset and the large model to be trained includes: Adopt a data parallel strategy to divide the training dataset into multiple data blocks according to the number of computing nodes, and dynamically adjust the data block size according to the computing power and storage capacity of each computing node; Perform hierarchical segmentation on the large model to be trained through a model parallel strategy to obtain multiple groups of model layers, and allocate each group of model layers to the corresponding computing node; Adopt a pipeline parallel strategy to divide the large model to be trained into multiple consecutive stages and allocate them to the corresponding computing nodes for sequential execution; Adopt a context parallel strategy to segment the input sequence according to the context features of the input sequence, allocate each segment to the matching computing node, and dynamically adjust the attention mechanism.

[0007] Preferably, the performing hierarchical segmentation on the large model to be trained through a model parallel strategy to obtain multiple groups of model layers, and allocating each group of model layers to the corresponding computing node includes: Determine the number of segmentation segments of the large model to be trained according to the number of computing nodes to obtain multiple stages, and allocate the consecutive model layers of each stage to the corresponding computing node; During forward propagation, the input sequence is sequentially passed through each computing node in order; During backpropagation, the gradient is passed layer by layer from the last computing node to the previous computing node to update the weights.

[0008] Preferably, the adopting a context parallel strategy to segment the input sequence according to the context features of the input sequence, allocate each segment to the matching computing node, and dynamically adjust the attention mechanism includes: Extract the input sequence according to the length of the input sequence and the attention weight distribution to obtain context features; Based on context features, segment the input sequence according to a preset window size, and allocate segments corresponding to different context complexities to matching computing nodes according to the context complexity of each segment and the load status of the current corresponding computing node; Dynamically adjust the attention mechanism according to the context complexity corresponding to different segments. After each computing node finishes processing the current segment, feedback the information of the context complexity to the scheduler, and optimize the task division and resource scheduling of the next micro-batch by the scheduler according to the information of the context complexity.

[0009] Preferably, monitoring the running state of each computing node and dynamically adjusting the training tasks of the corresponding computing node by introducing an adaptive scheduling mechanism includes: Obtain the load status of each computing node by monitoring the GPU utilization rate, memory occupancy, and task completion time of each computing node; Dynamically adjust the training task allocation of each stage in pipeline parallelism according to the load status corresponding to the computing node, and perform micro-batch data processing of different stages asynchronously.

[0010] Preferably, the training termination condition includes reaching a preset number of training rounds or the change amount of the loss value of the validation set is less than a preset threshold.

[0011] To achieve the above object, the present invention also provides a distributed training device for large-scale models, and the device includes: A configuration unit for configuring multiple computing nodes on the current device according to the scale of the large model to be trained and the training task, and configuring the hardware resources and software resources of each computing node to obtain a built distributed training environment; A division unit for dividing the training data set and the large model to be trained by using parallel strategies of different dimensions to obtain corresponding multiple data blocks and multiple groups of model layers, wherein the parallel strategies include data parallelism, model parallelism, pipeline parallelism, and context parallelism; A training unit for starting a training process by loading model parameters, multiple data blocks, and multiple groups of model layers onto the corresponding computing nodes in the distributed training environment; A monitoring unit for monitoring the running state of each computing node during the training process, dynamically adjusting the training tasks of the corresponding computing node by introducing an adaptive scheduling mechanism, and saving the target large model that has completed training when the preset training termination condition is met.

[0012] To achieve the above object, the present invention further provides a distributed training device for large-scale models, including a processor, a memory, and a computer program stored in the memory, and the computer program is executed by the processor to implement the steps of a distributed training method for large-scale models as described in the above embodiments.

[0013] To achieve the above object, the present invention further provides a computer-readable storage medium, on which a computer program is stored, and the computer program is executed by a processor to implement the steps of a distributed training method for large-scale models as described in the above embodiments.

[0014] To achieve the above object, the present invention further provides a computer program product, including a computer program / instructions, and when the computer program / instructions are executed by a processor, the steps of a distributed training method for large-scale models as described in the above embodiments are implemented.

[0015] Beneficial effects: In the above solution, a distributed training environment is built by configuring multiple computing nodes, and a variety of parallel strategies are comprehensively used to partition and train the training data set and the large model. At the same time, an adaptive scheduling mechanism is introduced during training to dynamically adjust tasks, realizing the efficient integration and utilization of hardware resources, effectively reducing the dependence on hardware resources, reducing the training cost. At the same time, the efficiency and stability of large model distributed training are significantly improved, which can meet the requirements of efficient training of ultra-large-scale language models and ensure high-quality model training with limited hardware resources.

[0016] By clarifying the hardware resources of each computing node (such as the number of GPUs / TPUs and the video memory capacity), and installing software resources such as a compatible operating system, driver program, distributed training framework, and dependent libraries on the computing node, it provides accurate and perfect operation environment guarantee for large-scale model distributed training, can ensure that each computing node has sufficient computing power and storage capacity to process the assigned model training tasks, and at the same time avoids training failures caused by problems such as software version incompatibility and driver missing, greatly improving the building efficiency and reliability of the training environment and laying a solid foundation for the smooth progress of the entire distributed training.

[0017] By coordinating multiple parallel strategies with each other and giving full play to their respective advantages, multi-dimensional efficient parallel training is achieved. Among them, data parallelism combined with dynamic adjustment of the data block size ensures load balancing among nodes; model parallelism and pipeline parallelism solve the problem of insufficient memory on a single device and improve device utilization by reasonably partitioning and allocating model layers and stages; context parallelism dynamically adjusts the attention mechanism according to context features and optimizes the model output quality. Overall, by adopting multi-dimensional parallel strategies, the defects of traditional parallel methods in load balancing, communication overhead, model update synchronization, and insufficient utilization of input data features are effectively overcome, significantly improving the training speed, resource utilization, and model performance, enhancing the system's adaptability and fault tolerance for large-scale model training, making the training process more efficient, stable, and flexible, and enabling better adaptation to the training requirements of different-scale models and dynamic workloads.

[0018] By real-time monitoring metrics such as the GPU utilization, memory occupancy, and task completion time of computing nodes, the load status of each node is accurately obtained, and based on this, the task allocation in pipeline parallelism is dynamically adjusted. Additionally, by combining asynchronous execution of micro-batch data processing in different stages, the scheduling and execution of training tasks are further optimized. This can effectively avoid bottlenecks or waiting phenomena caused by task overload on some nodes, fully exploit the parallel processing capabilities of hardware resources, and minimize resource idleness and waste, thereby further improving the efficiency and performance of the entire distributed training system and ensuring the efficient progress of the training process.

[0019] The set training termination conditions provide clear and flexible stop criteria for the training process. Stopping the training when the preset number of training epochs is reached can ensure that the model is fully trained to obtain good performance; while terminating the training based on the change in the validation set loss value being less than the preset threshold can effectively avoid overfitting, prevent the model from overfitting on the training set, and save unnecessary computational resource consumption. Through the flexible setting of termination conditions, it helps to control the training cost and ensure that the trained model has good generalization ability to meet the required performance of practical applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the following-described drawings are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0021] Figure 1 It is a flowchart showing a distributed training method for a large-scale model provided by an embodiment of the present invention.

[0022] Figure 2 The structural schematic diagram of a distributed training device for a large-scale model provided by an embodiment of the present invention.

[0023] The realization of the invention purpose, functional features and advantages will be further described with reference to the embodiments and the accompanying drawings. Specific embodiments

[0024] To make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope of protection of the present invention. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the claimed invention, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope of protection of the present invention.

[0025] The content of the present invention will be elaborated in detail below with reference to the embodiments.

[0026] Refer to Figure 1 The flowchart of a distributed training method for a large-scale model provided by an embodiment of the present invention is shown.

[0027] In this embodiment, the method includes: S11, configuring a plurality of computing nodes on the current device according to the scale of the large model to be trained and the training task, and configuring the hardware resources and software resources of each computing node to obtain a built distributed training environment; S12, dividing the training data set and the large model to be trained by using parallel strategies in different dimensions to obtain corresponding multiple data blocks and multiple groups of model layers, where the parallel strategies include data parallelism, model parallelism, pipeline parallelism, and context parallelism; S13, starting the training process by loading the model parameters, multiple data blocks, and multiple groups of model layers onto the corresponding computing nodes in the distributed training environment; S14, during the training process, monitoring the running state of each computing node, dynamically adjusting the training tasks of the corresponding computing nodes by introducing an adaptive scheduling mechanism, and saving the target large model that has completed training when the preset training termination condition is met.

[0028] This embodiment effectively integrates the advantages of multi-dimensional parallelism to achieve efficient and stable model training with limited hardware resources, significantly improving training efficiency and resource utilization, while reducing manual intervention and scheduling complexity, and is applicable to the distributed training scenario of ultra-large language models.

[0029] Further, in step S12, dynamic selection and combination of parallel strategies are performed according to different factors (including factors such as the scale of the large model, hardware resource configuration, task type, and input features), including: Select a parallel strategy based on the scale of the large model to be trained, including: when the number of parameters of the large model to be trained is less than 1 billion, only data parallelism is enabled; when the number of parameters is between 1 billion and 10 billion, data parallelism and model parallelism are enabled; when the number of parameters exceeds 10 billion, data parallelism, model parallelism, and pipeline parallelism are enabled; when the input sequence length is long and there are significant differences in context complexity, context parallelism is further enabled; Select a parallel strategy based on the hardware resource configuration, including: when the number of available GPUs / TPUs is large and the video memory is sufficient, the number of data parallel groups is preferentially increased; when the video memory of a single card is limited, model parallelism or pipeline parallelism is preferentially enabled to reduce the memory occupancy of a single device; when the communication bandwidth between computing nodes is high, the scale of the number of data parallel groups can be increased; for heterogeneous computing environments (such as a mixed deployment of different GPU models), combined with context parallelism, high-complexity tasks are assigned to high-performance GPUs to achieve load balancing; Select a parallel strategy based on the task type and the characteristics of the input data (the characteristics of the input data include the length of the input data, the number of tokens, semantic complexity, information content, etc.), including: for tasks such as short-sequence text classification, if the context complexity is low, context parallelism may not be enabled or the granularity of context parallelism may be reduced; for long-text generation tasks, due to strong context dependence and high information density, context parallelism is enabled and the attention mechanism is dynamically adjusted; in multi-modal tasks, the context parallel logic can also be extended to evaluate the complexity and optimize the scheduling of modal segments such as images and audio.

[0030] Even further, in step S12, each parallel strategy realizes the fusion and coordination of different-dimensional division results according to hierarchical division and existing communication mechanisms, including: Hierarchical Division: At the outermost layer of the large model, data parallelism is adopted to replicate the training tasks into multiple device groups (compute nodes), and each device group processes different micro-batches of data. On the basis of data parallelism, pipeline parallelism is adopted in the middle layer to divide the large model to be trained into multiple consecutively executable stages. Each stage is completed by a group of GPUs in cooperation, and parallel computing between different stages is achieved through the micro-batch scheduling mechanism. Inside each pipeline stage, model parallelism is further adopted to allocate different layers of the large model to different GPUs for execution, so as to adapt to the single-card video memory capacity limit and improve the scalability of the large model. In addition, context parallelism is introduced on the basis of the above parallel strategies. According to the complexity of the input sequence, the attention mechanism is dynamically adjusted, and combined with the load status of each current GPU, high-complexity tasks are assigned to GPUs with stronger computing capabilities, and low-complexity tasks are assigned to ordinary GPUs, so as to achieve efficient utilization of resources; The coordination mechanism includes: ensuring that the above parallel strategies at each level can operate in coordination through existing communication mechanisms. For example, at the data parallel level, task distribution is completed through data synchronization operations across device groups; at the pipeline parallel level, efficient transmission of intermediate results is achieved through point-to-point communication or inter-stage transfer mechanisms; at the model parallel level, parameter sharding and gradient aggregation techniques are adopted to ensure computational consistency between GPUs; for context parallelism, combined with the local load feedback mechanism, the task allocation is dynamically adjusted, and the fusion and execution of multi-dimensional division results can be achieved without introducing additional scheduling modules.

[0031] Further, in step S11, the configuration of the hardware resources and software resources of each compute node includes: The configuration of the hardware resources includes configuring the number of GPUs / TPUs and the video memory capacity for each compute node; The configuration of the software resources includes installing a compatible operating system, the corresponding GPU / TPU driver, a distributed training framework, and compatible dependency libraries on each compute node.

[0032] In this embodiment, according to the scale of the large model and the requirements of the training task, the number of required compute nodes and the hardware configuration of each compute node (such as the number of GPUs / TPUs, memory size, etc.) are determined; for example, for an ultra-large language model, hundreds of compute nodes equipped with multiple high-end GPUs may be required. The configuration of the software resources includes: Operating System and Driver Installation: Install a compatible operating system (such as Linux) on each compute node, and install the corresponding GPU / TPU driver (such as CUDA, cuDNN); Deep learning framework deployment: Select and install a suitable large-scale distributed training framework, such as TensorFlow, PyTorch, or a dedicated distributed training library like DeepSpeed; Dependency library installation: Install all necessary dependency libraries with compatible versions, such as scientific computing libraries like NumPy, SciPy, Matplotlib, etc.

[0033] Furthermore, in step S12, the parallel strategies of different dimensions for partitioning the training dataset and the large model to be trained include: S12-1, Using the data parallel strategy, partition the training dataset into multiple data blocks according to the number of computing nodes, and dynamically adjust the data block size based on the computing power and storage capacity of each computing node; S12-2, Through the model parallel strategy, hierarchically partition the large model to be trained to obtain multiple groups of model layers, and allocate each group of model layers to the corresponding computing nodes; S12-3, Using the pipeline parallel strategy, partition the large model to be trained into multiple consecutive stages, and allocate them to the corresponding computing nodes for sequential execution; S12-4, Using the context parallel strategy, segment the input sequence according to the context features of the input sequence, allocate each segment to the matching computing nodes, and dynamically adjust the attention mechanism.

[0034] In this embodiment, the parallel strategies of different dimensions used for partitioning the training dataset and the large model to be trained are specifically as follows: Data parallelism: Intelligently partition the training dataset to be trained according to the number of computing nodes, ensure that each computing node can obtain an appropriate data block, and dynamically adjust the size of the data block based on the computing power and storage capacity of each computing node to ensure load balancing; Model parallelism: Reasonably partition according to the structural characteristics of the large model to be trained to obtain multiple groups of model layers, so that even a super-large-scale model can be trained on limited hardware resources; for example, for a model with a hierarchical structure (such as the Transformer large model), use simple hierarchical partitioning; Pipeline parallelism: Partition the large model to be trained into multiple consecutive stages, and execute them sequentially on different computing nodes. By adopting the micro-batch processing method, different stages can be executed in parallel, and an asynchronous update mechanism is introduced to allow some stages to enter the calculation of the next micro-batch in advance, thereby reducing the "bubble" idle time and improving hardware utilization; Context Parallelism: Dynamically adjust the attention mechanism and generation strategy according to the context features of the input sequence (referring to each data block after the training dataset is partitioned; that is, the token sequence with context semantic structure that the model actually receives and processes during training or inference, which is the result of further segmentation and encoding of the input data), and combine the evaluation feedback mechanism of the model to be trained itself (such as the preference alignment ability of Llama-3.1) to optimize the output quality and improve the consistency and accuracy of inference and generation.

[0035] Further, in step S12-2, the parallel strategy of model parallelism is used to hierarchically partition the large model to be trained, obtaining multiple groups of model layers, and allocating each group of model layers to corresponding computing nodes, including: S12-2-1, Determine the number of segments for partitioning the large model to be trained according to the number of computing nodes, obtaining multiple stages, and allocate the consecutive model layers of each stage to the corresponding computing nodes; S12-2-2, During forward propagation, the input sequence is sequentially passed through each computing node in order; S12-2-3, During backpropagation, the gradient is passed layer by layer from the last computing node to the previous computing node to update the weights.

[0036] In this embodiment, the large model to be trained is a Transformer large model. Among them, the process of hierarchically partitioning the large model to be trained is to allocate different layers in the Transformer large model to corresponding computing nodes to achieve model parallelism. Specifically: Determine the partitioning strategy: According to the number of available computing nodes, determine how many parts to divide the Transformer model into; for example, if there are 4 GPUs, the model can be divided into 4 stages; Divide the model layers: Allocate consecutive layers of one stage to a specific GPU; assume the Transformer model has 12 layers and can be divided into 4 stages, with each stage containing 3 layers. Then: GPU1 is allocated layers 1-3, GPU2 is allocated layers 4-6, GPU3 is allocated layers 7-9, and GPU4 is allocated layers 10-12. Each GPU is responsible for processing the layers it is allocated; Forward propagation process: During training, the input sequence is first sent to the first GPU. After the first GPU finishes processing its part, it passes the output to the next GPU, and so on until the last GPU completes the entire forward propagation process; During the backpropagation process, the gradient is calculated starting from the last GPU and passed forward layer by layer. Each GPU updates its own weights according to the received gradient. For example, if there are 4 GPUs, the gradient is calculated starting from GPU4 and passed back to GPU3. GPU3 uses the received gradient to update its own weights and continues to forward the gradient.

[0037] Further, in step S12-4, the parallel strategy of adopting context parallelism segments the input sequence according to the context features of the input sequence, allocates each segment to a matching computing node, and dynamically adjusts the attention mechanism, including: S12-4-1, extracting the input sequence according to the length of the input sequence and the attention weight distribution to obtain context features; S12-4-2, segmenting the input sequence based on the context features according to a preset window size, and allocating segments corresponding to different context complexities to matching computing nodes according to the context complexity of each segment and the load status of the current corresponding computing node; S12-4-3, dynamically adjusting the attention mechanism according to the context complexity corresponding to different segments. After each computing node completes the processing of the current segment, the information of the context complexity is fed back to the scheduler, and the scheduler optimizes the task division and resource scheduling of the next micro-batch according to the information of the context complexity.

[0038] In this embodiment, the context parallelism method is adopted, specifically including: Extraction and analysis of context features: An evaluation method based on the length of the input sequence and the attention weight distribution is adopted to extract context features, including indicators of context features such as the length of the input sequence, the concentration of attention weights, and the context complexity; Context segmentation and GPU allocation: The input sequence is segmented by a fixed window, and is allocated according to the context complexity of each segment (evaluating the context complexity through the length of the input sequence and the attention weight distribution), combined with the load status of each current computing node (such as video memory occupancy, computing power), that is, allocating segments with high context complexity to GPUs with strong computing power; Dynamic adjustment of the attention mechanism: Based on the evaluation results of the context complexity, dynamically adjust the attention mechanism configuration inside the model. For segments with high context complexity, increase the number of attention heads or expand the attention window to improve the attention to key information; for segments with low context complexity, reduce the number of attention heads or narrow the window range to reduce the computational overhead; each GPU independently executes the attention configuration matching its allocated content to improve the overall parallel efficiency; Asynchronous Feedback and Rescheduling Mechanism: After each GPU finishes processing the current batch, it feeds back the context complexity information to the scheduler. Based on this, the scheduler optimizes the task partitioning and resource scheduling for the next micro-batch. By introducing an asynchronous update mechanism, some GPUs are allowed to enter the next micro-batch calculation in advance, reducing the pipeline idle time and improving the hardware utilization rate.

[0039] Further, in step S14, monitoring the operating status of each computing node and dynamically adjusting the training tasks of the corresponding computing node by introducing an adaptive scheduling mechanism includes: S14-1, obtaining the load status of each computing node by monitoring the GPU utilization rate, memory occupancy, and task completion time of each computing node; S14-2, dynamically adjusting the training task allocation in each stage of pipeline parallelism according to the load status corresponding to the computing node, and performing micro-batch data processing in different stages asynchronously.

[0040] In this embodiment, dynamically adjusting the training tasks of the corresponding computing node by introducing an adaptive scheduling mechanism is specifically as follows: obtaining the load conditions of each node by monitoring indicators such as the GPU utilization rate, memory occupancy, and task completion time of each computing node; when it is detected that a certain computing node is overloaded for a long time, migrating some training tasks to a node with a lighter load to achieve global load balancing; dynamically adjusting the training task allocation in each stage of pipeline parallelism according to the load conditions of the computing nodes to avoid some nodes being idle or overloaded for a long time; allowing different stages to process micro-batch data asynchronously to further improve the hardware utilization rate and reduce the performance loss caused by the "bubble" effect.

[0041] Further, the training termination conditions include reaching a preset number of training rounds or the change amount of the loss value of the validation set being less than a preset threshold.

[0042] In this embodiment, the training termination conditions include the configured number of training rounds (Epoch) or the convergence index (such as the change value of Loss) on the validation set. If the termination conditions are not met, return to step S12 to re-partition the training dataset and the large model to be trained, and start a new cycle of the training process.

[0043] This embodiment further illustrates the implementation of distributed training of the Qwen2.5-72B-Instruct model based on the Transformer architecture through examples: (1)Build a distributed training environment, including preparing computing nodes and configuring necessary software and hardware resources. Specifically: Configure 100 computing nodes, each equipped with 8 NVIDIA A100 GPUs (each GPU has 80GB of video memory), for a total of 800 GPUs, to train a Transformer architecture language model with 72 billion parameters; this model contains 60 Transformer Blocks, each Block contains a Self-Attention layer, a Feed Forward layer, and a LayerNormalization layer; since a single card cannot carry such a large-scale model, a multi-card collaborative model parallel strategy must be adopted. Install Ubuntu 20.04 LTS on each computing node and install necessary GPU driver programs such as CUDA 11.8 and cuDNN 8.6. Select PyTorch as the main deep learning training framework and integrate the DeepSpeed library to optimize distributed training performance. Ensure version compatibility of scientific computing libraries such as NumPy 1.23.0 and SciPy 1.9.0, as well as other libraries that may be required such as Transformers 4.20.0, etc.

[0044] (2)Divide the training datasets Wikipedia and the large model to be trained Qwen2.5-72B-Instruct using a multi-dimensional parallel strategy. Specifically: (2-1)Data parallelism: Clean the Wikipedia corpus. For 800 GPUs, intelligently divide the entire Wikipedia corpus according to the number of computing nodes, so that each GPU processes approximately 0.125% of the data volume; dynamically adjust the data block size according to the actual computing power and storage capacity of each computing node to ensure load balancing; (2-2)Model parallelism: The Qwen2.5-72B-Instruct model is a Transformer architecture language model with 72 billion parameters. Since Transformer is a model with an obvious hierarchical structure, a simple hierarchical segmentation method is adopted, including: Determine the segmentation strategy: Divide the model into multiple stages to be deployed on the GPUs of different computing nodes. For example, according to the available 800 GPUs and 60 Transformer Blocks, divide the entire model into 60 segments, each stage corresponds to a Transformer Block, and is assigned to an independent GPU for processing; Partition the model layers and allocate GPUs: Pack a set of consecutive model layers into a stage and allocate it to a specified GPU for execution. According to the partitioning strategy, GPU1 is allocated Transformer Block 1, GPU2 is allocated Transformer Block 2, and so on. GPU60 is allocated Transformer Block 60. Each GPU is responsible for processing the layers assigned to it; Forward propagation process: During training, the input sequence is first sent to GPU1. After GPU1 processes Transformer Block 1, it passes the output to GPU2, and so on, until the last GPU60 completes all forward computations; Backward propagation process: The gradient is calculated starting from GPU60 and propagated layer by layer forward to GPU1; After receiving the backward gradient, each GPU updates the model parameters it manages and continues to forward the gradient information to complete the complete parameter update process.

[0045] (2-3) Pipeline parallelism: After completing the model layer partitioning, the Qwen2.5-72B-Instruct model is divided into 60 Transformer Blocks and allocated to GPUs 1 to 60 for processing respectively; To further improve the training throughput and hardware utilization, this embodiment uses the remaining 740 GPUs (GPUs 61 to 800) for pipeline parallelism. Specifically: Divide the 60 Transformer Blocks into 10 stages, with each stage containing 6 Blocks, i.e.: Stage 1 contains Blocks 1 to 6, Stage 2 contains Blocks 7 to 12, and so on. Stage 10 contains Blocks 55 to 60; The input data (i.e., a single training batch) is further divided into multiple micro-batches for processing. Each micro-batch enters each pipeline stage in sequence as an input sequence. Each stage is executed in parallel by a group of GPUs, i.e.: GPUs 61 to 70 process Stage 1, GPUs 71 to 80 process Stage 2, and so on. GPUs 151 to 160 process Stage 10; After a stage completes the forward computation of the current batch, it immediately starts processing the next batch without waiting for other stages to complete; In the backward propagation stage, the asynchronous execution method by stage is also adopted. Once a stage receives the gradient from the previous layer, it can immediately start the backward computation of this stage. Such an asynchronous update mechanism can reduce the idle time; (2-4) Context parallelism: Dynamically adjust the calculation strategy of attention weights according to the length of the input data to optimize the output quality. For example, when processing an input sequence (a token sequence with a context semantic structure), prioritize the parts with high attention weights to improve efficiency.

[0046] (3) Load the corresponding data blocks and model layers on all computing nodes respectively, and start the forward propagation and backward propagation processes to initiate the training process.

[0047] (4) During the training process, monitor the running status of each computing node (such as GPU utilization and memory occupancy), and dynamically adjust the training tasks of the corresponding computing nodes. By configuring the number of training epochs to 30 times, or when the change in Loss on the validation set is less than a preset threshold (such as 0.0001), it is considered convergent. At this time, when the training reaches the termination condition, save the trained target large model.

[0048] In this embodiment, by combining the configuration of multiple computing nodes, the partitioning of the dataset and the model, and various parallel strategies, the memory pressure on a single device is effectively reduced. Through the adaptive scheduling mechanism, the task allocation is optimized, avoiding the synchronous waiting of computing nodes and improving the overall computing efficiency. Compared with traditional data parallel methods, it significantly reduces the dependence on the number of GPUs / TPUs, reduces the training cost, and solves problems such as high memory occupancy and large communication overhead. However, by adopting various parallel strategies, combining asynchronous micro-batch processing and hierarchical communication management, it overcomes the problems of uneven load in traditional model parallelism and pipeline bubble effect; effectively improves the system scalability and fault tolerance ability, ensuring efficient and stable operation under different scale models and dynamic workloads. And, through the context parallel mechanism, dynamically adjust the attention mechanism and generation strategy, and combine model evaluation feedback to optimize the output quality, not only improving the model's understanding and generation ability, but also enhancing the consistency of the output during the training process, improving the expressiveness of the final model, and solving the problem that the existing technology lacks effective utilization of input context features, resulting in unstable or inconsistent model generation results.

[0049] Refer to Figure 2 The following is a schematic structural diagram of a distributed training device for a large-scale model provided by an embodiment of the present invention.

[0050] In this embodiment, the device 20 includes: A configuration unit 21, configured to configure multiple computing nodes on the current device according to the scale of the large model to be trained and the training task, and configure the hardware resources and software resources of each computing node to obtain a built distributed training environment; A partitioning unit 22 for partitioning the training data set and the large model to be trained by using parallel strategies in different dimensions to obtain corresponding multiple data blocks and multiple groups of model layers, where the parallel strategies include data parallelism, model parallelism, pipeline parallelism, and context parallelism; A training unit 23 for starting a training process by loading model parameters, multiple data blocks, and multiple groups of model layers onto corresponding computing nodes in the distributed training environment; A monitoring unit 24 for monitoring the running state of each computing node during the training process, dynamically adjusting the training tasks of the corresponding computing nodes by introducing an adaptive scheduling mechanism, and saving the trained target large model when a preset training termination condition is met.

[0051] Each unit module of the apparatus 20 can respectively execute the corresponding steps in the above method embodiments, so the unit modules will not be elaborated here. For details, please refer to the descriptions of the above corresponding steps.

[0052] An embodiment of the present invention further provides a distributed training device for a large-scale model. The device includes the distributed training apparatus for a large-scale model as described above, where the distributed training apparatus for a large-scale model can adopt Figure 2 the structure of the embodiment, and correspondingly, can execute Figure 1 the technical solutions of the method embodiments shown. The implementation principles and technical effects are similar. For details, please refer to the relevant records in the above embodiments and will not be elaborated here.

[0053] The device includes: devices with a photographing function such as mobile phones, digital cameras, or tablet computers, or devices with an image processing function, or devices with an image display function. The device may include components such as a memory, a processor, an input unit, a display unit, and a power supply.

[0054] Among them, the memory can be used to store software programs and modules. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory. The memory mainly includes a program storage area and a data storage area. Among them, the program storage area can store an operating system, application programs required for at least one function (such as an image playback function, etc.); the data storage area can store data created according to the use of the device. In addition, the memory can include a high-speed random access memory, and can also include a non-volatile memory, such as at least one magnetic disk storage device, a flash memory device, or other volatile solid-state storage devices. Correspondingly, the memory can also include a memory controller to provide access to the memory for the processor and the input unit.

[0055] The input unit can be used to receive input digital, character or image information, and generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function control. Specifically, in addition to a camera, the input unit of this embodiment may further include a touch-sensitive surface (such as a touch display screen) and other input devices.

[0056] The display unit can be used to display information input by the user or information provided to the user, as well as various graphical user interfaces of the device. These graphical user interfaces can be composed of graphics, text, icons, videos and any combination thereof. The display unit may include a display panel. Optionally, the display panel can be configured in the form of an LCD (Liquid Crystal Display), an OLED (Organic Light-Emitting Diode), etc. Further, the touch-sensitive surface can cover the display panel. When the touch-sensitive surface detects a touch operation on or near it, it transmits the touch operation to the processor to determine the type of touch event. Subsequently, the processor provides a corresponding visual output on the display panel according to the type of touch event.

[0057] The embodiment of the present invention also provides a computer-readable storage medium. The computer-readable storage medium can be the computer-readable storage medium included in the memory in the above embodiment; it can also exist alone and be a computer-readable storage medium not assembled into the device. At least one instruction is stored in the computer-readable storage medium, and the instruction is loaded and executed by the processor to implement Figure 1 the distributed training method of the large-scale model shown. The computer-readable storage medium can be a read-only memory, a disk or an optical disc, etc.

[0058] The embodiment of the present invention also provides a computer program product, including a computer program / instructions. The computer program / instructions are loaded and executed by the processor to implement Figure 1 the distributed training method of a large-scale model shown.

[0059] It should be noted that the embodiments in this specification are all described in a progressive manner. Each embodiment focuses on the differences from other embodiments. The same or similar parts among the embodiments can be referred to each other. For the device embodiments, the equipment embodiments and the storage medium embodiments, since they are basically similar to the method embodiments, they are described relatively simply. For the relevant parts, refer to the partial description of the method embodiments.

[0060] Furthermore, in this document, the terms "comprises," "comprising," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or apparatus that includes a list of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not preclude the presence of additional identical elements in the process, method, article, or apparatus that includes the element.

[0061] While the foregoing description shows and describes preferred embodiments of the present invention, it should be understood that the present invention is not limited to the forms disclosed herein and should not be construed as excluding other embodiments. Rather, the present invention can be used in various other combinations, modifications, and environments, and can be modified within the scope of the present invention by the teachings herein or by techniques or knowledge in the relevant art. Modifications and variations made by those skilled in the art without departing from the spirit and scope of the present invention are intended to be within the scope of the appended claims.

Claims

1. A distributed training method for large-scale models, characterized in that The method includes: Configuring multiple computing nodes on the current device according to the scale of the large model to be trained and the training task, and configuring the hardware resources and software resources of each computing node to obtain a built distributed training environment; Dividing the training dataset and the large model to be trained using parallel strategies in different dimensions to obtain corresponding multiple data blocks and multiple groups of model layers, where the parallel strategies include data parallelism, model parallelism, pipeline parallelism, and context parallelism; Loading the model parameters, multiple data blocks, and multiple groups of model layers onto the corresponding computing nodes in the distributed training environment to start the training process; During the training process, monitoring the running status of each computing node, dynamically adjusting the training tasks of the corresponding computing nodes by introducing an adaptive scheduling mechanism, and saving the trained target large model when the preset training termination condition is met.

2. The distributed training method for a large-scale model according to claim 1, wherein The configuring of the hardware resources and software resources of each computing node includes: The configuration of the hardware resources includes configuring the number of GPUs / TPUs and the video memory capacity for each computing node; The configuration of the software resources includes installing a compatible operating system, the corresponding GPU / TPU driver, a distributed training framework, and compatible dependency libraries on each computing node.

3. The distributed training method for a large-scale model according to claim 1, wherein The dividing of the training dataset and the large model to be trained using parallel strategies in different dimensions includes: Using the parallel strategy of data parallelism to divide the training dataset into multiple data blocks according to the number of computing nodes, and dynamically adjusting the data block size according to the computing power and storage capacity of each computing node; Performing hierarchical segmentation on the large model to be trained through the parallel strategy of model parallelism to obtain multiple groups of model layers, and allocating each group of model layers to the corresponding computing nodes; Using the parallel strategy of pipeline parallelism to divide the large model to be trained into multiple consecutive stages and allocating them to the corresponding computing nodes for sequential execution; Using the parallel strategy of context parallelism to segment the input sequence according to the context features of the input sequence, allocating each segment to the matching computing nodes, and dynamically adjusting the attention mechanism.

4. The distributed training method for a large-scale model according to claim 3, wherein, The performing of hierarchical segmentation on the large model to be trained through the parallel strategy of model parallelism to obtain multiple groups of model layers, and allocating each group of model layers to the corresponding computing nodes includes: Determining the number of segmentation segments of the large model to be trained according to the number of computing nodes to obtain multiple stages, and allocating the consecutive model layers of each stage to the corresponding computing nodes; During forward propagation, the input sequence is sequentially passed through each computing node; During backpropagation, the gradients are passed layer by layer from the last computing node to the previous computing node to update the weights.

5. The distributed training method for a large-scale model according to claim 3, characterized in that, The using of the parallel strategy of context parallelism to segment the input sequence according to the context features of the input sequence, allocating each segment to the matching computing nodes, and dynamically adjusting the attention mechanism includes: Extracting the input sequence according to the length of the input sequence and the attention weight distribution to obtain context features; Segment the input sequence based on context features according to a preset window size, and allocate the segments corresponding to different context complexities to matching computing nodes according to the context complexity of each segment and the load status of the current corresponding computing node; Dynamically adjust the attention mechanism according to the context complexity corresponding to different segments. After each computing node finishes processing the current segment, feedback the information of the context complexity to the scheduler, and optimize the task division and resource scheduling of the next micro-batch by the scheduler according to the information of the context complexity.

6. The distributed training method for a large-scale model according to claim 1, wherein, Monitoring the running state of each computing node, and dynamically adjusting the training tasks of the corresponding computing node by introducing an adaptive scheduling mechanism, including: Obtain the load status of each computing node by monitoring the GPU utilization, memory occupancy, and task completion time of each computing node; Dynamically adjust the training task allocation of each stage in pipeline parallelism according to the load status corresponding to the computing node, and perform micro-batch data processing of different stages asynchronously.

7. A distributed training method for a large-scale model according to claim 1, characterized in that, The training termination conditions include reaching a preset number of training epochs or the change amount of the loss value of the validation set being less than a preset threshold.

8. A distributed training device for a large-scale model, characterized in that, The device includes: A configuration unit for configuring multiple computing nodes according to the scale of the large model to be trained and the training tasks on the current device, and configuring the hardware resources and software resources of each computing node to obtain a built distributed training environment; A partitioning unit for partitioning the training data set and the large model to be trained by using parallel strategies of different dimensions to obtain corresponding multiple data blocks and multiple groups of model layers, where the parallel strategies include data parallelism, model parallelism, pipeline parallelism, and context parallelism; A training unit for starting the training process by loading model parameters, multiple data blocks, and multiple groups of model layers to the corresponding computing nodes in the distributed training environment; A monitoring unit for monitoring the running state of each computing node during the training process, dynamically adjusting the training tasks of the corresponding computing node by introducing an adaptive scheduling mechanism, and saving the trained target large model when the preset training termination conditions are met.

9. A distributed training device for a large-scale model, characterized in that, It includes a processor, a memory, and a computer program stored in the memory, and the computer program is executed by the processor to implement the steps of a distributed training method for a large-scale model according to any one of claims 1 to 7.

10. A computer program product, characterized in that, It includes computer programs / instructions, and when the computer programs / instructions are executed by a processor, the steps of a distributed training method for a large-scale model according to any one of claims 1 to 7 are implemented.

Images