Data processing method and device, and computer device

Abstract

A data processing method and device and computer equipment are disclosed, and relate to the field of artificial intelligence. In the case of isolating at least one domain, the dependency relationship or / and parameter distribution information of the domain related to the at least one domain is updated. The data processing method provided in the application is a complete optimization scheme of MTTR and MFU. Starting from double-target optimization, low MTTR and high MFU in the elastic promotion and demotion training process are realized. For the MTTR problem, process-level recovery from the latest training step is realized without relying on checkpoints; the reconstruction chain overhead required by the domain is minimized, and the possible reconstruction chain demand in the load balancing process is reduced. For the MFU problem, a reliable large-scale hybrid parallel computing graph scheduling scheme is provided, and the system load imbalance problem caused by load migration in the elastic promotion and demotion process is minimized.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence, and more particularly to a data processing method, apparatus, and computer equipment. Background Technology

[0002] Currently, large-scale artificial intelligence (AI) models are trained collaboratively by multiple processors, and these models involve a large number of parameters. For example, large language models (LLMs) are trained using a multi-machine, multi-GPU distributed system. LLM training can take several months. During model training, checkpoint information is periodically saved; this information includes model parameters and optimizer data. In the event of processor, network, or software failures, model training restarts from the checkpoint based on this information. However, the frequent checkpointing during large-scale AI model training leads to frequent interruptions and restarts, resulting in long recovery times and significant resource overhead. Summary of the Invention

[0003] This application provides a data processing method, apparatus, and computer equipment, thereby eliminating the need for checkpoint-based model training recovery and effectively shortening the model training recovery time.

[0004] Firstly, a data processing method is provided, applied to a computing system including a general-purpose processor and multiple dedicated processors. The method includes: the multiple dedicated processors training an artificial intelligence model based on a first computational graph. The first computational graph describes the dependencies between multiple domains of the artificial intelligence model and parameter distribution information. The parameter distribution information indicates the distribution of model parameters and / or optimizer parameters across multiple domains. With at least one domain isolated, the general-purpose processor updates the dependencies and / or parameter distribution information of the domains related to the at least one domain to obtain a second computational graph. The multiple dedicated processors train the artificial intelligence model based on the second computational graph. The second computational graph describes the dependencies between domains other than the at least one domain and parameter distribution information. The metrics of the second computational graph meet target metrics. Target metrics include Mean Time to Recover (MTTR) and Model FLOPs Utilization (MFU).

[0005] The data processing method provided in this application is a complete scheme for optimizing MTTR and MFU. Starting from dual-objective optimization, it achieves low MTTR and high MFU during the flexible upgrade and downgrade model training process. Isolating at least one domain causes the communication relationship between the isolated domain and other related domains to become invalid. Therefore, the dependency relationships and / or parameter distribution information of the domains related to the isolated domain are updated. Thus, without relying on checkpoints or interrupting model training, the first computation graph is locally updated, effectively shortening the model training recovery time and reducing resource overhead.

[0006] In one possible implementation, the domain associated with at least one domain includes the domain of the pipeline parallel stage to which the at least one domain belongs, and the domain of the pipeline parallel stage adjacent to the pipeline parallel stage to which the at least one domain belongs.

[0007] Optionally, the domain associated with at least one domain also includes the domain to which at least one domain migrates in the pipeline parallelization stage, and at least one of the domains of the pipeline parallelization stage adjacent to the domain to which at least one domain migrates in the pipeline parallelization stage.

[0008] In one possible implementation, dependencies are used to indicate communication relationships between multiple domains; updating the dependencies of domains associated with at least one domain includes updating the communication relationships between domains of at least one pipeline parallel stage associated with at least one domain.

[0009] At least one pipeline parallel stage associated with at least one domain includes the pipeline parallel stage to which at least one domain belongs, and the pipeline parallel stage adjacent to the pipeline parallel stage to which at least one domain belongs.

[0010] Optionally, at least one pipeline parallel stage associated with at least one domain further includes at least one pipeline parallel stage to which the domain is migrated, and at least one of the pipeline parallel stages adjacent to the pipeline parallel stage to which the at least one domain is migrated.

[0011] In another possible implementation, at least one domain includes a first domain belonging to a first pipeline parallel stage; at least one pipeline parallel stage includes the first pipeline parallel stage, and at least one pipeline parallel stage also includes a pipeline parallel stage adjacent to the first pipeline parallel stage; updating the communication relationships between domains of at least one pipeline parallel stage associated with at least one domain includes: updating the communication relationships between domains of the first pipeline parallel stage other than the first domain; updating the communication relationships between domains of the first pipeline parallel stage other than the first domain and domains of pipeline parallel stages adjacent to the first pipeline parallel stage.

[0012] By minimizing the overhead of rebuilding communication relationships to the overhead of rebuilding communication relationships in the pipeline parallel stage, the initialization overhead of global reconstruction is avoided, effectively shortening the recovery time of model training.

[0013] In some embodiments, the domains other than the first domain in the first pipeline parallelization stage are the unisolated domains in the first pipeline parallelization stage. The domains other than the first domain in the first pipeline parallelization stage include at least two unisolated domains.

[0014] Optionally, the first domain includes one or more domains. One or more domains in the first pipeline parallel phase are isolated. Communication relationships between domains in the first pipeline parallel phase that are not isolated, excluding the first domain, are updated.

[0015] Optionally, if the first pipeline parallel phase includes a non-isolated domain in addition to the first domain, then it is not necessary to update the communication relationships of the domains in the first pipeline parallel phase other than the first domain.

[0016] The domains of the pipeline parallelization stage adjacent to the first pipeline parallelization stage include at least one unisolated domain. Update the communication relationships between the unisolated domains of the first pipeline parallelization stage (excluding the first domain) and the unisolated domains of the pipeline parallelization stages adjacent to the first pipeline parallelization stage.

[0017] In other embodiments, at least one isolated domain includes domains from at least two pipeline parallel stages. For each pipeline parallel stage, the communication relationships between domains not isolated in the pipeline parallel stage other than the isolated domain are updated, and the communication relationships between domains not isolated in the pipeline parallel stage other than the isolated domain and domains not isolated in adjacent pipeline parallel stages are updated.

[0018] In another possible implementation, dependencies are also used to indicate the data distribution used for model training, which includes training data or intermediate activations; updating dependencies on domains associated with at least one domain includes: performing data resplitting on the data used for model training of the domains associated with at least one pipeline parallel stage.

[0019] After updating the communication relationships between domains, the data used for model training is re-partitioned to balance the data used by each domain, thereby achieving overall load balancing of the computation graph.

[0020] In another possible implementation, at least one domain includes a first domain belonging to a first pipeline parallel stage, which is adjacent to a second pipeline parallel stage; data resegmentation is performed on the data used for model training of the domains of at least one pipeline parallel stage associated with at least one domain, including: segmenting the data used for model training of the domains of the first pipeline parallel stage according to the number of domains of the second pipeline parallel stage, wherein the domains of the second pipeline parallel stage use the data output by the first pipeline parallel stage for model training.

[0021] In another possible implementation, the first pipeline parallel stage is adjacent to the third pipeline parallel stage; data resegmentation is performed on the data used for model training of the domain of at least one pipeline parallel stage associated with at least one domain, including: splitting the data used for model training of the domain of the third pipeline parallel stage according to the number of domains of the first pipeline parallel stage, wherein the domain of the first pipeline parallel stage uses the data output by the third pipeline parallel stage for model training.

[0022] The data used for model training is redistributed based on the number of domains in the pipeline parallel stage, and the amount of data used by each domain is balanced to achieve overall load balancing of the computation graph.

[0023] In another possible implementation, the method further includes migrating the parameters of the first domain to a domain in the fourth pipeline parallel stage, the parameters including model parameters and / or optimizer parameters.

[0024] In another possible implementation, updating the communication relationships between domains of at least one pipeline parallelization stage associated with at least one domain further includes: updating the communication relationships between domains of the fourth pipeline parallelization stage other than the domain to which the parameters of the first domain are migrated; and updating the communication relationships between the domains of the fourth pipeline parallelization stage other than the domain to which the parameters of the first domain are migrated and the domains of pipeline parallelization stages adjacent to the fourth pipeline parallelization stage.

[0025] In another possible implementation, the first pipeline parallel phase includes a domain in the fourth pipeline parallel phase, and the method further includes: updating the communication relationship between the unisolated domains of the first pipeline parallel phase (excluding the first domain) and a domain in the fourth pipeline parallel phase; updating the communication relationship between the unisolated domains of the first pipeline parallel phase (excluding the first domain) and a domain in the fourth pipeline parallel phase, and between the unisolated domains of the pipeline parallel phase adjacent to the first pipeline parallel phase.

[0026] Transferring isolated domains to carry models can reduce the complexity of subsequent upgrade / downgrade scheduling or improve the system reliability of training downgrade models.

[0027] In another possible implementation, the parameters of the first domain are migrated to a domain in the fourth pipeline parallelization stage, which includes: migrating the parameters of the first domain to a domain in the fourth pipeline parallelization stage based on the memory state of the fourth pipeline parallelization stage, where the memory state of the fourth pipeline parallelization stage is superior to that of the first pipeline parallelization stage.

[0028] For example, based on the memory state of each domain in the second pipeline parallelization stage, the parameters of the first domain are migrated to a domain in the second pipeline parallelization stage.

[0029] Based on the parameters of the memory state transition domain for each domain in the pipeline parallel stage, the memory consumed by each domain for model training is made similar, so that the system can achieve the target indicators.

[0030] In another possible implementation, the parameters of the first domain are migrated to a domain in the fourth pipeline parallel stage, including: migrating the parameters of the first domain to a domain in the fourth pipeline parallel stage based on the network distance between the first domain and the fourth pipeline parallel stage.

[0031] For example, based on the network distance between each domain in the first and second pipeline parallel stages, the parameters of the first domain are migrated to the domain in the second pipeline parallel stage that is closest to the first domain.

[0032] Optionally, based on the network distance between the device mapped to the first domain and the device mapped to each domain in the second pipeline parallelization stage, the parameters of the first domain are migrated to the device mapped to the domain that is closest to the device mapped to the first domain in the second pipeline parallelization stage.

[0033] If the network distance is too large, causing a delay in data transmission, then the parameters of the first domain will be migrated to a domain with the smallest possible network distance to ensure data transmission speed.

[0034] In another possible implementation, the fourth pipeline parallel stage is adjacent to the fifth pipeline parallel stage; performing data resegmentation on the data used for model training of the domain of at least one pipeline parallel stage associated with at least one domain further includes: segmenting the data used for model training of the domain of the fourth pipeline parallel stage according to the number of domains of the fifth pipeline parallel stage, and the domain of the fifth pipeline parallel stage uses the data output by the fourth pipeline parallel stage for model training.

[0035] In another possible implementation, the fourth pipeline parallel stage is adjacent to the sixth pipeline parallel stage; performing data resegmentation on the data used for model training of the domain of at least one pipeline parallel stage associated with at least one domain further includes: segmenting the data used for model training of the domain of the sixth pipeline parallel stage according to the number of domains in the fourth pipeline parallel stage, wherein the domain of the fourth pipeline parallel stage uses the data output by the sixth pipeline parallel stage for model training.

[0036] In another possible implementation, updating the parameter distribution information of the domains associated with at least one domain includes: updating the dependencies of the domains associated with at least one domain to obtain a third computation graph; performing parameter resplitting based on the metrics obtained from model training using the third computation graph, wherein parameter resplitting is used to indicate the resplitting of model parameters and optimizer parameters among multiple domains other than at least one domain.

[0037] For example, if the metrics obtained from model training based on the third computation graph do not meet the target conditions, parameter resplitting is performed according to the metrics of the third computation graph. The target conditions include similar MFU, memory state, communication duration, and computation duration for each pipeline parallel stage, or similar MFU, memory state, communication duration, and computation duration for each domain. This results in higher MFU, similar memory usage, lower average communication duration, and lower average computation duration across multiple pipeline parallel stages, achieving a system equilibrium.

[0038] If the metrics obtained from model training based on the third computation graph do not meet the target metrics, parameter resplitting is performed according to the metrics of the third computation graph. The target metrics include memory state and model training performance metrics, which include MFU, domain communication duration, and computation duration.

[0039] In another possible implementation, the metrics of the third computation graph include metrics for each pipeline parallel stage; at least one domain includes a first domain belonging to the first pipeline parallel stage; the metrics obtained from model training based on the third computation graph are subjected to parameter resplitting, including: migrating the parameters of the domains of the first pipeline parallel stage other than the first domain to the pipeline parallel stage adjacent to the first pipeline parallel stage.

[0040] For example, if the unisolated domain in the first pipeline parallel stage includes the second domain, and the memory usage of the metrics in the second domain is high, the parameters of the second domain are migrated to a domain with lower memory usage in a pipeline parallel stage adjacent to the first pipeline parallel stage.

[0041] In another possible implementation, the parameter resplitting of the metrics obtained from model training based on the third computation graph also includes: migrating the parameters of the domain of the seventh pipeline parallel stage to a pipeline parallel stage adjacent to the seventh pipeline parallel stage. The seventh pipeline parallel stage is any pipeline parallel stage in the third computation graph other than the first pipeline parallel stage.

[0042] In the pipeline parallel stage, the parameters of model training are re-splitted, and the corresponding model parameters and optimizer parameters are passed to the corresponding stage for continuous dynamic adjustment, ultimately achieving the maximum MFU of hybrid parallel training.

[0043] In another possible implementation, at least one domain includes the first domain, and the method further includes: in the case of restoring the first domain, a general-purpose processor updates the dependency relationships and / or parameter distribution information of the domains related to the restored domain to obtain a fourth computation graph. The fourth computation graph is used to describe the dependencies and parameter distribution information between multiple domains other than isolated domains. The domains other than isolated domains include the first domain. The metrics of the fourth computation graph meet the target metrics. Multiple dedicated processors train an artificial intelligence model based on the fourth computation graph.

[0044] In a second aspect, a data processing apparatus is provided, comprising modules for executing the methods of the first aspect or any possible design of the first aspect using a dedicated processor. For example, the data processing apparatus includes a communication module, a training module, and a computational graph update module.

[0045] The training module is used to train an artificial intelligence model based on a first computation graph, which describes the dependencies and parameter distribution information of multiple domains of the artificial intelligence model. The parameter distribution information indicates the distribution of model parameters and optimizer parameters across multiple domains. The computation graph update module is used to update the dependencies and / or parameter distribution information of domains related to at least one domain by a general-purpose processor, while isolating at least one domain, to obtain a second computation graph. The second computation graph describes the dependencies and parameter distribution information between domains other than at least one domain in the multiple domains. The metrics of the second computation graph satisfy target metrics, including MTTR and MFU. The training module is also used by multiple dedicated processors to train the artificial intelligence model based on the second computation graph.

[0046] In one possible implementation, when the computation graph update module updates the dependencies of domains associated with at least one domain, it is specifically used to update the communication relationships between domains of at least one pipeline parallel stage associated with at least one domain.

[0047] In another possible implementation, at least one domain includes a first domain, which belongs to a first pipeline parallel stage; at least one pipeline parallel stage includes the first pipeline parallel stage, and at least one pipeline parallel stage also includes a pipeline parallel stage adjacent to the first pipeline parallel stage; when the computation graph update module updates the communication relationship between the domains of at least one pipeline parallel stage related to at least one domain, it is specifically used to update the communication relationship between the domains of the first pipeline parallel stage; and to update the communication relationship between the domains of the first pipeline parallel stage and the domains of the pipeline parallel stage adjacent to the first pipeline parallel stage.

[0048] In another possible implementation, dependencies are also used to indicate the data distribution used for model training, which includes training data or intermediate activations; the computation graph update module is also used to perform data resplitting on the domain of at least one pipeline parallel stage using the data used for model training.

[0049] In another possible implementation, at least one domain includes a first domain, which belongs to a first pipeline parallel stage, and the first pipeline parallel stage is adjacent to a second pipeline parallel stage. When the computation graph update module performs data resegmentation on the data used for model training of the domains of at least one pipeline parallel stage associated with at least one domain, it is specifically used to segment the data used for model training of the domains of the first pipeline parallel stage according to the number of domains of the second pipeline parallel stage. The domains of the second pipeline parallel stage use the data output by the first pipeline parallel stage for model training.

[0050] In another possible implementation, the first pipeline parallel stage is adjacent to the third pipeline parallel stage; when the computation graph update module performs data resegmentation on the data used for model training of the domain of at least one pipeline parallel stage associated with at least one domain, it is specifically used to segment the data used for model training of the domain of the third pipeline parallel stage based on the number of domains in the first pipeline parallel stage, and the domain of the first pipeline parallel stage uses the data output by the third pipeline parallel stage for model training.

[0051] In another possible implementation, the computation graph update module is further configured to migrate parameters of the first domain to a domain in the fourth pipeline parallel stage, the parameters including model parameters and / or optimizer parameters, the first pipeline parallel stage containing a domain in the fourth pipeline parallel stage; the computation graph update module updates the communication relationships between domains in at least one pipeline parallel stage associated with at least one domain, and is further configured to: update the communication relationships between domains in the fourth pipeline parallel stage other than the domain to which the parameters of the first domain are migrated; update the communication relationships between domains in the fourth pipeline parallel stage other than the domain to which the parameters of the first domain are migrated and domains in pipeline parallel stages adjacent to the fourth pipeline parallel stage.

[0052] In another possible implementation, when the computation graph update module migrates the parameters of the first domain to a domain in the second pipeline parallel stage, it specifically migrates the parameters of the first domain to a domain in the second pipeline parallel stage based on the memory state of the second pipeline parallel stage, where the memory state of the second pipeline parallel stage is superior to that of the first pipeline parallel stage.

[0053] In another possible implementation, when the computation graph update module migrates the parameters of the first domain to a domain in the second pipelined parallel stage, it specifically migrates the parameters of the first domain to a domain in the second pipelined parallel stage based on the network distance between each domain in the first and second pipelined parallel stages.

[0054] In another possible implementation, the fourth pipeline parallel stage is adjacent to the fifth pipeline parallel stage; the computation graph update module is also used to divide the domains of the fourth pipeline parallel stage for model training based on the number of domains in the fifth pipeline parallel stage, and the domains of the fifth pipeline parallel stage use the data output by the fourth pipeline parallel stage for model training.

[0055] In another possible implementation, the fourth pipeline parallel stage is adjacent to the sixth pipeline parallel stage; the computation graph update module is also used to divide the domains of the sixth pipeline parallel stage for model training based on the number of domains in the fourth pipeline parallel stage, and the domains of the fourth pipeline parallel stage use the data output by the sixth pipeline parallel stage for model training.

[0056] In another possible implementation, when the computation graph update module updates the parameter distribution information of the domains related to at least one domain, it is specifically used to update the dependencies of the domains related to at least one domain to obtain a third computation graph. Based on the third computation graph, the metrics obtained from model training are used to perform parameter resplitting. Parameter resplitting is used to indicate the resplitting of model parameters and optimizer parameters among multiple domains other than at least one domain. The target metrics include memory status and model training performance metrics. The model training performance metrics include MFU, domain communication duration, and computation duration.

[0057] In another possible implementation, the metrics of the third computation graph include the metrics of each pipeline parallel stage; at least one domain includes a first domain, which belongs to the first pipeline parallel stage; when the computation graph update module performs parameter resplitting based on the metrics obtained from model training according to the third computation graph, it is specifically used to migrate the parameters of the domains of the first pipeline parallel stage other than the first domain to the pipeline parallel stage adjacent to the first pipeline parallel stage.

[0058] In another possible implementation, the computation graph update module is also used to migrate the parameters of the domain of the seventh pipeline parallel stage to a pipeline parallel stage adjacent to the seventh pipeline parallel stage, which is any pipeline parallel stage in the third computation graph other than the first pipeline parallel stage.

[0059] In another possible implementation, at least one domain includes a first domain. The computation graph update module is further used to update the dependencies and / or parameter distribution information of the domains related to the restored domain when the first domain is restored, to obtain a fourth computation graph. The fourth computation graph is used to describe the dependencies and parameter distribution information between multiple domains other than isolated domains. The domains other than isolated domains include the first domain. The metrics of the fourth computation graph meet the target metrics. Multiple dedicated processors train an artificial intelligence model based on the fourth computation graph.

[0060] Thirdly, a computing system is provided, which includes a general-purpose processor and multiple special-purpose processors, and a memory for storing a set of computer instructions; when the processor executes the set of computer instructions, the general-purpose processor and the multiple special-purpose processors jointly execute the operation steps of the method in the first aspect or any possible implementation of the first aspect.

[0061] Fourthly, a computer device is provided, the computer device including a memory and a plurality of processors, the memory being used to store a set of computer instructions; when the processors execute the set of computer instructions, the plurality of processors jointly execute the operation steps of the method as described in the first aspect or any possible implementation of the first aspect.

[0062] Fifthly, a computer-readable storage medium is provided, comprising: computer software instructions; when the computer software instructions are executed in a processor, causing the processor to perform operational steps of the method as described in the first aspect or any possible implementation thereof.

[0063] In a sixth aspect, a computer program product is provided that, when run on a computer, causes the computer to perform the operational steps of the method as described in the first aspect or any possible implementation thereof.

[0064] The technical effects of any of the design methods in aspects two through six can be found in aspect one or in different design methods in aspect one, and will not be repeated here.

[0065] Based on the implementation methods provided in the above aspects, this application can be further combined to provide more implementation methods. Attached Figure Description

[0066] Figure 1 A schematic diagram of a large model training method provided in this application;

[0067] Figure 2 A schematic diagram of a checkpoint provided in this application;

[0068] Figure 3 A schematic diagram of the architecture of a data processing system provided in this application;

[0069] Figure 4 A schematic diagram of the structure of a domain is provided in this application;

[0070] Figure 5 A flowchart illustrating a data processing method provided in this application;

[0071] Figure 6 A schematic diagram of a chain establishment method provided for this application;

[0072] Figure 7 A schematic diagram illustrating a data resegmentation method provided in this application;

[0073] Figure 8 A schematic diagram of a migration provided for this application;

[0074] Figure 9 A schematic diagram of parameter resegmentation provided in this application;

[0075] Figure 10 A schematic diagram of the structure of a data processing device provided in this application;

[0076] Figure 11 This is a schematic diagram of the structure of a computer device provided in this application. Detailed Implementation

[0077] To facilitate understanding, the main terms used in this application will be explained first.

[0078] Large models refer to ultra-large-scale artificial intelligence (AI) models. They are widely used in natural language processing (NLP) and are fundamentally changing the landscape of NLP tasks, giving rise to more powerful and intelligent language technologies. Large models are a key direction in AI development. They also excel in various NLP tasks, such as text classification, sentiment analysis, summarization, and translation. Large models are used in multiple application areas, including automatic writing, chatbots, virtual assistants, voice assistants, and automatic translation. Examples include Large Language Models (LLMs), Bidirectional Encoder Representations from Transformers (BERT), Generative Pre-Trained Transformers (GPT), GPT3, GPT4, and MoE. Large models possess the following characteristics.

[0079] 1. Huge scale: The model size reaches hundreds of gigabytes (GB) or even larger. Large models contain billions, hundreds of billions or even trillions of model parameters. Such huge-scale models provide powerful expressive and learning capabilities.

[0080] 2. Multi-task learning: Large models handle a variety of different natural language processing tasks, such as machine translation, text summarization, question answering systems, etc., enabling the models to learn a broader and more generalized language understanding ability.

[0081] 3. Powerful computing resources: Training large models typically requires hundreds or even thousands of dedicated processors and a significant amount of time; for example, training a large model can take weeks to months. Powerful computing resources accelerate the training process while preserving the capabilities of large models. Dedicated processors include, but are not limited to, graphics processing units (GPUs), data processing units (DPUs), neural processing units (NPUs), and embedded neural network processing units (NPUs).

[0082] 4. Abundant data: Large models are trained using a large amount of training data, leveraging the scale advantage of the model parameters.

[0083] Model parameters refer to the variables or weights that need to be learned or adjusted in an artificial intelligence model. Model parameters affect the model's predictive ability and performance. During model training, the model optimizes its performance by trying different combinations of parameters. Common model parameters include weights, biases, learning rates, and regularization coefficients. When using the model for prediction, these model parameters are used to calculate the output results.

[0084] Model training refers to training an artificial intelligence model using a training set, enabling the model to predict or classify unknown data. During training, the model learns from features and target values ​​in the training set, generating a model that can then be used to predict or classify unknown data. Model training is one of the most crucial steps in machine learning, significantly impacting the model's accuracy and reliability.

[0085] An optimizer is an algorithm used to adjust the parameters of an artificial intelligence (AI) model so that the model can more accurately predict outputs. The goal of an optimizer is to minimize the loss function, which is the difference between the model's predictions and the actual values. Common optimizers include stochastic gradient descent (SGD), adaptive moment estimation (Adam), adaptive gradient (Adagrad), and root mean square propagation (RMSprop). These optimizers use different strategies to update model parameters to achieve better performance and accuracy for the AI ​​model. Typically, the input to an optimizer is the gradient, and the output is the model parameters.

[0086] A checkpoint (CKPT) refers to the periodic saving of data during model training, either at regular intervals or after each training epoch, to facilitate subsequent model recovery and training. Checkpoints include model parameters, optimizer parameters, and optimizer state (OS). Optimizer state includes variance and momentum. For example, if an unexpected event interrupts model training, training can be resumed based on the most recent checkpoint without restarting the process. Checkpoints are also used for model evaluation and debugging. For instance, by observing the model's performance at different training stages based on different checkpoints, the training effectiveness and optimization direction can be determined. In deep learning, checkpoints are a way to save and manage model parameters.

[0087] Resuming training from a standby point is a deep learning training technique that allows model training to be paused during the training process, saving the current data. This current data typically includes model parameters and optimizer parameters. When training resumes, the model continues from where it left off, rather than starting from scratch. This technique accelerates the model training process, reduces wasted computing resources, and handles various problems that may arise during training, such as computer failures or network interruptions.

[0088] Figure 1 This is a schematic diagram illustrating a large model training method provided in this application. For example... Figure 1 As shown, the computing system 100 includes multiple cards. These cards are also called training cards, accelerator cards, or dedicated processors. These multiple cards reside on multiple servers. The servers are also called hosts. Parallel techniques are used to accelerate the training of large models based on these multiple cards. Parallel techniques include data parallelism (DP), model parallelism (MP), pipeline parallelism (PP), and tensor parallelism (TP).

[0089] Data parallelism refers to dividing multiple cards into multiple data parallel paths, such as... Figure 1 As shown, multiple cards are divided into X data parallel paths. The training set is split based on the data parallel paths, and the cards in each data parallel path train the large model based on the sub-training set. The sub-training sets used by multiple data parallel paths to train the large model in parallel are different.

[0090] Tensor model parallelism refers to a strategy of partitioning the parameters of a large model across different GPUs for computation. For example, based on the model size and the number of GPUs, parameters within the same layer of the model can be partitioned across different GPUs. Each GPU performs computation based on its assigned parameters. Because the parameters are partitioned, each GPU processes a portion of the data, thus reducing the computational burden and memory usage of a single GPU. During computation, the GPUs communicate to exchange intermediate results or gradients, and aggregate their respective computational results using collective communication protocols (such as AllReduce, AllGather, etc.) to obtain the global final result.

[0091] Pipeline parallelism refers to dividing a large model into multiple layers based on the logical order between these layers, and then training each layer of the large model in parallel or sequentially using multiple GPUs, based on that logical order. The logical order between layers refers to the dependencies between them. For example, if the output data of the first layer is the input data of the second layer, then the first and second layers are trained sequentially using two GPUs.

[0092] The data stored in the cards mainly includes activation, model parameters, and optimizer parameters. Each card performs operations such as forward pass, activation, backward pass, gradient generation, and gradient accumulation. It also distributes gradients to multiple cards through the AllReduce operation, and the cards update optimizer data with the same identifier as the gradients based on the gradients. The AllGather operation obtains the model parameters transformed from the optimizer parameters, thereby realizing model training.

[0093] In some embodiments, the card periodically saves checkpoints, that is, it saves checkpoints every certain period of time. These checkpoints are also called periodic checkpoints. If model training fails at any point in time, model training is resumed based on the previous checkpoint.

[0094] For example, such as Figure 2 As shown in (a), card n fails, and cards 1 and 2 persist the model parameters and optimizer data to the storage system. Figure 2 As shown in (b), during model training, the i-th checkpoint and the (i+1)-th checkpoint are saved. After the (i+1)-th checkpoint, a fault occurs, and model training is interrupted. The fault is detected, the fault mode is determined, and it is decided whether to restart training. If model training is restarted, it is resumed based on the (i+1)-th checkpoint.

[0095] Because the model is trained using the previous checkpoint, some training loss occurs. For example, the optimizer data and updated model parameters from the (i+1)th checkpoint to the failure point are lost.

[0096] When the number of computing nodes changes or a failure occurs, the model training process may need to be paused to coordinate new resources. This interruption will extend training time. ① Resuming training from a checkpoint is very time-consuming. For large-scale models, loading model parameters and optimizer data takes even longer. ② Resuming training from the most recent checkpoint incurs the overhead of recomputation. ③ There is the overhead of re-establishing communication links in the system.

[0097] Checkpoint saving overhead: To reduce training progress loss due to failures or scaling, checkpoints need to be saved frequently. This consumes significant input / output (I / O) bandwidth, impacting training speed. Large-scale models have large checkpoint files, and frequent saving increases storage costs. In a distributed environment, multiple nodes simultaneously saving checkpoints can become I / O bottlenecks in the storage system, further degrading performance.

[0098] Training resource utilization efficiency is reduced. For example, the granularity of elastic adjustment in DLRover is coarse, resulting in direct cutting of DP paths. If a card fails, the data parallel path containing that card needs to be directly cut, which is too costly.

[0099] The elasticity requirements of model training include a low Mean Time to Recover (MTTR) and a high Model FLOPs Utilization (MFU). The main factors contributing to MTTR include: ① reconstruction of high-performance communication links; ② checkpoint retrieval; ③ recalculation time from the checkpoint to training interruption; and ④ time for system rescheduling of the computation graph. These overheads become increasingly severe in larger-scale model training, reducing system availability. The main factors contributing to MFU include load balancing during elastic scaling up and down training, such as whether it introduces additional idling time overhead into the training process.

[0100] To address the issues of long training recovery times and high resource consumption in model training, this application provides a data processing method, specifically a method for training and recovering artificial intelligence models in a distributed manner. The computational system used in this method includes a general-purpose processor and multiple dedicated processors, with the dedicated processors used to train the artificial intelligence model. The method includes: multiple dedicated processors training the artificial intelligence model based on a first computational graph. The first computational graph describes the dependencies and parameter distribution information of multiple domains of the artificial intelligence model. The parameter distribution information indicates the distribution of model parameters and optimizer parameters across multiple domains. The dependencies indicate the communication relationships between multiple domains and the distribution of data used for model training, including training data or intermediate activations. With at least one domain isolated, the general-purpose processor updates the dependencies and / or parameter distribution information of the domains related to at least one domain, resulting in a second computational graph. The multiple dedicated processors then train the artificial intelligence model based on this second computational graph. The second computational graph describes the dependencies and parameter distribution information between domains other than at least one domain. The metrics of the second computational graph satisfy target metrics, including MTTR and MFU.

[0101] The data processing method provided in this application is a complete scheme for optimizing MTTR and MFU. Starting from dual-objective optimization, it achieves low MTTR and high MFU during the training process of the elastic upgrade and downgrade model. Since isolating at least one domain causes the communication relationship between the isolated domain and other related domains to fail, the dependency relationships and / or parameter distribution information of the domains related to at least one domain are updated, i.e., a local update of the first computation graph is performed. Thus, model training can be resumed from the most recent training step without relying on checkpoints, and the overhead of rebuilding communication relationships required for isolating at least one domain is minimized, reducing the need for rebuilding communication relationships during load balancing of parameter updates. Regarding the MFU problem, this application provides a reliable large-scale hybrid parallel computation graph scheduling scheme to minimize the system load imbalance caused by load migration during the training process of the elastic upgrade and downgrade model.

[0102] Model FLOPs Utilization (MFU) refers to the ratio of the actual computational cost used by a model during forward and backward computations to the theoretical maximum computational cost. For example, MFU measures the ratio of the matrix computational cost to the machine computational cost in a single iteration.

[0103] Mean Time to Recover (MTTR) is the average time required for a system or device to recover from a failure. It is an important indicator of the reliability of a system or device.

[0104] The implementation methods of the data processing method provided in this application will be described in detail below with reference to the accompanying drawings.

[0105] Figure 3 This is a schematic diagram of the architecture of a data processing system provided in this application. Figure 3 As shown, the data processing system 300 includes a client 310, a computing cluster 320, and a storage cluster 330.

[0106] Storage cluster 330 comprises multiple storage nodes 331. Each storage node 331 includes one or more controllers, a network interface card (NIC), and multiple hard drives. Hard drives are used to store data. Hard drives are disks or other types of storage media, such as solid-state drives (SSDs) or shingled magnetic recording (SMR) hard drives. The NICs are used to communicate with the compute nodes 321 included in compute cluster 320. The controllers are used to write data to or read data from the hard drives based on read / write data requests sent by the compute nodes 321. During the read / write process, the controller needs to translate the address carried in the read / write data request into an address that the hard drive can recognize.

[0107] The computing cluster 320 contains multiple computing nodes 321. For example, a computing node 321 is a computing device, such as a server.

[0108] In some embodiments, the computing cluster 320 is a heterogeneous computing architecture to provide high-performance computing. For example, computing nodes 321 include computing units with computing capabilities such as central processing units (CPUs), graphics processing units (GPUs), data processing units (DPUs), neural processing units (NPUs), and neural-network processing units (NPUs) to provide high-performance computing.

[0109] In other embodiments, multiple computing nodes 321 are connected via network devices (such as switches, network interface cards, etc.) based on high-speed interconnect technology, enabling communication between the multiple computing nodes 321.

[0110] In this application, multiple computing nodes 321 are used to train an artificial intelligence model. Each computing node 321 includes at least one dedicated processor. This application does not limit the number of dedicated processors in a computing node. Different computing nodes may have the same or different numbers of dedicated processors.

[0111] A computation graph is a graph structure used to describe and organize the operations of an artificial intelligence model. A graph structure consists of nodes and edges. Nodes represent computational units, such as addition, multiplication, activation functions, linear transformations, and other operations. In an AI model, each node stores its input and output. Edges represent data flow, that is, the transfer of data between computational units. If the input of one operation depends on the output of another operation, then there is a dependency between the two operations, and they are connected by an edge.

[0112] Forward Propagation: During model training, input data is passed through the network layer by layer along the edges of the computation graph, undergoing a series of calculations and activation function processing to finally obtain the output result. This process is the foundation for the model's prediction capabilities.

[0113] Backward Propagation: After obtaining the output, the derivative of the output is calculated using the loss function, and the gradient is propagated back to each node along the backward path of the computation graph, thereby optimizing and updating the parameters. This process is crucial for the model to learn and adjust its parameters to reduce prediction errors.

[0114] In some embodiments, the artificial intelligence model is divided into multiple domains based on distributed training. A computation graph is constructed for each domain, which describes the dependencies and parameter distribution information of the domains. The parameter distribution information indicates the distribution of model parameters and optimizer parameters across the multiple domains; that is, it indicates the model parameters and optimizer parameters corresponding to each domain. Dependencies indicate the communication relationships between the multiple domains and the distribution of data used for model training. The data used for model training includes training data or intermediate activations. For example, a domain may contain the first L layers of the model, and the data used for training the model in that domain includes the training data. Similarly, a domain may contain one or more intermediate layers of the model, and the data used for training the model in that domain includes intermediate activations.

[0115] For example, such as Figure 4 As shown in (a), the computation graph comprises (m+1)*(n+1) domains of the artificial intelligence model based on data parallelism and pipeline parallelism. The computation graph includes m+1 data parallel paths and n+1 pipeline parallel stages.

[0116] Each of the (m+1)*(n+1) domains represents the intersection of the data parallel path and the pipelined parallel stage. Each domain carries a portion of the layers in the model. For example, the domain (DP) in the pipelined parallel stage n... m PP n After the L-layer of the carrying model, the domain (DP) in the pipeline parallel stage 0 m ,PP0) Bearing model front L layer.

[0117] Each column represents a data parallel path, which comprises multiple pipelined parallel stages. A data parallel path is used to train a complete model. Each domain of a data parallel path carries a different layer in the model. A domain of a data parallel path is used to train one or more layers in the model. Different data parallel paths use different training data to train the model. For example, data parallel path (DP0) includes domains (DP0, PP0) and (DP0, PP1). Domain (DP0, PP0) uses data 1 to train layer 1, and domain (DP0, PP1) uses the data output from domain (DP0, PP0) to train layer 2.

[0118] Each row represents a pipeline parallel stage, and a pipeline parallel stage comprises multiple data parallel paths. A pipeline parallel stage is used to train a subset of layers in the model. A pipeline parallel stage contains multiple domains. A pipeline parallel stage contains multiple domains used to train the same layers in the model. Different domains within a pipeline parallel stage are used to train the same layers in the model using different training data. For example, the pipeline parallel stage (PP1) includes domains (DP0, PP1) and (DP1, PP1), where domain (DP0, PP1) trains layer 2 using data 1, and domain (DP1, PP1) trains layer 2 using data 2.

[0119] Optionally, other parallel methods such as tensor parallelism (TP) can also be used within the domain.

[0120] Optionally, from the first pipeline parallel stage to the nth pipeline parallel stage, the memory pressure decreases, that is, the memory usage decreases, or the available remaining memory increases.

[0121] This application maps a computation graph to multiple computation nodes. The specific mapping relationships are not limited in this application. For example, one domain may be mapped to one computation node. Another example is a domain mapped to a dedicated processor. Yet another example is at least two domains mapped to one computation node and at least two domains mapped to a dedicated processor. A computation node or a dedicated processor trains a portion of the layers of a model hosted by one or more domains.

[0122] Multiple computing nodes establish links based on the communication relationships between multiple domains, and multiple computing nodes train artificial intelligence models based on the computing graph.

[0123] For example, such as Figure 4As shown in (b), domain (DP3, PP0) is mapped to compute node 1. Domain (DP3, PP1) is mapped to compute node 2. Domain (DP3, PP2) is mapped to compute node 3. Domain (DP3, PP3) is mapped to compute node 4. Domain (DP3, PP4) is mapped to compute node 5. Compute node 1 is linked to compute node 2, compute node 2 is linked to compute node 3, compute node 3 is linked to compute node 4, and compute node 4 is linked to compute node 5.

[0124] Optionally, the general-purpose processor in the system can be used as a central scheduler to collect memory status and model training performance metrics for each domain.

[0125] Optional, such as Figure 4 As shown in (c), the first domain of each pipeline parallelization stage is used as the PP scheduler to collect the domain's health status, memory status, and model training performance metrics during the pipeline parallelization stage. The PP scheduler feeds back the domain's health status, memory status, model training performance metrics, and computation graph to the central scheduler. The central scheduler determines the computation graph update strategy based on these metrics and instructs on the updated computation graph.

[0126] In some embodiments, client 310 communicates with computing cluster 320 and storage cluster 330 via network 340. For example, client 310 sends a request to computing cluster 320 via network 340, requesting computing cluster 320 to perform model training. Network 340 refers to an enterprise intranet (e.g., a local area network, LAN) or the Internet. Client 310 refers to a computer connected to network 340, also known as a workstation. Different clients share network resources (e.g., computing resources, storage resources).

[0127] Optionally, the computing cluster 320 may also include a control node 322. For example, the control node and the computing nodes may be independent physical devices. Alternatively, the control node and multiple computing nodes may reside on the same physical device. The control node is the CPU. The multiple computing nodes include computing units such as GPUs, NPUs, and DPUs. The control node 322 is used to manage and allocate tasks, allowing multiple computing nodes to execute multiple tasks in parallel to improve data processing speed.

[0128] Optionally, the control node 322 is configured to instruct multiple computing nodes to perform model training based on the distributed training described in the above embodiments, upon request. The control node 322 is also configured to update the dependencies and / or parameter distribution information of the domains related to at least one domain, while isolating at least one domain, thereby enabling the resumption of model training without restarting or interrupting it.

[0129] In this embodiment of the application, the storage cluster 330 stores parameters, including model parameters and optimizer parameters, etc.

[0130] In other embodiments, client 310 has client program 311 installed. Client 310 runs client program 311 and displays a user interface (UI). User 350 manipulates the user interface to submit a request. For example, user 350 manipulates the user interface to submit a request. After receiving the request, control node 322 loads optimizer parameters from storage cluster 330 and distributes the optimizer parameters to multiple compute nodes, enabling the multiple compute nodes to perform model training.

[0131] Optionally, the system administrator 360 can configure system information, such as configuring a computing graph for a computing node, by calling the application platform interface (API) 312 or the command-line interface (CLI) 313 through the client 310.

[0132] Figure 3 This is merely an illustrative diagram; the embodiments of this application do not limit the device connection method or the number of devices in the data processing system. For example, the data processing system includes multiple clients. One client connects to multiple computing nodes. Different clients establish connections with different computing nodes.

[0133] Next, the data processing procedure will be explained in detail with reference to the attached diagram.

[0134] Figure 5 This is a flowchart illustrating a data processing method provided in this application. Here, we mainly focus on... Figure 4 The training of the recovery model is illustrated using multiple domains as examples.

[0135] Step 510: Multiple dedicated processors train an artificial intelligence model based on the first computation graph.

[0136] The first computational graph is used to describe the dependencies between multiple domains of the artificial intelligence model and the parameter distribution information. The parameter distribution information is used to indicate the distribution of model parameters and / or optimizer parameters across multiple domains.

[0137] The first computational graph is the initial computational graph after model partitioning, i.e., the computational graph without isolating any domain. Alternatively, the first computational graph is an updated computational graph after isolating one or more domains; this application does not limit the scope.

[0138] For an explanation of the first computational graph, please refer to the above. Figure 4 The explanation of the computational graph.

[0139] Step 520: With at least one domain isolated, the general-purpose processor updates the dependencies and / or parameter distribution information of the domains associated with at least one domain to obtain a second computation graph.

[0140] This application does not limit the application scenario for isolating at least one domain. For example, if the hardware carrying a domain fails, then the domain carried by the failed hardware (e.g., a dedicated processor) is isolated. Hardware failure refers to the interruption of the connection between a dedicated processor such as a GPU or NPU, or a computing node containing a dedicated processor, due to hardware damage, overheating, software errors, network interruption, system crash, power supply problems, or external environmental influences. This results in the system being unable to identify the faulty device, the faulty device being unable to participate in computing tasks, or being unable to work normally. During the training of large models, such single-point failures lead to global restarts. When the model scale is very large (e.g., a cluster of tens of thousands of cards or more), the frequency of hardware failures will increase significantly, resulting in huge overhead for global restarts and seriously affecting the training efficiency of the cluster.

[0141] For example, based on proactive elasticity requirements, at least one domain can be isolated to reduce the size of the computation graph.

[0142] In some embodiments, during the training of an artificial intelligence model based on a first computation graph by multiple dedicated processors, the health status of a domain is acquired. For example, the health status indicates whether the dedicated processors mapped to the domain are faulty. The dedicated processors report the health status of the domain to the general-purpose processor.

[0143] For example, the PP scheduler periodically reports the health status of a domain to the central scheduler. For instance, in period i, it reports the first health status, indicating that the first domain has not failed. After reporting the first health status, if the first domain fails, it waits for period i+1 to arrive, at which point it reports the second health status, indicating that the first domain has failed.

[0144] For example, the PP scheduler reports the health status of domains to the central scheduler in real time. For instance, if a failure occurs in the first domain, it reports that a failure has occurred in the first domain.

[0145] After the general-purpose processor determines that at least one domain is faulty, it isolates at least one domain. The specific isolation method is not limited in this application.

[0146] The dedicated processor reports the model training metrics for each domain to the general-purpose processor. These metrics include memory status and model training performance metrics. Memory status refers to memory usage. Model training performance metrics include memory utilization (MFU), communication duration for each domain, and computation duration.

[0147] Optionally, a dedicated processor can be used to store the domain's health status and model training metrics. For example, the domain's health status can be stored in the dedicated processor's memory.

[0148] Assume that at least one isolated domain includes a first domain, which belongs to the first pipeline parallel stage and the first data parallel path. The following explanation uses the isolation of the first domain to illustrate the recovery of model training; the isolation of other domains is similar to the explanation of the first domain.

[0149] The domains associated with the isolated first domain include the domains of the pipeline parallel stages to which the first domain belongs, as well as the domains of the pipeline parallel stages adjacent to the pipeline parallel stages to which the first domain belongs.

[0150] At least one pipeline parallel stage associated with the first domain includes at least one of the pipeline parallel stage to which the first domain belongs, the pipeline parallel stage adjacent to the pipeline parallel stage to which the first domain belongs, the pipeline parallel stage to which the first domain migrates, and the pipeline parallel stage adjacent to the pipeline parallel stage to which the first domain migrates.

[0151] Optionally, the first domain includes one or more domains. This isolates one or more domains within the first pipeline parallel phase.

[0152] Because each domain mapping has a dedicated processor used to train one or more layers in the model. A data parallel path contains at least one dedicated processor for a domain mapping used to train the complete model. A pipeline parallel stage contains dedicated processors for each domain mapping used to train one or more identical layers in the model. Isolating the first domain causes communication between the first domain and other domains related to it to fail, resulting in an incomplete model carried by the first data parallel path, and some layers in the model carried by the first data parallel path cannot be trained. Updating the dependencies and / or parameter distribution information of the domains related to at least one domain restores model training. The following describes updating the computation graph to restore model training through one or more implementations.

[0153] In a first possible implementation, dependencies are used to indicate communication relationships between multiple domains. Updating the dependencies of domains associated with at least one domain includes updating the communication relationships between domains in at least one pipeline parallel stage associated with at least one domain.

[0154] At least one pipeline parallel stage includes a pipeline parallel stage belonging to an isolated domain and a pipeline parallel stage associated with the pipeline parallel stage belonging to the isolated domain.

[0155] If at least two domains are isolated and belong to different pipeline parallel stages, then the pipeline parallel stages to which the isolated domains belong contain at least two different pipeline parallel stages.

[0156] Updating the communication relationships between domains in at least one pipeline parallel stage includes updating the communication relationships between non-isolated domains in the pipeline parallel stage to which an isolated domain belongs, and the communication relationships between non-isolated domains between the pipeline parallel stage to which an isolated domain belongs and adjacent pipeline parallel stages, so that each data parallel path carries the complete model through the newly established communication relationships.

[0157] For example, at least one pipeline parallelization stage includes a first pipeline parallelization stage, and at least one pipeline parallelization stage also includes a pipeline parallelization stage adjacent to the first pipeline parallelization stage. The communication relationships between the domains of the first pipeline parallelization stage are updated; and the communication relationships between the domains of the first pipeline parallelization stage and the domains of the pipeline parallelization stages adjacent to the first pipeline parallelization stage are updated.

[0158] For example, such as Figure 4 As shown in (a), it is assumed that the domain (DP) in pipeline parallel stage 0 is isolated. m If , then update the domains (DP0,PP0), (DP1,PP0), and (DP2,PP0) in the pipeline parallel stage 0. m-1 ,PP0) and domain (DP) m Update the communication relationships between unisolated domains in pipeline parallel phase 0 and pipeline parallel phase 1. Unisolated domains in pipeline parallel phase 1 include domain (DP0, PP1), domain (DP1, PP1), and domain (DP...). m-1 ,PP1) and domain (DP) m ,PP1).

[0159] For example, at least one pipeline parallel stage includes a second pipeline parallel stage, and at least one pipeline parallel stage also includes a pipeline parallel stage adjacent to the second pipeline parallel stage. The communication relationships between the domains of the second pipeline parallel stage are updated; and the communication relationships between the domains of the second pipeline parallel stage and the domains of the pipeline parallel stages adjacent to the second pipeline parallel stage are updated.

[0160] For example, such as Figure 4 As shown in (a), it is assumed that the domain (DP) in pipeline parallel stage 0 is isolated. m ,PP0) and the domain (DP) in pipeline parallel stage 2 m-1(PP2). Update the communication relationships of the unisolated domains in pipeline parallelization stage 0, and the communication relationships between the unisolated domains in pipeline parallelization stage 0 and the unisolated domains in pipeline parallelization stage 1. Update the communication relationships of the unisolated domains in pipeline parallelization stage 2, and the communication relationships between the unisolated domains in pipeline parallelization stage 2 and the unisolated domains in pipeline parallelization stage 1, and the communication relationships between the unisolated domains in pipeline parallelization stage 2 and the unisolated domains in pipeline parallelization stage 3.

[0161] After updating the communication relationships between domains that are associated with at least one pipelined parallel stage, update the communication relationships between dedicated processors that carry at least one pipelined parallel stage, i.e., establish a link between dedicated processors corresponding to the domains of at least one pipelined parallel stage.

[0162] For example, such as Figure 6 As shown, the devices corresponding to the five parallel pipeline stages are connected normally, and model training is proceeding smoothly. The devices corresponding to the parallel pipeline stages are configured with a Collective Communication Library (CCL). The CCL implements relevant interfaces including point-to-point (P2P) communication (such as SEND, RECV, etc.) and collective communication (such as REDUCE, BROADCAST, ALLREDUCE, etc.). Data communication between multiple dedicated processors is efficient and reliable.

[0163] After domain isolation is implemented in pipeline parallel phase 2, local link breaks occur, specifically between devices in communication group 2 and communication group 3, and between devices in communication group 2 and communication group 1. Without restarting model training, dynamic scaling down re-establishes links between devices in communication group 2 and communication group 3, and between devices in communication group 2 and communication group 1, reducing MTTR overhead. Model training resumes in a degraded state after the computation graph is updated.

[0164] The first domain carries one or more layers in the model; other domains besides the first domain carry one or more layers in the model during the first pipeline parallelization stage; the first domain carries one or more identical layers in the model with other domains; other domains besides the first domain share the functions of the first domain during the first pipeline parallelization stage.

[0165] In a second possible implementation, dependencies are also used to indicate the data distribution used for model training, which includes training data or intermediate activations. Updating dependencies on domains associated with at least one domain involves performing data resplitting on the data used for model training of the domains in at least one pipeline parallel stage.

[0166] The data used for model training is divided into domains in the first parallel pipeline stage based on the number of domains in the pipeline stage adjacent to the first parallel pipeline stage.

[0167] For example, the first parallel pipeline stage is adjacent to the second parallel pipeline stage. Based on the number of domains in the second parallel pipeline stage, the domains of the first parallel pipeline stage are partitioned for model training. The domains of the second parallel pipeline stage use the data output from the first parallel pipeline stage for model training.

[0168] Based on the number of domains in the first pipeline parallel stage, the data used for model training is divided into domains in the pipeline parallel stage adjacent to the first pipeline parallel stage.

[0169] For example, the first pipeline parallel stage is adjacent to the third pipeline parallel stage. Based on the number of domains in the first pipeline parallel stage, the domains in the third pipeline parallel stage are divided for model training data, while the domains in the first pipeline parallel stage use the data output from the third pipeline parallel stage for model training.

[0170] For example, such as Figure 7 As shown, the computation graph includes four data parallel paths and five pipeline parallel stages. Each stage contains four domains. After isolating one domain in PP2, the communication relationships between PP2 and PP1 and PP3 are broken, and communication relationships are established between the domains of PP2 and the domains of PP1 and PP3, respectively. Data resplitting is performed on the data used for model training in the three domains of PP2. For example, the intermediate activations of the three domains included in PP2 are evenly distributed to the four domains of PP3. The intermediate activations of the four domains of PP1 are evenly distributed to the three domains of PP2.

[0171] In other embodiments, when the memory pressure of each domain in the first pipeline parallelization stage is high, other domains in the first pipeline parallelization stage besides the first domain cannot share the functionality of the first domain. Therefore, the parameters of the first domain are migrated to other pipeline parallelization stages with lower memory pressure. These parameters include model parameters and / or optimizer parameters. The first pipeline parallelization stage to which the first domain belongs is in a degraded state.

[0172] Optionally, the PP scheduler periodically reports the memory state and computation graph of a domain to the central scheduler. The central scheduler determines the domain transfer strategy based on the memory state and computation graph. For example, the strategy indicates the parameters of the domain that will carry it.

[0173] For example, based on the memory state of the fourth pipeline parallelization stage, the parameters of the first domain are migrated to a domain in the fourth pipeline parallelization stage. The memory state of the fourth pipeline parallelization stage is superior to the memory state of the first pipeline parallelization stage.

[0174] For example, such as Figure 8 As shown in (a), the first domain is assumed to be the domain where the data parallel path m and the pipeline parallel stage 0 intersect (DP). m ,PP0). Because the memory pressure is high in pipeline parallel stage 0 and low in pipeline parallel stage n, the domain (DP) m The parameters of PP0 are transferred to a domain in the pipeline parallel stage n. For example, the parameters of domain (DP) are transferred to a domain in the pipeline parallel stage n. m The parameters of PP0 are migrated to the domain where the data parallel path m and the pipeline parallel stage n intersect (DP). m PP n The pipeline parallel phase 0 includes the domain (DP). m PP n ).

[0175] like Figure 8 As shown in (b), assume the first domain is the domain (DP2, PP1) where data parallel path 2 and pipeline parallel stage 1 intersect. Since pipeline parallel stage 1 has higher memory pressure than pipeline parallel stage n-1, the parameters of domain (DP2, PP1) are migrated to a domain within pipeline parallel stage n-1. For example, the parameters of domain (DP2, PP1) are migrated to the domain (DP2, PP1) where data parallel path 2 and pipeline parallel stage n-1 intersect. n-1 The pipeline parallel phase 1 includes domains (DP2, PP). n-1 ).

[0176] For example, based on the network distance between each domain in the first and fourth pipeline parallel stages, the parameters of the first domain can be migrated to a domain in the fourth pipeline parallel stage.

[0177] For example, parameters of the first domain can be migrated to a domain in the fourth pipeline parallel phase based on the network distance between the device carrying the first domain and the device carrying each domain in the fourth pipeline parallel phase.

[0178] If the network distance is too large, causing a delay in data transmission, the parameters of the first domain will be migrated to a domain with the smallest possible network distance to ensure data transmission speed.

[0179] For example, such as Figure 8 As shown in (b), the domain (DP2,PP1) is close to the domain (DP2,PP2). The parameters of the domain (DP2,PP1) are moved to the domain (DP2,PP2) where data parallel path 2 and pipeline parallel stage 2 intersect. Similarly, the domain (DP2,PP1) is close to the domain (DP1,PP2). The parameters of the domain (DP2,PP1) are moved to the domain (DP1,PP2) where data parallel path 1 and pipeline parallel stage 2 intersect.

[0180] By transferring isolated domains, the complexity of subsequent upgrade / downgrade scheduling algorithms can be reduced, or the system reliability of downgrade training itself can be improved.

[0181] In other embodiments, the domain associated with the isolated first domain also includes the domain of the pipeline parallel stage to which the first domain migrates, and at least one of the domains of the pipeline parallel stage adjacent to the domain of the pipeline parallel stage to which the first domain migrates.

[0182] At least one pipeline parallelization stage associated with the first domain also includes a pipeline parallelization stage to which the first domain is migrated, and at least one of the pipeline parallelization stages adjacent to the pipeline parallelization stage to which the first domain is migrated.

[0183] After migrating the parameters of the first domain to a domain in the fourth pipeline parallel stage, update the communication relationships between the domains in the fourth pipeline parallel stage other than the domain to which the parameters of the first domain were migrated; update the communication relationships between the domains in the fourth pipeline parallel stage other than the domain to which the parameters of the first domain were migrated and the domains of the adjacent pipeline parallel stages.

[0184] The first pipeline parallel phase contains a domain from the fourth pipeline parallel phase. Update the communication relationships between the unisolated domains (excluding the first domain) in the first pipeline parallel phase and a domain from the fourth pipeline parallel phase; update the communication relationships between the unisolated domains (excluding the first domain) in the first pipeline parallel phase and a domain from the fourth pipeline parallel phase, and between unisolated domains in pipeline parallel phases adjacent to the first pipeline parallel phase.

[0185] Optionally, if the domain to which the parameters of the first domain are migrated in the fourth pipeline parallelization stage also carries a layer of the model in the fourth pipeline parallelization stage (i.e., the domain to which the parameters of the first domain are migrated is the same as the layer of the model carried by other domains in the fourth pipeline parallelization stage), then it is not necessary to update the communication relationship between the domains in the fourth pipeline parallelization stage. Alternatively, the communication relationship between the domain to which the parameters of the first domain are migrated in the fourth pipeline parallelization stage and the domains in the first pipeline parallelization stage can be updated.

[0186] For example, such as Figure 8 As shown in (a), the isolation domain (DP) m ,PP0), will the domain (DP) m Parameters of PP0) are transferred to the domain (DP). m PP n The pipeline parallel stage 0, except for the domain (DP) m Communication relationships are established outside the domain (DP0). In the pipeline parallel phase 0, communication relationships are established outside the domain (DP0). m The domain (DP0) establishes communication relationships with the domain in the parallel phase 1 of the pipeline, in addition to the domain (DP0).m Domains outside of PP0 include domains (DP) m PP n The pipeline parallel stage n, besides the domain (DP) m PP n Establish communication relationships with domains other than DP. In the pipeline parallel phase n, besides the domain (DP... m PP n The domain outside of the pipeline establishes a communication relationship with the domain of the parallel stage n-1.

[0187] like Figure 8 As shown in (b), the domain (DP2,PP1) is isolated, and the parameters of the domain (DP2,PP1) are migrated to the domain (DP2,PP1). n-1 In pipeline parallelization stage 1, communication relationships are established between all domains except those in domain (DP2, PP1). These domains, excluding those in domain (DP2, PP1), establish communication relationships with domains in pipeline parallelization stage 0, and with domains in pipeline parallelization stage 2. The domains in pipeline parallelization stage 1, excluding those in domain (DP2, PP1), include those in domain (DP2, PP1). n-1 In the parallel pipeline stage n-1, besides (DP2, PP) n-1 Establish communication relationships with domains other than (DP2, PP). In the pipeline parallel stage n-1, excluding (DP2, PP)... n-1 The domains other than (DP2, PP) establish communication relationships with the domains of pipeline parallelization stage n-2, and the domains of pipeline parallelization stage n-1 other than (DP2, PP) establish communication relationships with the domains of pipeline parallelization stage n-2. n-1 The domain outside of the pipeline parallel stage n establishes a communication relationship with the domain of the pipeline parallel stage n.

[0188] In other embodiments, performing data resegmentation on the data used for model training of the domains in at least one pipeline parallel stage further includes segmenting the data used for model training of the domains in the fourth pipeline parallel stage based on the number of domains in the pipeline parallel stage adjacent to the fourth pipeline parallel stage.

[0189] For example, the fourth pipeline parallel stage is adjacent to the fifth pipeline parallel stage. Based on the number of domains in the fifth pipeline parallel stage, the domains of the fourth pipeline parallel stage are divided for model training, while the domains of the fifth pipeline parallel stage use the data output from the fourth pipeline parallel stage for model training.

[0190] In other embodiments, performing data resegmentation on the data used for model training of the domains in at least one pipeline parallel stage further includes, based on the number of domains in the fourth pipeline parallel stage, segmenting the data used for model training of the domains in the pipeline parallel stages adjacent to the fourth pipeline parallel stage.

[0191] For example, the fourth pipeline parallel stage is adjacent to the sixth pipeline parallel stage. Based on the number of domains in the fourth pipeline parallel stage, the domains of the sixth pipeline parallel stage are divided for model training data, and the domains of the fourth pipeline parallel stage use the data output from the sixth pipeline parallel stage for model training.

[0192] The PP domain scheduler and the central scheduler determine the initial decision for computation graph scheduling based on the memory state, namely, establishing communication relationships and data resharding based on isolated domains. To reduce the overhead of establishing local communication relationships and data resharding, the failure of communication relationships between some domains and their upstream and downstream components in the pipeline parallelization stage is limited, while communication relationships between other domains are kept reusable before and after flexible upgrades and downgrades.

[0193] In a third possible implementation, updating the parameter distribution information of the domains related to at least one domain includes updating the dependencies of the domains related to at least one domain, resulting in a third computation graph; parameter resplitting is then performed based on the metrics obtained from model training using the third computation graph. Parameter resplitting is used to instruct the resplitting of model parameters and / or optimizer parameters among multiple domains, excluding at least one domain. Target metrics include memory state and model training performance metrics, including memory availability (MFU), domain communication time, and computation time.

[0194] For example, if the metrics obtained from model training based on the third computation graph do not meet the target conditions, parameter resplitting is performed according to the metrics of the third computation graph. The target conditions include similar MFU, memory state, communication duration, and computation duration for each pipeline parallel stage, or similar MFU, memory state, communication duration, and computation duration for each domain. This results in higher MFU, similar memory usage, lower average communication duration, and lower average computation duration across multiple pipeline parallel stages, achieving a system equilibrium.

[0195] If the metrics obtained from model training based on the third computation graph do not meet the target conditions, parameter resplitting is performed according to the metrics of the third computation graph. The target metrics include memory state and model training performance metrics, which include MFU, domain communication duration, and computation duration.

[0196] The metrics in the third computational graph include those for each pipeline parallelization stage. It is determined whether the metrics for each pipeline parallelization stage meet the target metrics. If any pipeline parallelization stage fails to meet any of the target metrics, parameter resplitting is performed on that pipeline parallelization stage.

[0197] For example, due to the isolation of the first domain in the first pipeline parallelization stage, the domains in the first pipeline parallelization stage experience high memory pressure, low MFU (Memory Function Requirement), and long communication or computation times. The parameters of the domains other than the first domain in the first pipeline parallelization stage can be migrated to the pipeline parallelization stage adjacent to the first pipeline parallelization stage.

[0198] For example, the metrics of the third computation graph include the metrics of each domain in each pipeline parallelization stage. It is determined whether the metrics of each domain in each pipeline parallelization stage meet the target metrics. If the metrics of any domain in any pipeline parallelization stage do not meet any of the target metrics, parameter resplitting is performed on that domain.

[0199] For example, in the first pipeline parallelization stage, a domain other than the first domain may have high memory pressure, low MFU (Memory Validation Function), long communication time, or long computation time. The parameters of the domain other than the first domain in the first pipeline parallelization stage can be migrated to a pipeline parallelization stage adjacent to the first pipeline parallelization stage.

[0200] Optionally, some parameters of domains other than the first domain in the first pipeline parallelization stage can be migrated to a pipeline parallelization stage adjacent to the first pipeline parallelization stage. For example, if the model training metrics obtained by performing model training based on the third computation graph do not meet the target metrics, some parameters of domains other than the first domain in the first pipeline parallelization stage can be migrated to a pipeline parallelization stage adjacent to the first pipeline parallelization stage.

[0201] For example, such as Figure 9 As shown in (a), after isolating the domain (DP3,PP2) in PP2, the parameters of the domain (DP2,PP2) in PP2 are transferred to the domain (DP3,PP3) in PP3, and the domain (DP3,PP3) in PP3 carries the parameters of the domain (DP2,PP2) in PP2 for model training.

[0202] For example, the parameters of the domain in the seventh pipeline parallel stage can be migrated to a pipeline parallel stage adjacent to the seventh pipeline parallel stage. The seventh pipeline parallel stage is any pipeline parallel stage in the third computation graph other than the first pipeline parallel stage. For instance, if the model training metrics obtained from training the model based on the third computation graph do not meet the target metrics, the parameters of the domain in the seventh pipeline parallel stage can be migrated to a pipeline parallel stage adjacent to the seventh pipeline parallel stage.

[0203] For example, such as Figure 9 As shown in (a), the parameters of the domain (DP1,PP0) in PP0 are transferred to the domain (DP1,PP1) in PP1, and the domain (DP1,PP1) in PP1 carries the parameters of the domain (DP1,PP0) in PP0 for model training.

[0204] Optionally, parameter resplitting can be performed on a portion of the parameters in a domain, that is, migrating a portion of the parameters in a domain to a domain in an adjacent pipeline parallel stage.

[0205] Optionally, if the model training metrics obtained from training the model based on the third computation graph meet the target metrics, there is no need to perform parameter resplitting based on the metrics of the third computation graph. Multiple dedicated processors train the artificial intelligence model based on the third computation graph.

[0206] Optionally, after resegmenting the parameters based on the metrics of the third computation graph, a fourth computation graph is obtained. If the model training metrics obtained by training the model based on the fourth computation graph meet the target metrics, there is no need to resegment the parameters based on the metrics of the fourth computation graph. Multiple dedicated processors train the artificial intelligence model based on the fourth computation graph.

[0207] If the model training metrics obtained from training the model based on the fourth computation graph do not meet the target metrics, the parameters are re-segmented according to the metrics of the fourth computation graph.

[0208] like Figure 9 As shown in (b), the general-purpose processor determines whether the model training metrics meet the target metrics, decides whether to perform parameter resplitting, and updates the computation graph. After i iterations of training, this continues until the updated computation graph's metrics meet the target metrics. The artificial intelligence model is then trained based on the updated computation graph.

[0209] After training the model based on the updated computation graph, the parameters are re-segmented according to the metrics based on the updated computation graph, dividing the different parts of the model in real time to ensure that the model training can flexibly adapt to the available hardware performance.

[0210] It should be noted that, assuming the metrics obtained from model training based on the updated computation graph meet the target metrics, the metrics of each domain in the updated computation graph also meet the target metrics. Furthermore, the memory state, MFU, communication duration, and computation duration of each domain in the updated computation graph reach a stable state, ensuring that the model training task can flexibly adapt to the available hardware performance.

[0211] In other embodiments, after isolating the first domain and then restoring it, the general-purpose processor updates the dependencies and / or parameter distribution information of the domains related to the first domain to obtain an updated computation graph. The updated computation graph is used to describe the dependencies and parameter distribution information between multiple domains other than the isolated domain. The domains other than the isolated domain include the first domain. The metrics of the updated computation graph meet the target metrics. Multiple dedicated processors train an artificial intelligence model based on the updated computation graph.

[0212] For example, after the first domain is restored, the domains that were not isolated in the first pipeline parallel phase include the first domain.

[0213] For example, if the first domain is migrated to the fourth pipeline parallel stage, and the first domain is restored, the fourth pipeline parallel stage will include the first domain.

[0214] For the dependency relationships and / or parameter distribution information of the domains related to updates and recovery in the scaling-up scenario, please refer to the description of the scaling-down scenario above.

[0215] In this application, it is assumed that a second computation graph is obtained by updating the dependency relationships and / or parameter distribution information between domains related to at least one domain based on a first computation graph.

[0216] Step 530: Multiple dedicated processors train an artificial intelligence model based on the second computation graph.

[0217] The second computation graph is used to describe the dependencies and parameter distribution information among multiple domains, excluding at least one domain. The metrics of the second computation graph meet the target metrics.

[0218] The data processing method provided in this application is widely applicable to various scenarios requiring large-scale model training, especially in training ultra-large-scale clusters where high reliability, continuity, and resource utilization are critical. This application offers an efficient and reliable solution for distributed hybrid parallel training by achieving completely restart-free elastic training, reducing MTTR, and improving MFU (Mean Time Between Training and Training). It is suitable for large-scale pre-training scenarios in various enterprises and cloud service platforms. It also enables hierarchical scheduling of ultra-large-scale systems, ensuring the efficiency and reliability of elastic training during system upgrades and downgrades.

[0219] This application presents a scheme for minimizing fault tolerance overhead in large model training. It uses the reconstruction chain within the same pipeline parallel stage as the starting point for load balancing, based on the smallest fault domain, and further narrows the isolation domain range according to the refinement of the fault location. It aims to support the reliability requirements of elastic training technologies for large models with scales of 100,000 to 1 million GPUs in the future, as well as the elastic resource scaling requirements for training task scheduling in clusters of corresponding sizes. The scheme's workflow is applicable to different model sizes and distributed training algorithms.

[0220] It is understood that, in order to achieve the functions in the above embodiments, the computer device includes hardware structures and / or software modules corresponding to the execution of each function. Those skilled in the art should readily recognize that, based on the units and method steps described in conjunction with the embodiments disclosed in this application, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed by hardware or by computer software driving hardware depends on the specific application scenario and design constraints of the technical solution.

[0221] The above text combines Figures 1 to 9 The data processing method provided in this application is described in detail below, and will be combined with... Figure 10 This application describes the apparatus provided according to the present application. These apparatuses are used to implement the functions of the dedicated processor or general-purpose processor in the above-described method embodiments, and therefore also achieve the beneficial effects of the above-described method embodiments. In this embodiment, the apparatus is as follows: Figure 5 The processor shown is either a dedicated or general-purpose processor, or a module (such as a chip) used in computer equipment.

[0222] like Figure 10 As shown, the data processing device 1000 includes a communication module 1001, a training module 1002, a computation graph update module 1003, and a storage module 1004.

[0223] The data processing device 1000 is used to implement the above. Figure 5 The method embodiments shown illustrate the functions of a general-purpose processor or a special-purpose processor.

[0224] Training module 1002 is used to train an artificial intelligence model. For example, training module 1002 is used to execute... Figure 5 Steps 510 and 530.

[0225] The computation graph update module 1003 is used to update the dependency relationships and / or parameter distribution information of domains related to at least one domain, while isolating at least one domain. For example, the computation graph update module 1003 is used to perform... Figure 5 Step 520.

[0226] Optionally, the communication module 1001 is used to transmit parameters.

[0227] Storage module 1004 is used to store computation graphs, parameters, memory status, model training performance metrics, etc., to facilitate model training.

[0228] It should be understood that the data processing apparatus 1000 in this application embodiment is implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD can be a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof. It can also be implemented using software. Figure 5The method shown, and its various modules, are also software modules, as are the data processing device 1000 and its various modules.

[0229] The data processing apparatus 1000 according to the embodiments of this application can be configured to execute the methods described in the embodiments of this application, and the above and other operations and / or functions of each unit in the data processing apparatus 1000 are respectively for implementing Figure 5 For the sake of brevity, the corresponding processes of each method in the code will not be elaborated here.

[0230] Figure 11 A schematic diagram of the structure of a computer device 1100 provided in this application. (See attached diagram.) Figure 11 As shown, computer device 1100 includes processor 1110, bus 1120, memory 1130, communication interface 1140, main memory 1150 (also called main memory unit) and processor 1160. Processor 1110, processor 1160, memory 1130, main memory 1150 and communication interface 1140 are connected via bus 1120.

[0231] It should be understood that in this embodiment, processor 1110 is a CPU, but it can also be other general-purpose processors, digital signal processors (DSPs), ASICs, FPGAs, or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor is a microprocessor or any conventional processor.

[0232] Computer device 1100 also includes a graphics processing unit (GPU), a neural network processing unit (NPU), a microprocessor, an ASIC, or one or more integrated circuits for controlling the execution of programs according to the present application. For example, processor 1160 is a GPU or an NPU.

[0233] The communication interface 1140 is used to enable communication between the computer device 1100 and external devices or components.

[0234] In this application, computer device 1100 is used to implement Figure 5 The dedicated processor shown in the diagram uses communication interface 1140 for transmitting strategies, etc., so that processor 1160 can train a model based on the strategy.

[0235] Computer device 1100 is used to implement Figure 5As shown in the diagram, the general-purpose processor's communication interface 1140 is used to transmit health status, performance metrics, etc., so that the processor 1160 can update the computation graph in the event of a domain failure.

[0236] Bus 1120 includes a pathway for transferring information between the aforementioned components (such as processor 1110, memory 1150, and storage 1130). In addition to the data bus, bus 1120 also includes a power bus, control bus, and status signal bus. However, for clarity, all buses are labeled as bus 1120 in the diagram. Bus 1120 is a Peripheral Component Interconnect Express (PCIe) bus, or an Extended Industry Standard Architecture (EISA) bus, a unified bus (Ubus or UB), a compute express link (CXL), a cachecoherent interconnect for accelerators (CCIX), etc. Bus 1120 is divided into address bus, data bus, and control bus.

[0237] As an example, computer device 1100 includes multiple processors. A processor is a multi-core (multi-CPU) processor. Here, a processor refers to one or more devices, circuits, and / or computing units used to process data (e.g., computer program instructions).

[0238] It is worth noting that, Figure 11 Taking computer device 1100, which includes one processor 1110 and one memory 1130, as an example, the processor 1110 and memory 1130 are used to indicate a type of device or equipment. In specific embodiments, the number of each type of device or equipment is determined according to business requirements. For example, computer device 1100 includes multiple GPUs or NPUs.

[0239] Memory 1150 can be non-transitory memory, which can be a pool of volatile memory or a pool of non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory is random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous linked dynamic random access memory (SLDRAM), and direct rambus RAM (DR RAM). Memory 1150 is used to store computation graphs, parameters, memory states, model training performance metrics, etc.

[0240] The memory 1130 corresponds to the storage medium used in the above method embodiment to store information such as computation graph, parameters, memory status, and model training performance indicators, for example, a disk, such as a mechanical hard disk or a solid-state hard disk.

[0241] The aforementioned computer device 1100 may be a general-purpose device or a special-purpose device. For example, computer device 1100 may also be a server or other device with computing capabilities.

[0242] It should be understood that the computer device 1100 according to this embodiment may correspond to the data processing device 1000 in this embodiment, and to the device executing the data processing device according to this embodiment. Figure 5 The corresponding subject in any of the methods, and the above and other operations and / or functions of each module in the data processing apparatus 1000 are respectively for the purpose of implementing Figure 5 For the sake of brevity, the corresponding processes of each method in the code will not be elaborated here.

[0243] The method steps in this embodiment can be implemented in hardware or by a processor executing software instructions. The software instructions can consist of corresponding software modules, which can be stored in random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, portable hard disks, CD-ROMs, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor, enabling the processor to read information from and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and storage medium can reside in an ASIC. Alternatively, the ASIC can reside in a computing device. Of course, the processor and storage medium can also exist as discrete components in the computing device.

[0244] Some embodiments of this application provide a computer-readable storage medium (e.g., a non-transitory computer-readable storage medium) storing computer program instructions that, when executed on a computer, cause the computer to perform one or more steps of a data processing method as described in any of the above embodiments.

[0245] For example, the aforementioned computer-readable storage media include, but are not limited to: magnetic storage devices (e.g., hard disks, floppy disks, or magnetic tapes), optical discs (e.g., CDs (Compact Disks), DVDs (Digital Versatile Disks), etc.), smart cards, and flash memory devices (e.g., EPROMs (Erasable Programmable Read-Only Memory), cards, sticks, or key drives, etc.). The various computer-readable storage media described in the embodiments of this application may represent one or more devices and / or other machine-readable storage media for storing information. The term "machine-readable storage medium" may include, but is not limited to, wireless channels and various other media capable of storing, containing, and / or carrying instructions and / or data.

[0246] Some embodiments of this application also provide a computer program product. This computer program product includes computer program instructions carried on a non-transitory computer-readable storage medium, which, when executed on a computer, cause the computer to perform one or more steps of the data storage method as described in the above embodiments.

[0247] The beneficial effects of the computer-readable storage medium and computer program product described above are the same as those of the data storage method described in some of the above embodiments, and will not be repeated here.

[0248] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer programs or instructions. When the computer program or instructions are loaded and executed on a computer, the processes or functions described in the embodiments of this application are performed entirely or partially. The computer can be a general-purpose computer, a special-purpose computer, a computer network, a network device, a user equipment, or other programmable device. The computer program or instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another. For example, the computer program or instructions can be transferred from one website, computer, server, or data center to another website, computer, server, or data center via wired or wireless means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium, such as a floppy disk, hard disk, or magnetic tape; it can also be an optical medium, such as a digital video disc (DVD); or it can be a semiconductor medium, such as a solid-state drive (SSD). The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A data processing method, characterized in that, Applied to a computing system, the computing system including a general-purpose processor and multiple dedicated processors, the method includes: The multiple dedicated processors train an artificial intelligence model based on a first computation graph, which is used to describe the dependencies of multiple domains of the artificial intelligence model and parameter distribution information. The parameter distribution information is used to indicate the distribution of model parameters and / or optimizer parameters in the multiple domains. In the case of isolating at least one domain, the general-purpose processor updates the dependency relationships and / or parameter distribution information between the domains related to the at least one domain to obtain a second computation graph. The second computation graph is used to describe the dependency relationships and parameter distribution information between the domains other than the at least one domain in the plurality of domains. The metrics of the second computation graph meet the target metrics, which include the mean recovery time (MTTR) and the model utilization rate (MFU). The multiple dedicated processors train the artificial intelligence model based on the second computation graph.

2. The method according to claim 1, characterized in that, The dependency relationship is used to indicate the communication relationship between the multiple domains; Updating the dependencies of domains related to the at least one domain includes: Update the communication relationships between domains in at least one pipeline parallel stage associated with the at least one domain.

3. The method according to claim 2, characterized in that, The at least one domain includes a first domain, which belongs to a first pipeline parallel stage; the at least one pipeline parallel stage includes the first pipeline parallel stage, and the at least one pipeline parallel stage also includes a pipeline parallel stage adjacent to the first pipeline parallel stage. Updating the communication relationships between domains in at least one pipeline parallel stage associated with the at least one domain includes: Update the communication relationships between domains other than the first domain in the first pipeline parallel phase; Update the communication relationships between the domains of the first pipeline parallelization stage (excluding the first domain) and the domains of the pipeline parallelization stages adjacent to the first pipeline parallelization stage.

4. The method according to any one of claims 1-3, characterized in that, The dependency is also used to indicate the distribution of data used for model training, which includes training data or intermediate activations. Updating the dependencies of domains related to the at least one domain includes: Data resplitting is performed on the data used for model training of at least one pipeline parallel stage associated with the at least one domain.

5. The method according to claim 4, characterized in that, The at least one domain includes a first domain, which belongs to a first pipeline parallel stage, and the first pipeline parallel stage is adjacent to a second pipeline parallel stage. Perform data resplitting on the data used for model training of at least one pipeline parallel stage associated with the at least one domain, including: Based on the number of domains in the second pipeline parallel stage, the data used for model training in the domains of the first pipeline parallel stage is divided, and the domains of the second pipeline parallel stage use the data output from the first pipeline parallel stage for model training.

6. The method according to claim 4 or 5, characterized in that, The first parallel phase of the pipeline is adjacent to the third parallel phase of the pipeline; Perform data resplitting on the data used for model training of at least one pipeline parallel stage associated with the at least one domain, including: Based on the number of domains in the first pipeline parallel stage, the domains in the third pipeline parallel stage are divided for model training data, and the domains in the first pipeline parallel stage use the data output by the third pipeline parallel stage for model training.

7. The method according to any one of claims 3-6, characterized in that, The method further includes: The parameters of the first domain are migrated to a domain in the fourth pipeline parallel stage, the parameters including model parameters and / or optimizer parameters; Updating the communication relationships between domains of at least one pipeline parallel stage associated with the at least one domain further includes: Update the communication relationships between domains in the fourth pipeline parallel phase, excluding the domain to which the parameters of the first domain are migrated. Update the communication relationship between the domains of the fourth pipeline parallel stage (excluding the domain to which the parameters of the first domain are migrated) and the domains of the adjacent pipeline parallel stages.

8. The method according to claim 7, characterized in that, Migrating the parameters of the first domain to a domain in the fourth pipeline parallel stage includes: Based on the memory state of the fourth pipeline parallel stage, the parameters of the first domain are migrated to a domain in the fourth pipeline parallel stage, where the memory state of the fourth pipeline parallel stage is superior to that of the first pipeline parallel stage.

9. The method according to claim 7 or 8, characterized in that, Migrating the parameters of the first domain to a domain in the fourth pipeline parallel stage includes: Based on the network distance between the first domain and the fourth pipeline parallel stage, the parameters of the first domain are migrated to a domain in the fourth pipeline parallel stage.

10. The method according to any one of claims 7-9, characterized in that, The fourth parallel pipeline stage is adjacent to the fifth parallel pipeline stage; Performing data resplitting on the data used for model training of at least one pipeline parallel stage associated with the at least one domain, further includes: Based on the number of domains in the fifth pipeline parallel stage, the domains of the fourth pipeline parallel stage are divided for model training data, and the domains of the fifth pipeline parallel stage use the data output from the fourth pipeline parallel stage for model training.

11. The method according to claim 10, characterized in that, The fourth parallel pipeline stage is adjacent to the sixth parallel pipeline stage; Performing data resplitting on the data used for model training of at least one pipeline parallel stage associated with the at least one domain, further includes: Based on the number of domains in the fourth pipeline parallel stage, the domains of the sixth pipeline parallel stage are divided for model training data, and the domains of the fourth pipeline parallel stage use the data output by the sixth pipeline parallel stage for model training.

12. The method according to any one of claims 1-11, characterized in that, Updating the parameter distribution information of the domains associated with the at least one domain includes: Update the dependencies of the domains related to the at least one domain to obtain a third computation graph; The parameters are resegmented based on the metrics obtained from model training according to the third computation graph. The parameter resegmentation is used to indicate the resegmentation of model parameters and / or optimizer parameters among the multiple domains, excluding the at least one domain.

13. The method according to claim 12, characterized in that, The metrics of the third computation graph include metrics for each pipeline parallelization stage; the at least one domain includes a first domain, which belongs to the first pipeline parallelization stage; The metrics execution parameters are re-splitting based on the model training obtained from the third computation graph, including: The parameters of the domains other than the first domain in the first pipeline parallelization stage are migrated to the pipeline parallelization stage adjacent to the first pipeline parallelization stage.

14. The method according to claim 13, characterized in that, The re-segmentation of the index execution parameters obtained from model training based on the third computation graph also includes: The parameters of the domain of the seventh pipeline parallel stage are migrated to the pipeline parallel stage adjacent to the seventh pipeline parallel stage, which is any pipeline parallel stage in the third computation graph other than the first pipeline parallel stage.

15. The method according to any one of claims 1-14, characterized in that, The at least one field includes a first field, and the method further includes: In the case of restoring the first domain, the general processor updates the dependency relationship and / or parameter distribution information of the domains related to the first domain to obtain a fourth computation graph. The fourth computation graph is used to describe the dependency relationship and parameter distribution information between the domains other than the isolated domains. The domains other than the isolated domains include the first domain. The index of the fourth computation graph satisfies the target index. The multiple dedicated processors train the artificial intelligence model based on the fourth computation graph.

16. A data processing apparatus, characterized in that, include: The training module is used to train an artificial intelligence model based on a first computation graph, wherein the first computation graph is used to describe the dependencies of multiple domains of the artificial intelligence model and the parameter distribution information, wherein the parameter distribution information is used to indicate the distribution of model parameters and / or optimizer parameters in the multiple domains; The computation graph update module is used to update the dependency relationships and / or parameter distribution information between domains related to the at least one domain while isolating at least one domain, to obtain a second computation graph. The second computation graph is used to describe the dependency relationships and parameter distribution information between domains other than the at least one domain in the plurality of domains. The metrics of the second computation graph meet the target metrics, including the mean recovery time (MTTR) and the model computational efficiency (MFU). The training module is also used to train the artificial intelligence model based on the second computation graph.

17. The apparatus according to claim 16, characterized in that, The dependency relationship is used to indicate the communication relationship between the multiple domains; When the computation graph update module updates the dependencies of domains related to the at least one domain, it is specifically used to update the communication relationships between domains of at least one pipeline parallel stage related to the at least one domain.

18. The apparatus according to claim 16 or 17, characterized in that, The dependency is also used to indicate the distribution of data used for model training, which includes training data or intermediate activations. When the computation graph update module updates the dependencies of the domains related to the at least one domain, it is specifically used to perform data resegmentation on the data used for model training of the domains of at least one pipeline parallel stage related to the at least one domain.

19. The apparatus according to any one of claims 16-18, characterized in that, The computation graph update module is also used to migrate the parameters of the first domain to a domain in the fourth pipeline parallel stage, the parameters including model parameters and / or optimizer parameters; When the computation graph update module updates the communication relationship between domains of at least one pipeline parallel stage related to the at least one domain, it is specifically used to update the communication relationship between domains of the fourth pipeline parallel stage other than the domain to which the parameters of the first domain are migrated. Update the communication relationship between the domains of the fourth pipeline parallel stage (excluding the domain to which the parameters of the first domain are migrated) and the domains of the adjacent pipeline parallel stages.

20. The apparatus according to any one of claims 16-19, characterized in that, When the computation graph update module updates the parameter distribution information of the domains related to the at least one domain, it is specifically used to update the dependencies of the domains related to the at least one domain to obtain a third computation graph; the parameters are re-splitting based on the metrics obtained from model training according to the third computation graph, and the parameter re-splitting is used to indicate the re-splitting of model parameters and / or optimizer parameters among the domains other than the at least one domain in the plurality of domains.

21. A computer device, characterized in that, The computer device includes a memory and a plurality of processors, the memory being used to store a set of computer instructions; when the processors execute the set of computer instructions, the plurality of processors jointly execute the operation steps of the method according to any one of claims 1-15.

22. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program or instructions, which, when executed by a data processing device, implement the method as described in any one of claims 1-15.

23. A computer program product, characterized in that, The computer program product includes a computer program or instructions, which, when executed by a communication device, implement the method as described in any one of claims 1-15.

Images