Model parameter processing method and device, storage medium and program product

Abstract

The invention provides a model parameter processing method and device, a storage medium and a program product, and the method comprises the steps: determining a second process from non-abnormal processes of a plurality of processes in response to the detection that a first process in the plurality of processes used for executing a distributed training task is abnormal, the multiple processes update parameters in different model parameter fragments of the distributed training task; distributing the model parameter fragments updated by the first process and the gradient fragments corresponding to the first process to a second process; and updating parameters in the model parameter fragments corresponding to the first process based on the gradient fragments corresponding to the first process through the second process. According to the method, by providing a model parameter fault-tolerant updating mechanism for the fault process, the training task can be prevented from being interrupted when the fault process occurs in the model training process, and the reliability and efficiency of model training are improved.

Description

Technical Field

[0001] This disclosure relates to the field of deep learning distributed training technology, specifically to the field of large-scale distributed training technology, and particularly to a model parameter processing method and apparatus, storage medium and program product. Background Technology

[0002] With the dramatic increase in the number of parameters during deep learning model training, traditional single-machine training methods can no longer meet the demands, making large-scale distributed training an inevitable choice. Zero Redundancy Optimizer (ZeRO) technology enables the training of large-scale models by optimizing memory usage. In large-scale distributed training, when employing the stage-one (ZeRO-1) strategy (also known as optimizer state sharding), the memory usage of a single Graphics Processing Unit (GPU) can be significantly reduced.

[0003] However, the ZeRO-1 strategy introduces the Single Point of Failure (SPOF) problem. If a process hangs or completely fails, the model parameters it is responsible for will not be updated. Since the update of model parameters is global, the absence of any part will lead to subsequent calculation errors, ultimately causing the entire training task to fail, or to roll back to the previous checkpoint and restart training, wasting computational resources.

[0004] Therefore, improving the reliability and efficiency of large-scale distributed training has become an urgent technical problem to be solved. Summary of the Invention

[0005] This disclosure provides a model parameter processing method, a model parameter processing device, an electronic device, a computer-readable storage medium, and a computer program product.

[0006] In a first aspect, embodiments of this disclosure propose a model parameter processing method, comprising: in response to detecting an anomaly in a first process among multiple processes used to perform a distributed training task, determining a second process from the non-abnormal processes of the multiple processes, wherein the multiple processes update parameters in different model parameter slices of the distributed training task respectively; allocating the model parameter slices updated by the first process and the gradient slices corresponding to the first process to the second process; and updating the parameters in the model parameter slices corresponding to the first process by the second process based on the gradient slices corresponding to the first process.

[0007] In some optional implementations of the embodiments of this disclosure, the above-mentioned updating of the parameters in the model parameter slice corresponding to the first process by the second process based on the gradient slice corresponding to the first process includes: updating the parameters in the model parameter slice corresponding to the first process by the second process based on the first optimizer and the gradient slice corresponding to the first process, wherein the first optimizer is an optimizer that does not rely on historical optimization information and / or the first optimizer is an optimizer whose video memory usage is less than or equal to a first video memory threshold.

[0008] In some optional implementations of the embodiments of this disclosure, the above-mentioned updating of the parameters in the model parameter slice corresponding to the first process by the second process based on the first optimizer and the gradient slice corresponding to the first process includes: in response to determining that the video memory remaining space corresponding to the second process is less than or equal to the second video memory threshold, updating the parameters in the model parameter slice corresponding to the first process by the second process based on the first optimizer and the gradient slice corresponding to the first process.

[0009] In some optional implementations of the embodiments of this disclosure, the first optimizer is configured with an independent learning rate and a learning rate scheduler.

[0010] In some optional implementations of the embodiments of this disclosure, the above-mentioned updating of the parameters in the model parameter slice corresponding to the first process by the second process based on the gradient slice corresponding to the first process includes: updating the parameters in the model parameter slice corresponding to the first process by the second process based on the second optimizer and the gradient slice corresponding to the first process, wherein the second optimizer is an optimizer that depends on historical optimization information and / or the second optimizer is an optimizer whose video memory usage is greater than a third video memory threshold.

[0011] In some optional implementations of the embodiments of this disclosure, the above-mentioned updating of the parameters in the model parameter slice corresponding to the first process by the second process based on the gradient slice corresponding to the second optimizer and the first process includes: in response to determining that the video memory remaining space corresponding to the second process is greater than the fourth video memory threshold, updating the parameters in the model parameter slice corresponding to the first process by the second process based on the gradient slice corresponding to the second optimizer and the first process.

[0012] In some optional implementations of the embodiments of this disclosure, the above-mentioned determination of the second process from the non-abnormal processes of multiple processes includes: determining the process with the fewest parameters to be updated among the non-abnormal processes adjacent to the first process as the second process; and the above-mentioned model parameter processing method further includes: synchronizing all model parameter fragments updated by the second process, including the updated model parameter fragments corresponding to the first process, to each non-abnormal process other than the second process based on the first set communication operation.

[0013] In some optional implementations of the embodiments of this disclosure, the above-mentioned determination of the second process from the non-abnormal processes of multiple processes includes: determining all remaining non-abnormal processes as the second process; and the above-mentioned model parameter processing method further includes: synchronizing other model parameter fragments updated by each second process, excluding the updated model parameter fragments corresponding to the first process, to other non-abnormal processes other than themselves based on the first set communication operation.

[0014] In some optional implementations of the embodiments of this disclosure, the above model parameter processing method further includes: reallocating consecutive process identifiers for non-abnormal processes among multiple processes; processing the optimizer state corresponding to the distributed training task according to a preset strategy to obtain optimizer state slices corresponding to each non-abnormal process after the process identifiers have been reallocated.

[0015] In some optional implementations of the embodiments of this disclosure, the above model parameter processing method further includes: obtaining the current full gradient information corresponding to the distributed training task in response to the end of a new backpropagation of the distributed training task, processing the current full gradient information based on the second set communication operation, and obtaining gradient slices corresponding to each non-abnormal process after the process identifier is reassigned.

[0016] Secondly, embodiments of this disclosure propose a model parameter processing apparatus, comprising: a determining unit, an allocating unit, and a processing unit. The determining unit is configured to, in response to detecting an anomaly in a first process among multiple processes performing a distributed training task, determine a second process from the non-abnormal processes of the multiple processes, wherein the multiple processes update parameters in different model parameter slices of the distributed training task, respectively; the allocating unit is configured to allocate the model parameter slices updated by the first process and the gradient slices corresponding to the first process to the second process; the processing unit is configured to, through the second process, update the parameters in the model parameter slices corresponding to the first process based on the gradient slices corresponding to the first process.

[0017] In some optional implementations of the embodiments of this disclosure, the above processing unit is further configured to: update the parameters in the model parameter slices corresponding to the first process through the second process based on the first optimizer and the gradient slices corresponding to the first process, wherein the first optimizer is an optimizer that does not rely on historical optimization information and / or the first optimizer is an optimizer whose video memory usage is less than or equal to a first video memory threshold.

[0018] In some optional implementations of the embodiments of this disclosure, the above processing unit is further configured to: in response to determining that the remaining video memory corresponding to the second process is less than or equal to the second video memory threshold, update the parameters in the model parameter slice corresponding to the first process through the second process based on the first optimizer and the gradient slice corresponding to the first process.

[0019] In some optional implementations of the embodiments of this disclosure, the first optimizer is configured with an independent learning rate and a learning rate scheduler.

[0020] In some optional implementations of the embodiments of this disclosure, the above-mentioned processing unit is further configured to: update the parameters in the model parameter slices corresponding to the first process through the second process based on the gradient slices corresponding to the second optimizer and the first process, wherein the second optimizer is an optimizer that depends on historical optimization information, and / or the second optimizer is an optimizer whose video memory usage is greater than a third video memory threshold.

[0021] In some optional implementations of the embodiments of this disclosure, the above processing unit is further configured to: in response to determining that the remaining video memory corresponding to the second process is greater than the fourth video memory threshold, update the parameters in the model parameter slice corresponding to the first process through the second process based on the second optimizer and the gradient slice corresponding to the first process.

[0022] In some optional implementations of the embodiments of this disclosure, the determining unit is further configured to: determine the process with the fewest parameters to be updated among the non-abnormal processes adjacent to the first process as the second process; and the processing unit is further configured to: synchronize all model parameter fragments updated by the second process, including the updated model parameter fragments corresponding to the first process, to each non-abnormal process other than the second process based on the first set communication operation.

[0023] In some optional implementations of the embodiments of this disclosure, the determining unit is further configured to: determine all remaining non-abnormal processes as second processes; and the processing unit is further configured to: synchronize other model parameter fragments updated by each second process, excluding the updated model parameter fragments corresponding to the first process, to other non-abnormal processes other than itself based on the first set communication operation.

[0024] In some optional implementations of the embodiments of this disclosure, the above processing unit is further configured to: reallocate consecutive process identifiers for non-abnormal processes among multiple processes; process the optimizer state corresponding to the distributed training task according to a preset strategy to obtain optimizer state slices corresponding to each non-abnormal process after the process identifiers are reallocated.

[0025] In some optional implementations of the embodiments of this disclosure, the above processing unit is further configured to: obtain the current full gradient information corresponding to the distributed training task in response to the end of a new backpropagation of the distributed training task, process the current full gradient information based on the second set communication operation, and obtain gradient slices corresponding to each non-abnormal process after the process identifier is reassigned.

[0026] Thirdly, embodiments of this disclosure provide an electronic device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, which, when executed by the at least one processor, enable the at least one processor to implement the model parameter processing method described in any of the implementations of the first aspect above.

[0027] Fourthly, embodiments of this disclosure provide a non-transitory computer-readable storage medium storing computer instructions that enable a processor, when executed, to implement the model parameter processing method as described in any implementation of the first aspect.

[0028] Fifthly, embodiments of this disclosure provide a computer program product including a computer program that, when executed by a processor, can implement the model parameter processing method as described in any implementation of the first aspect.

[0029] According to the solution provided in this disclosure, when a faulty process (i.e., the first process) occurs during the parallel execution of a distributed training task by multiple processes, a second process can be identified from the non-faulty processes (i.e., non-abnormal processes). The model parameter slices and their corresponding gradient slices that were originally the responsibility of the first process to update are then assigned to the second process. The second process then uses these gradient slices to update the parameters in the model parameter slices that were originally the responsibility of the first process, thus seamlessly taking over the model parameter update task from the faulty process. In this way, by providing a fault-tolerant update mechanism for model parameters in the case of a faulty process, the execution of the model training task can continue even when a faulty process occurs during model training, avoiding interruption of the training task. This improves the reliability and efficiency of large-scale distributed training, avoids unnecessary waste of computational resources, and increases computational resource utilization and training throughput.

[0030] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0031] Other features, objects, and advantages of this disclosure will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings: Figure 1 A schematic diagram of a model parameter update framework using the ZeRO-1 strategy for distributed training, provided in an embodiment of this disclosure; Figure 2 This is a schematic diagram of a normal parameter update process in a fault-free distributed training process provided in an embodiment of this disclosure; Figure 3 A flowchart of a model parameter processing method provided in this embodiment of the disclosure; Figure 4 This is a schematic diagram illustrating the process of rebuilding a communication process group according to an embodiment of the present disclosure; Figure 5 A flowchart of another model parameter processing method provided in this disclosure embodiment; Figure 6 A schematic diagram of a model parameter update framework for a faulty process in distributed training, provided in an embodiment of this disclosure; Figure 7 This is a schematic diagram of a parameter update process in distributed training of a faulty process, provided as an embodiment of the present disclosure. Figure 8 This is a schematic diagram of a distributed training framework for executing model parameter processing schemes in an application scenario, provided by an embodiment of this disclosure. Figure 9 A structural block diagram of a model parameter processing device provided in an embodiment of this disclosure; Figure 10 This is a schematic diagram of the structure of an electronic device suitable for performing a model parameter processing method, provided as an embodiment of the present disclosure. Detailed Implementation

[0032] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding; these should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description. It should be noted that, unless otherwise specified, the embodiments and features described in this disclosure can be combined with each other.

[0033] It should be noted that the collection, acquisition, storage, processing, transmission, provision, disclosure, and application of user personal information (such as user image information and biometric information to be collected in distributed training) involved in the technical solution disclosed herein are all performed with the user's knowledge and explicit authorization, comply with the provisions of relevant laws and regulations, and do not violate public order and good morals.

[0034] With the rapid increase in the number of parameters during deep learning model training, traditional single-machine training modes can no longer meet the demands, making large-scale distributed training an inevitable choice, such as distributed training in a multi-GPU cluster environment. ZeRO technology enables the training of large-scale models by optimizing memory usage, with the ZeRO-1 strategy being a commonly used strategy in model training. In large-scale distributed training, the ZeRO-1 strategy can be used to partition or split the optimizer state and distribute it across multiple parallel processes. Each process can correspond to one GPU, and each process is only responsible for updating and saving its corresponding optimizer state slice and corresponding model parameters, thereby significantly reducing GPU memory usage.

[0035] However, the working mode of the ZeRO-1 strategy introduces the SPOF problem. In the ZeRO-1 strategy, each process is a critical single point. If any process hangs or completely fails / malfunctions due to hardware failure, network interruption, software error, etc., the model parameters it is responsible for will not be updated. Since the update of model parameters is global, the absence of any part will lead to subsequent calculation errors. Furthermore, in subsequent training steps, the convergence of the model will be affected due to the missing optimizer state corresponding to the failed process, which may even cause the entire training task to fail and need to be restarted from the beginning. This will waste a lot of computing resources and time.

[0036] Therefore, improving the reliability and efficiency of large-scale distributed training has become an urgent technical problem to be solved.

[0037] The following description of several optional embodiments illustrates the technical solutions of this disclosure and the technical effects produced by these solutions. It should be noted that the following embodiments can be referenced, learned from, or combined with each other. Identical terms, similar features, and similar implementation steps in different embodiments will not be repeated.

[0038] In one optional implementation of this disclosure, a ZeRO-1 strategy is used for distributed training. The optimizer states (such as the momentum and variance of the adaptive moment estimation (Adam) optimizer or the decoupled weight decay (AdamW) optimizer) are partitioned and distributed to N (which can be integers greater than 1) processes. Figure 1 As shown, the optimizer state can be divided and distributed to four processes: process 0 (rank0, also known as rank 0), process 1 (rank1, also known as rank 1), process 2 (rank2, also known as rank 2), and process 3 (rank3, also known as rank 3). In mixed-precision training, the divided optimizer state can also include high-precision model parameters, such as single-precision floating-point (Floating-Point 32-bit, FP32) model parameters. Each process is responsible for storing and updating one and only one optimizer state shard (shard_i, 0≤i≤N-1) (such as a high-precision model parameter shard Pi and its corresponding momentum shard mi and variance shard vi).

[0039] like Figure 1 As shown, process rank0 stores high-precision model parameter slices P0 and P1, momentum slices m0 and m1, and variance slices v0 and v1. Other processes follow the same pattern, which will not be elaborated further. Simultaneously, each process contains all model parameters (which can be low-precision model parameters, such as FP16 model parameters) and gradient information. The full set of model parameters can be stored in parameter buckets or parameter grouping buckets, and the full set of gradient information can be stored in gradient buckets or gradient grouping buckets, such as... Figure 1 The complete set of model parameters 0 to 7 (Param0 to Param7) is stored in a parameter bucket or parameter grouping bucket (ParamBucket), and the complete set of gradient information grad0 to grad7 is stored in a gradient bucket or gradient grouping bucket (GradBucket). The complete set of model parameters on each process or GPU is used to perform backpropagation and forward propagation to calculate the local gradient. Then, gradient synchronization can be achieved through set communication operations such as merge-hash (or reduce-scatter), ensuring that each process or GPU has a copy of the complete gradient information or the complete global gradient information.

[0040] When updating the model parameter slices under its responsibility, each process extracts the gradient slices corresponding to the model parameter slices it is responsible for from the complete global gradient information stored therein, such as... Figure 1As shown, gradient slices G0 and G1 correspond to process rank0, and so on, which will not be elaborated further here. Then, each process uses the corresponding gradient slice to update the locally stored optimizer state (e.g., ...). Figure 1 The momentum slices m0 and m1, variance slices v0 and v1, etc. shown in the diagram, are then used based on the updated optimizer state, using parameter update amounts determined by, for example, Adam, and then using these parameter update amounts to slice the model parameters it is responsible for (e.g., ...). Figure 1 The parameter slices P0 and P1 shown in the figure are updated accordingly to obtain the updated model parameter slices (e.g., ...). Figure 1 The parameter partitions P0' and P1' shown can be referenced as follows: Figure 2 Step 201 is shown.

[0041] Then, since each process only stores the local optimizer state and not other optimizer state slices, the parameter slices corresponding to these unstored optimizer states are still the old versions locally. To perform a new round of training, after each process updates its assigned model parameter slices, before the next forward propagation, it can use an all-gather communication operation to broadcast its updated model parameter slices to other processes. See [example example]. Figure 2 Step 202, as shown, ensures that each process or GPU has an updated copy of the full set of model parameters (e.g., ...). Figure 1 The parameters 0' to 7' (Param0' to Param7') shown in the figure are then used to proceed to the next training round after the current iteration is completed. This can be referenced as follows: Figure 2 Step 203 is shown.

[0042] However, when a process exits training due to a failure in one step or round of distributed training, in order to achieve rapid fault-tolerant recovery of process failures in distributed training, the technical solutions described in the following embodiments can be used.

[0043] Please refer to Figure 3 , Figure 3 A flowchart of a model parameter processing method provided in this disclosure embodiment, wherein process 300 includes the following steps: Step 301: In response to detecting an anomaly in the first process among the multiple processes used to perform the distributed training task, determine the second process from the non-abnormal processes among the multiple processes.

[0044] In some optional implementations of the embodiments of this disclosure, the aforementioned multiple processes update the parameters in different model parameter slices of the distributed training task. In this embodiment, when the distributed training task is executed in parallel by multiple processes, the full set of model parameters of the distributed training task can be divided into model parameter slices corresponding to each of the multiple processes, and each process is responsible for updating its assigned portion of the model parameters. The types of parameters in the model parameter slices may include, but are not limited to, at least one of weights and biases.

[0045] In this embodiment, when multiple processes execute a distributed training task in parallel, and a failed process (the first process) occurs due to hardware failure, network interruption, software error, or other reasons, an operation can be performed to determine a second process from the non-failed processes (i.e., non-abnormal processes) among the multiple processes. This allows the model training task to continue, rather than restarting training by rolling back the training progress. This avoids a serious waste of computing resources. The non-abnormal process can refer to the remaining or available surviving processes other than the first process, i.e., processes that have not failed.

[0046] In some optional implementations of the embodiments of this disclosure, if an anomaly is detected in the first process among the multiple processes used to perform the distributed training task, the operation of determining the second process can be triggered by broadcasting a process failure message.

[0047] In some optional implementations of the embodiments of this disclosure, the multiple processes used to perform the distributed training task can be regarded as an old communication process group (or original communication process group (ProcessGroup)). All or some of the remaining non-abnormal processes, excluding the faulty process (including the first process), can be regarded as a new communication process group. The number of these non-abnormal processes can include one or more, for example, such as... Figure 4 As shown, the original communication process group consisting of processes 0 to 3 (i.e., rank0 to rank3) becomes a new communication process group consisting of processes 0' to 2' after process 1 (rank1) fails (or fails) and the process identifier is reallocated.

[0048] Step 302: Assign the model parameter slices updated by the first process and the gradient slices corresponding to the first process to the second process.

[0049] In this step, the task of updating the model parameter slices corresponding to the first process based on the gradient slices corresponding to the first process can be transferred to the second process.

[0050] Step 303: The second process updates the parameters in the model parameter slice corresponding to the first process based on the gradient slice corresponding to the first process.

[0051] In this embodiment, if the second process is determined from the remaining surviving processes other than the first process, the second process can take over the model parameter update task that was originally handled by the faulty first process. That is, the model parameter slices and their corresponding gradient slices that were originally handled by the first process can be assigned to the second process, and then the second process can use the gradient slices to update the parameters in the model parameter slices that should have been updated by the first process, thereby avoiding the interruption of the training task.

[0052] In some optional implementations of the embodiments of this disclosure, the process of the second process taking over the model parameter update task of the faulty process (i.e., the first process) may include: the second process identifying the model parameter slices to be updated by the first process and their corresponding gradient slices based on a pre-stored mapping relationship between slices and processes, and then allocating the model parameter slices to be updated by the first process and their corresponding gradient slices to the second process. The pre-stored mapping relationship between slices and processes may optionally be used to indicate the following: the correspondence between model parameter slices and processes, and the correspondence between gradient slices and processes.

[0053] According to the model parameter processing method provided in this disclosure, when a faulty process (i.e., the first process) occurs during the parallel execution of a distributed training task by multiple processes, a second process can be identified from the non-faulty processes (i.e., non-abnormal processes). The model parameter slices and their corresponding gradient slices that were originally the responsibility of the first process to update are then assigned to the second process. The second process then uses the gradient slices to update the parameters in the model parameter slices that were originally the responsibility of the first process, thus seamlessly taking over the model parameter update task from the faulty process. In this way, by providing a fault-tolerant update mechanism for model parameters in the case of a faulty process, the execution of the model training task can continue even when a faulty process occurs during model training, reducing the loss of training progress and preventing training task interruption. This improves the overall reliability, stability, and efficiency of large-scale distributed training, avoids unnecessary waste of computing resources, and increases the utilization of computing resources and training throughput.

[0054] Please refer to Figure 5 , Figure 5 A flowchart of another model parameter processing method provided in this disclosure embodiment, wherein process 500 includes the following steps: Step 501: In response to detecting an anomaly in the first process among the multiple processes used to perform the distributed training task, determine the second process from the non-abnormal processes among the multiple processes.

[0055] Step 502: Assign the model parameter slices updated by the first process and the gradient slices corresponding to the first process to the second process.

[0056] Steps 501-502 above are the same as those mentioned above. Figure 3 The steps 301-302 shown are the same. For the same parts, please refer to the corresponding parts of the previous embodiment. They will not be repeated here.

[0057] Step 503: The second process updates the parameters in the model parameter slices corresponding to the first process based on the objective optimizer and the gradient slices corresponding to the first process.

[0058] In this embodiment, when a process-level failure occurs during distributed training tasks, especially large-scale distributed training tasks, a second process can be identified from among the non-abnormal processes to take over the model parameter update task originally handled by the failed first process. Furthermore, if a target optimizer for parameter updates is identified, the second process can perform parameter update operations on the model parameter slices corresponding to the first process, based on the target optimizer and the gradient slices corresponding to the first process. Thus, by having a non-abnormal process efficiently take over the parameter update task corresponding to the abnormal process, rapid fault-tolerant recovery can be achieved to ensure the smooth progress of the distributed training task.

[0059] In different implementations of the embodiments of this disclosure, the target optimizer described above can be implemented as a corresponding optimizer to adapt to the fault tolerance and recovery requirements of faulty processes in different scenarios.

[0060] In some optional implementations of the embodiments of this disclosure, in the above... Figure 5 Based on the corresponding embodiment, the target optimizer can be a first optimizer, wherein the first optimizer can be an optimizer that does not rely on historical optimization information, and / or the first optimizer is an optimizer whose video memory usage is less than or equal to a first video memory threshold. Then, step 503 can be executed as follows: the second process updates the parameters in the model parameter slice corresponding to the first process based on the first optimizer and the gradient slice corresponding to the first process.

[0061] In this embodiment, an optimizer that does not rely on historical optimization information (such as momentum and / or variance) and / or whose video memory usage is less than or equal to a first video memory threshold can be used to update the parameters in the model parameter slices corresponding to the first process, which can effectively control the video memory usage.

[0062] Optionally, the above step of updating the parameters in the model parameter slice corresponding to the first process through the second process based on the first optimizer and the gradient slice corresponding to the first process can be implemented as follows: in response to determining that the video memory remaining amount corresponding to the second process is less than or equal to the second video memory threshold, the parameters in the model parameter slice corresponding to the first process are updated through the second process based on the first optimizer and the gradient slice corresponding to the first process.

[0063] In this embodiment, if it is determined that the GPU memory available for the second process, which is to take over the model parameter update task handled by the faulty process (i.e., the first process), is insufficient (i.e., less than or equal to the second GPU memory threshold), such as having reached its usage limit, the first optimizer can be selected to update the parameters in the model parameter slices corresponding to the first process. Thus, by bypassing the reliance on missing and complex historical optimization information, no additional GPU memory usage is added, and / or by controlling the optimizer's GPU memory usage, no significant increase in GPU memory overhead is achieved. Therefore, while fully considering the GPU memory available for the second process, rapid fault-tolerant recovery of model parameter updates handled by the faulty process in distributed training can be achieved, ensuring the smooth progress of the distributed training task.

[0064] It should be noted that the values ​​of the first and second memory thresholds mentioned above can be determined based on the specific circumstances of the distributed training task or the specific circumstances of the parameter update amount allocated to the second process mentioned above, and are not specifically limited here.

[0065] In some optional implementations of the embodiments of this disclosure, the first optimizer may include, but is not limited to, stochastic gradient descent (SGD), which can rely on gradients and learning rates to update parameters, wherein the learning rate may be a global learning rate or a learning rate used in the previous training step.

[0066] In some optional implementations of the embodiments of this disclosure, in order to improve the adaptability between the learning rate and the first optimizer when updating parameters, an independent learning rate and learning rate scheduler can be configured for the first optimizer, that is, the first optimizer can be configured with an independent learning rate and learning rate scheduler.

[0067] In some other optional implementations of the embodiments of this disclosure, in the above... Figure 5Based on the corresponding embodiment, the target optimizer can be a second optimizer, wherein the second optimizer can be an optimizer that depends on historical optimization information, and / or the second optimizer is an optimizer whose video memory usage is greater than a third video memory threshold. Then, step 503 can be executed as follows: the second process updates the parameters in the model parameter slice corresponding to the first process based on the first optimizer and the gradient slice corresponding to the first process.

[0068] In this embodiment, an optimizer that relies on historical optimization information (such as, but not limited to, momentum and / or variance) and / or has a memory usage greater than a third memory threshold can be used to update the parameters in the model parameter slices corresponding to the first process, thereby improving the convergence of the trained model. The third memory threshold can be the same as or different from the first memory threshold; for example, the third memory threshold can be greater than the first memory threshold.

[0069] Optionally, the above step of updating the parameters in the model parameter slice corresponding to the first process through the second process based on the first optimizer and the gradient slice corresponding to the first process can be implemented as follows: in response to determining that the video memory remaining space corresponding to the second process is greater than the fourth video memory threshold, the parameters in the model parameter slice corresponding to the first process are updated through the second process based on the second optimizer and the gradient slice corresponding to the first process.

[0070] In this embodiment, if it is determined that the second process, which is used to take over the model parameter update task handled by the faulty process (i.e., the first process), has sufficient GPU memory (i.e., greater than the fourth GPU memory threshold), the second optimizer can be selected to update the parameters in the model parameter slice corresponding to the first process. In this way, while fully considering the GPU memory availability of the second process, rapid fault-tolerant recovery of model parameter updates handled by the faulty process in distributed training can be achieved, while simultaneously improving the convergence of the trained model. The fourth GPU memory threshold can be the same as or different from the second GPU memory threshold; for example, the fourth GPU memory threshold can be greater than the second GPU memory threshold.

[0071] It should be noted that the values ​​of the third and fourth memory thresholds mentioned above can be determined based on the specific circumstances of the distributed training task or the specific circumstances of the parameter update amount allocated to the second process mentioned above, and are not specifically limited here.

[0072] In some optional implementations of the embodiments of this disclosure, the second optimizer may include optimizers that consume more video memory, such as Adam and AdamW.

[0073] Optionally, the step of updating the parameters in the model parameter slice corresponding to the first process through the second process based on the first optimizer and the gradient slice corresponding to the first process can also be implemented as follows: In response to determining that the remaining GPU memory corresponding to the second process is greater than the fourth GPU memory threshold, the parameters in the model parameter slice corresponding to the first process are updated through the second process based on the third optimizer and the gradient slice corresponding to the first process. Here, the third optimizer can be an optimizer that relies on a small amount of optimization information (e.g., less than or much less than the amount of historical optimization information relied upon by the second optimizer), and / or the third optimizer is an optimizer whose GPU memory usage is greater than the first GPU memory threshold but less than the third GPU memory threshold, where the third GPU memory threshold is greater than the first GPU memory threshold. Thus, by bypassing the dependence on missing and complex optimizer information, and / or by appropriately controlling the optimizer's GPU memory usage, the convergence of the trained model can be improved while effectively controlling the increase in GPU memory usage. The complexity of the optimization information relied upon by the third optimizer can be lower than the complexity of the historical optimization information relied upon by the second optimizer. For example, the complexity of the optimizer state can be measured based on the number of types of optimizer states.

[0074] In some optional implementations of the embodiments of this disclosure, the historical optimization information relied upon by the second process can be, for example, the optimizer state in the pre-segmented optimizer state slice obtained when performing distributed training using the ZeRO-1 strategy. The optimizer state slice can include multiple types of optimizer states, such as momentum and variance. The limited optimization information relied upon by the third optimizer can be a newly allocated, for example, randomly allocated, single-type optimizer state, such as momentum alone.

[0075] In some optional implementations of the embodiments of this disclosure, the third optimizer may be of the same type as the first optimizer, such as SGD, or it may be another optimizer of a different type than the first optimizer that does not depend on a more complex optimizer state.

[0076] It should be noted that in the model parameter processing method provided in this embodiment, when the second process updates the model parameter slices that it should be responsible for updating, there is no need to consider the memory consumption problem. The pre-allocated optimizer state slices of the second process can be used to update these model parameters. For example, the second process can update the parameters in the model parameter slices corresponding to the pre-allocated optimizer state slices (such as Adam and AdamW mentioned above) and the gradient slices corresponding to the second process.

[0077] In some optional implementations of any embodiment of this disclosure, the process of determining the second process from the non-abnormal processes of multiple processes in step 301 or step 501 can be implemented through different schemes as needed or depending on the specific circumstances: Option 1: The process with the fewest parameters to update among the non-abnormal processes adjacent to the first process can be designated as the second process. By selecting an available non-abnormal process adjacent to the failed process that has the fewest parameters to update as the second process to take over the parameter update task, the re-division of the slice corresponding to each process can be avoided, enabling rapid fault-tolerant recovery in the event of process failure, while minimizing the need for memory copying or resetting.

[0078] In this scheme one, in some optional implementations, the above model parameter processing method may also include the following: synchronizing all model parameter fragments updated by the second process, including the updated model parameter fragments corresponding to the first process, to each non-abnormal process other than the second process based on the first set communication operation.

[0079] In this embodiment, when completing the parameter update task in the current training step or the current round (i.e., the current step), the second process needs to synchronize the parameters in the updated model parameter slice it is responsible for to all other surviving non-abnormal processes (excluding the second process) through a first set communication operation. This ensures that each surviving non-abnormal process stores the full set of model parameters, including the updated part of the model parameters corresponding to the faulty process (i.e., the first process), for use in the next step or the next round of training's forward and backward broadcasting. Accordingly, to ensure that each surviving non-abnormal process stores the full set of updated model parameters, the second process also needs to receive the updated part of the model parameters it is responsible for from each of the other surviving non-abnormal processes through the first set communication operation. Furthermore, the updated parameters among the surviving non-abnormal processes also need to be synchronized, meaning that the updated model parameter slices from each non-abnormal process (excluding the second process) need to be synchronized to other non-abnormal processes (excluding the non-abnormal processes themselves) based on the first set communication operation.

[0080] Option 2: All non-abnormal processes other than the first process can be designated as the second process. Each surviving non-abnormal process is responsible for updating the corresponding portion of the model parameters for the failed process. This avoids the need to re-divide the slices for each process, enabling rapid fault-tolerant recovery in the event of process failure.

[0081] In this second scheme, in some optional implementations, the above model parameter processing method may also include the following: synchronizing other model parameter fragments updated by each second process, excluding the updated model parameter fragments corresponding to the first process, to other non-abnormal processes other than themselves based on the first set communication operation.

[0082] In this embodiment, when completing the parameter update task in the current training step or the current round (i.e., the current step), each second process, i.e. each surviving non-abnormal process, needs to synchronize the updated model parameters (excluding the part of the model parameters corresponding to the faulty process, i.e. the first process) to each other surviving non-abnormal process through the first set communication operation, so as to ensure that each surviving non-abnormal process has stored the full updated model parameters of the distributed training task for use in the forward broadcast and backward broadcast in the next step or the next round of training.

[0083] In some optional implementations of any embodiment of this disclosure, the first set communication operation described above may include, but is not limited to, All-gather.

[0084] For example, such as Figure 6 As shown, the content of the same or corresponding parts can be referred to. Figure 1 The relevant description states that when process 1 (rank1) does not fail or malfunction, the optimizer state can be split and distributed to the four processes in the original communication process group, namely process 0 (rank0), process 1 (rank1), process 2 (rank2) and process 3 (rank3). Each process contains all the model parameters and gradient information, which means that a single-machine 4-card distributed training task can be started normally. The data parallelism (DP) ZeRO-1 strategy is used to split only the optimizer state (such as the first moment m and the second moment v), and each process is responsible for updating its corresponding model parameter slice.

[0085] After the distributed training task started and trained normally for 100 steps, process rank1 suddenly exited training due to a fault at the 101st step. That is, when rank1 encountered a fault or error, such as... Figure 6 As shown, a non-abnormal process adjacent to it, such as process 0 (rank0), can take over the model parameter slices P2 and P3 corresponding to the faulty process rank1. That is, the parameters and gradients that rank1 is responsible for are assigned to process rank0. At the same time, rank1 is responsible for updating the model parameter slices P0 and P1 that were originally assigned. At this time, rank1 exits and the processes rank0', rank1' (i.e., the original rank2) and rank2' (i.e. the original rank3) after re-identifying and re-numbering the process, form a new communication process group.

[0086] When updating parameters, if the parameters in model parameter slices P0 and P1 are updated via rank0', the pre-assigned optimizer state momentum slices m0 and m1, variance slices v1 and v2, and their corresponding optimizers AdamW can continue to be used. The same applies to other processes rank1' and rank2', and will not be elaborated further (see [reference needed]). Figure 7 Step 701 is shown, and the updated model parameter slices P0', P1', P4', P5', P6', and P7' are obtained respectively. When updating the parameters in the model parameter slices P2 and P3 corresponding to the faulty process rank1 through process rank0', the dependency on the missing, pre-allocated, complex optimizer state momentum slices m2 and m3, and variance slices v2 and v3 will be bypassed. It can use the optimizer SGD to complete the update of the parameters in the model parameter slices P2 and P3 without depending on the optimizer state (see step 701). Figure 7 As shown in step 701, the updated model parameter slices P2' and P3' are obtained. To avoid increased memory consumption, the model parameters originally corresponding to rank0' are updated using a normal optimizer such as AdamW, while the model parameters in rank1 taken over by rank0' are updated using SGD. It should be noted that when there is sufficient memory, the parameters in rank1 taken over by rank0' can also be updated using SGD with momentum or directly using AdamW.

[0087] Then, in order to conduct a new round of training, after each process updates its assigned model parameter slice, before the next forward propagation, it can use an all-gather communication operation to broadcast its updated model parameter slice to other processes in the new communication process group. This ensures that each process or GPU in the new communication process group has a copy of the updated, full set of model parameters (e.g., ...). Figure 6 The parameters 0' to 7' (Param0' to Param7') shown can be referenced as follows: Figure 7 As shown in step 702, after updating the parameters, an all-gather operation is performed on the updated parameters in the new communication process group to aggregate them, obtaining the full set of updated parameters in each process. Then, after this round of iteration is completed, the next round of training begins, which can be referred to as follows: Figure 7 As shown in step 703, following the communication order such as the model training steps, after backpropagation, each process calculates the gradient information of its own process parameters, and then performs reduce-scatter operation under a new communication process group according to a new sharding strategy.

[0088] In some optional implementations of any of the above embodiments of this disclosure, the model parameter slices (such as the model parameter slices P2 and P3) and their corresponding gradient slices (such as...) that were originally updated by the first process are... Figure 6 The gradient slices G2 and G3 shown can refer to the slices corresponding to the first process in the full set of model parameters and gradient information stored locally by the second process. Optionally, based on the pre-stored mapping relationship between slices and processes, the model parameter slices corresponding to the first process can be selected from the full set of model parameters, and the gradient slices corresponding to the first process can be selected from the full set of gradient information, and then allocated to the second process so that the second process can update the model parameter slices corresponding to the first process based on the gradient slices.

[0089] In some optional implementations of any of the above embodiments disclosed herein, the model parameter processing method may further include the following: reallocating consecutive process identifiers for non-abnormal processes among multiple processes; processing the optimizer state corresponding to the distributed training task according to a preset strategy to obtain optimizer state slices corresponding to each non-abnormal process after the process identifiers have been reallocated.

[0090] In this embodiment, when multiple processes used to execute a distributed training task fail during training, in order to ensure the smooth execution of the distributed training task, it is necessary to reallocate consecutive process identifiers to all remaining non-abnormal processes (i.e., new communication process groups). Optionally, the process identifiers of non-abnormal processes whose process identifiers are located after the process identifier of the failed process (i.e., the first process) can be incremented by 1 in sequence, i.e., shifted forward by one position to complete the process identifier update. Then, according to a preset strategy, optimizer state segmentation is performed based on the new communication process group to obtain optimizer state slices corresponding to each non-abnormal process after the process identifiers are reallocated, which can be used for parameter updates in the next step or the next round of training, thereby improving the overall reliability and stability of large-scale distributed training.

[0091] In some optional implementations of the embodiments of this disclosure, the preset strategy may include, but is not limited to, the ZeRO-1 strategy.

[0092] In some optional implementations of any of the above embodiments of this disclosure, the above model parameter processing method may further include the following: in response to the end of a new backpropagation of the distributed training task, the current full gradient information corresponding to the distributed training task is obtained; the current full gradient information is processed based on the second set communication operation to obtain gradient slices corresponding to each non-abnormal process after the process identifier is reassigned.

[0093] In this embodiment, after completing a new backpropagation of the distributed training task and obtaining the new, i.e., the current full gradient information corresponding to the distributed training task, in order to ensure the smooth execution of the distributed training task, a second set of communication operations is required to re-slice the gradient based on the new communication process group for a new parameter update, thereby improving the overall reliability and stability of large-scale distributed training.

[0094] In some optional implementations of the embodiments of this disclosure, the second set communication operation may include, but is not limited to, reduce-scatter.

[0095] To enhance understanding, this disclosure also provides a specific implementation scheme based on a particular application scenario. Please refer to the example below. Figure 8 The distributed training framework shown is 800.

[0096] like Figure 8 As shown, when executing the training script 801 for distributed training, the communication collection library 802 can provide heartbeat packets for process status monitoring and communication status with the collection communication library. The fault detection module 803 in the fault-tolerant control plane monitors the running status of each process in real time based on the timeout monitoring mechanism of the heartbeat packets and the communication status of the collection communication library. This allows for timely detection of processes that have failed or are hanging. When a process is detected to be malfunctioning, the rank number (i.e., process identifier) ​​of the malfunctioning process or the faulty process j (corresponding to the first process mentioned above) is reported, and a new communication process group is rebuilt to replace the original communication process group. Optionally, the fault message corresponding to the malfunctioning process can be broadcast to all surviving processes through the master process (rank 0) or other coordinating nodes, triggering a fault-tolerant update mode. When training script 801 is executed, forward propagation and backward propagation can be implemented to generate parameters and corresponding gradients, which are stored in parameter gradient storage module 806. In the case of using the ZeRO-1 strategy, the optimizer state is sharded, but each process can utilize its locally stored copy of the complete model parameters after the most recent All-Gather and the gradient shards obtained through Reduce-Scatter.

[0097] In this implementation, when a faulty process j is detected, the fault detection module 803 can trigger a parameter update task to take over through the shard management module 804. Optionally, the fault-tolerant update engine 805 can trigger each surviving process (or a specified process / group of processes) to identify the parameter shard that originally belonged to the faulty process j based on the known shard mapping relationship. To avoid re-dividing each process shard, the parameters and gradients of the faulty process j can be taken over by the j+1 process or j-1 process adjacent to the faulty process j. If the adjacent process of the faulty process j is also a faulty process, the search continues to find a nearby process to take over. Whether to find the left or right process to take over depends on the number of parameters handled by the left and right processes; optionally, the process with fewer parameters can be chosen. The other parameters handled by the process h that takes over the parameter update task of the faulty process j can be updated according to the original logic, while the update of parameters in the parameter shard belonging to the faulty process j can be done in one of the following ways: (1) When there is enough video memory, process h can directly inject the parameters it takes over into the param_groups of the current optimizer (commonly AdamW) so that process h can update the parameters it was originally responsible for updating and the parameters of the faulty process j it took over. The parameters can be updated normally according to the original parameters, that is, the current optimizer uses the corresponding momentum and variance terms to update the parameters.

[0098] (2) When the video memory is already at its limit, updating parameters using AdamW will increase video memory usage due to the influence of momentum and variance terms. In this case, a hybrid device optimizer strategy (HybridDeviceOptimizer) can be used, where the parameters of the faulty process j being taken over can be updated according to the SGD optimizer, such as... Figure 8 As shown, an SGD-based update is performed, where the update formula is: Updated parameters = Parameters - Learning rate × Gradient. This SGD optimizer does not rely on complex optimizer states such as momentum; it only needs the current parameters, gradient, and learning rate to complete the parameter update. The model parameters and gradient information used by the fault-tolerant update engine 805 when triggering process h to update the parameters of the faulty process j can be provided by the parameter gradient storage module 806. After the SGD-based update is completed, the updated parameter values ​​of the faulty process j are synchronized to all surviving processes along with the updated parameters of process h. This can be achieved by calling an All-Gather operation from the communication collection library 802 to ensure that all surviving processes hold the updated, consistent, and complete model parameters. Subsequently, the training task can continue from the current step, as if the parameters of the faulty process j have been updated normally. The faulty process j can be replaced and rejoined, or processed after the task ends; no specific limitations are imposed here.

[0099] According to the implementation scheme provided in this disclosure, compared to the need to periodically save model checkpoints in large-scale distributed training, when a fault (single point fault or multi-point fault) occurs, the method of rolling back to the previous checkpoint before the fault occurred and restarting training can avoid losing all training progress from the previous checkpoint to the fault point (e.g., which can last for hours or even days). Recovery only requires pausing for one iteration and performing one SGD update, minimizing progress loss and thus avoiding serious waste of computing resources and reducing fault recovery costs. This achieves a low-overhead, high-efficiency fault tolerance mechanism, optionally manifested in: (1) Reduced time cost: Since restarting training requires reloading the model, optimizer state, dataset, etc., and repeating the iterations that have been calculated since the restart checkpoint, the technical solution provided by this disclosure does not require rolling back the training progress, thus avoiding the corresponding training time delay while avoiding a huge waste of computing resources.

[0100] (2) Reduced storage costs: In order to minimize the rollback distance during restart, checkpoints need to be saved frequently. However, checkpoint files containing complete model parameters and optimizer states are huge (e.g., hundreds of GB or even TB). The technical solution provided by this disclosure does not require frequent saving, thus avoiding the occupation of massive storage space and preventing the normal training process from being interfered with by frequent I / O operations.

[0101] (3) Improve system resource utilization: Due to the frequent saving of checkpoints and the high probability of task interruption and restart, a lot of computing resources (such as GPUs) are used for repetitive calculations and waiting. However, through the technical solution provided in the embodiments of this disclosure, computing resources are used for effective model training, thus improving the average resource utilization and training throughput of the entire cluster.

[0102] Therefore, the lightweight fault-tolerant mechanism provided by this disclosure, which maintains the continuity of training tasks and enables rapid recovery of faulty processes without significantly increasing memory overhead, can utilize the model parameters (which can be low-precision) and gradient information already distributed among processes through the All-Gather operation of the ZeRO-1 strategy. When a process-level fault occurs, i.e., a fault is detected in a certain process, one or more other non-abnormal processes in the cluster can quickly take over the optimizer parameter update task of the faulty process. Optionally, a simple SGD algorithm can be used to update the parameters in the faulty slice, thereby bypassing the dependence on missing and complex optimizer states (such as the optimizer state corresponding to AdamW) to achieve rapid recovery, minimizing the loss of training progress, avoiding complete interruption of training tasks, and effectively preventing the complete failure of distributed training tasks. This improves the reliability, overall stability, and efficiency of large-scale distributed training tasks, thereby ensuring the stable operation of model training. In particular, it can improve the reliability and stability of training in a Wanka cluster environment, providing efficient and practical reliability assurance for large-scale deep learning training.

[0103] Further reference Figure 9 As an implementation of the methods shown in the above figures, this disclosure provides an embodiment of a model parameter processing device, which is similar to... Figure 3 Corresponding to the method embodiments shown, the device can be specifically applied to various electronic devices (e.g., servers).

[0104] like Figure 9 As shown, the model parameter processing apparatus 900 of this embodiment may include: a determining unit 901, an allocation unit 902, and a processing unit 903.

[0105] The determining unit 901 is configured to, in response to detecting an anomaly in a first process among multiple processes used to perform a distributed training task, determine a second process from the non-abnormal processes among the multiple processes, wherein the multiple processes update parameters in different model parameter slices of the distributed training task respectively; the allocating unit 902 is configured to allocate the model parameter slices updated by the first process and the gradient slices corresponding to the first process to the second process; and the processing unit 903 is configured to update the parameters in the model parameter slices corresponding to the first process through the second process based on the gradient slices corresponding to the first process.

[0106] In this embodiment of the disclosure, the specific processing of the determining unit 901, the allocation unit 902, and the processing unit 903 in the model parameter processing apparatus 900, and the resulting technical effects, can be referred to respectively. Figure 3 The relevant descriptions of steps 301-303 in the corresponding embodiments will not be repeated here.

[0107] In some optional implementations of the embodiments of this disclosure, the processing unit 903 is further configured to: update the parameters in the model parameter slices corresponding to the first process through the second process based on the first optimizer and the gradient slices corresponding to the first process, wherein the first optimizer is an optimizer that does not rely on historical optimization information and / or the first optimizer is an optimizer whose video memory usage is less than or equal to a first video memory threshold.

[0108] In some optional implementations of the embodiments of this disclosure, the processing unit 903 is further configured to: in response to determining that the video memory remaining amount corresponding to the second process is less than or equal to the second video memory threshold, update the parameters in the model parameter slice corresponding to the first process based on the first optimizer and the gradient slice corresponding to the first process through the second process.

[0109] In some optional implementations of the embodiments of this disclosure, the first optimizer is configured with an independent learning rate and a learning rate scheduler.

[0110] In some optional implementations of the embodiments of this disclosure, the processing unit 903 is further configured to: update the parameters in the model parameter slice corresponding to the first process through the second process based on the gradient slices corresponding to the second optimizer and the first process, wherein the second optimizer is an optimizer that depends on historical optimization information, and / or the second optimizer is an optimizer whose video memory usage is greater than a third video memory threshold.

[0111] In some optional implementations of the embodiments of this disclosure, the processing unit 903 is further configured to: in response to determining that the remaining video memory corresponding to the second process is greater than the fourth video memory threshold, update the parameters in the model parameter slice corresponding to the first process based on the second optimizer and the gradient slice corresponding to the first process through the second process.

[0112] In some optional implementations of the embodiments of this disclosure, the determining unit 901 is further configured to: determine the process with the fewest parameters to be updated among the non-abnormal processes adjacent to the first process as the second process; and the processing unit 903 is further configured to: synchronize all model parameter fragments updated by the second process, including the updated model parameter fragments corresponding to the first process, to each non-abnormal process other than the second process based on the first set communication operation.

[0113] In some optional implementations of the embodiments of this disclosure, the determining unit 901 is further configured to: determine all remaining non-abnormal processes as second processes; and the processing unit 903 is further configured to: synchronize other model parameter fragments updated by each second process, excluding the updated model parameter fragments corresponding to the first process, to other non-abnormal processes other than itself based on the first set communication operation.

[0114] In some optional implementations of the embodiments of this disclosure, the processing unit 903 is further configured to: reallocate consecutive process identifiers for non-abnormal processes among multiple processes; process the optimizer state corresponding to the distributed training task according to a preset strategy to obtain optimizer state slices corresponding to each non-abnormal process after the process identifiers are reallocated.

[0115] In some optional implementations of the embodiments of this disclosure, the processing unit 903 is further configured to: obtain the current full gradient information corresponding to the distributed training task in response to the end of a new backpropagation of the distributed training task, process the current full gradient information based on the second set communication operation, and obtain gradient slices corresponding to each non-abnormal process after the process identifier is reassigned.

[0116] This embodiment exists as a device embodiment corresponding to the above method embodiment. When a faulty process (the first process) occurs during the parallel execution of a distributed training task by multiple processes, a second process can be determined from the non-faulty processes (non-abnormal processes). The model parameter slices and their corresponding gradient slices that were originally the responsibility of the first process to update are then assigned to the second process. The second process then uses the gradient slices to update the parameters in the model parameter slices that were originally the responsibility of the first process, thus seamlessly taking over the model parameter update task from the faulty process. In this way, by providing a fault-tolerant update mechanism for model parameters in the case of a faulty process, the execution of the model training task can continue even when a faulty process occurs during model training, avoiding training interruption. This improves the reliability and efficiency of large-scale distributed training, avoids unnecessary waste of computational resources, and increases computational resource utilization and training throughput.

[0117] Figure 10A schematic block diagram of an example electronic device 1000 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0118] like Figure 10 As shown, the electronic device 1000 includes a computing unit 1001, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1002 or a computer program loaded from a storage unit 1008 into a random access memory (RAM) 1003. The RAM 1003 may also store various programs and data required for the operation of the electronic device 1000. The computing unit 1001, ROM 1002, and RAM 1003 are interconnected via a bus 1004. An input / output (I / O) interface 1005 is also connected to the bus 1004.

[0119] Multiple components in electronic device 1000 are connected to I / O interface 1005, including: input unit 1006, such as keyboard, mouse, etc.; output unit 1007, such as various types of displays, speakers, etc.; storage unit 1008, such as disk, optical disk, etc.; and communication unit 1009, such as network card, modem, wireless transceiver, etc. Communication unit 1009 allows electronic device 1000 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0120] The computing unit 1001 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 1001 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 1001 performs the various methods and processes described above, such as model parameter processing methods. For example, in some embodiments, the model parameter processing method may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 1008. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 1000 via ROM 1002 and / or communication unit 1009. When the computer program is loaded into RAM 1003 and executed by the computing unit 1001, one or more steps of the model parameter processing method described above may be performed. Alternatively, in other embodiments, the computing unit 1001 may be configured to perform a model parameter processing method by any other suitable means (e.g., by means of firmware).

[0121] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0122] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0123] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0124] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0125] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0126] Computer systems can include clients and servers. Clients and servers are generally geographically separated and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, distributed system servers, or servers incorporating blockchain technology. Cloud servers, also known as cloud computing servers or cloud hosts, are a hosting product within the cloud computing service ecosystem, designed to address the shortcomings of traditional physical hosts and Virtual Private Servers (VPS) services, such as high management difficulty and weak business scalability.

[0127] According to the technical solution of this disclosure, when a faulty process (i.e., the first process) occurs during the parallel execution of a distributed training task by multiple processes, a second process can be identified from the non-faulty processes (i.e., non-abnormal processes). The model parameter slices and their corresponding gradient slices that were originally the responsibility of the first process to update are then assigned to the second process. The second process then uses the gradient slices to update the parameters in the model parameter slices that were originally the responsibility of the first process, thus achieving seamless takeover of the model parameter update task from the faulty process. In this way, by providing a fault-tolerant update mechanism for model parameters in the case of a faulty process, the execution of the model training task can continue even when a faulty process occurs during model training, avoiding interruption of the training task. This improves the reliability and efficiency of large-scale distributed training, avoids unnecessary waste of computing resources, and increases the utilization of computing resources and training throughput.

[0128] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0129] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A method for processing model parameters, comprising: In response to detecting an anomaly in a first process among a plurality of processes used to perform a distributed training task, a second process is determined from the non-abnormal processes of the plurality of processes, wherein the plurality of processes update parameters in different model parameter slices of the distributed training task respectively; The model parameter slices updated by the first process and the gradient slices corresponding to the first process are assigned to the second process; The second process updates the parameters in the model parameter slice corresponding to the first process based on the gradient slice corresponding to the first process.

2. The method according to claim 1, wherein, The step of updating the parameters in the model parameter slice corresponding to the first process through the second process based on the gradient slice corresponding to the first process includes: The second process updates the parameters in the model parameter slices corresponding to the first process based on the first optimizer and the gradient slices corresponding to the first process. The first optimizer is an optimizer that does not rely on historical optimization information and / or the first optimizer is an optimizer whose memory usage is less than or equal to a first memory threshold.

3. The method according to claim 2, wherein, The step of updating the parameters in the model parameter slices corresponding to the first process through the second process based on the first optimizer and the gradient slices corresponding to the first process includes: In response to determining that the remaining video memory corresponding to the second process is less than or equal to the second video memory threshold, the parameters in the model parameter slice corresponding to the first process are updated by the second process based on the first optimizer and the gradient slice corresponding to the first process.

4. The method according to claim 2, wherein, The first optimizer is configured with an independent learning rate and learning rate scheduler.

5. The method according to claim 1, wherein, The step of updating the parameters in the model parameter slice corresponding to the first process through the second process based on the gradient slice corresponding to the first process includes: The second process updates the parameters in the model parameter slices corresponding to the first process based on the second optimizer and the gradient slices corresponding to the first process. The second optimizer is an optimizer that depends on historical optimization information and / or the second optimizer is an optimizer whose memory usage is greater than a third memory threshold.

6. The method according to claim 5, wherein, The step of updating the parameters in the model parameter slices corresponding to the first process through the second process based on the second optimizer and the gradient slices corresponding to the first process includes: In response to determining that the remaining video memory corresponding to the second process is greater than the fourth video memory threshold, the parameters in the model parameter slice corresponding to the first process are updated by the second process based on the second optimizer and the gradient slice corresponding to the first process.

7. The method according to claim 1, wherein, The step of determining the second process from the non-abnormal processes of the plurality of processes includes: The process with the fewest parameters to update among the non-abnormal processes adjacent to the first process is identified as the second process; and The method further includes: All model parameter fragments updated through the second process, including the updated model parameter fragments corresponding to the first process, are synchronized to all non-abnormal processes other than the second process based on the first set communication operation.

8. The method according to claim 1, wherein, The step of determining the second process from the non-abnormal processes of the plurality of processes includes: All remaining non-abnormal processes are identified as the second process; and The method further includes: Other model parameter fragments, which are updated through each of the second processes but do not include the updated model parameter fragments corresponding to the first process, are synchronized to other non-abnormal processes other than themselves based on the first set communication operation.

9. The method according to claim 1, further comprising: Reassign consecutive process identifiers to the non-abnormal processes among the plurality of processes; The optimizer state corresponding to the distributed training task is processed according to a preset strategy to obtain optimizer state fragments corresponding to each non-abnormal process after the process identifier is reassigned.

10. The method of claim 7, further comprising: In response to the end of a new backpropagation of the distributed training task, the current full gradient information corresponding to the distributed training task is obtained. The current full gradient information is processed based on the second set communication operation to obtain gradient slices corresponding to each non-abnormal process after the process identifier is reassigned.

11. A model parameter processing device, characterized in that, include: The determining unit is configured to, in response to detecting an anomaly in a first process among a plurality of processes used to perform a distributed training task, determine a second process from the non-abnormal processes of the plurality of processes, wherein the plurality of processes update parameters in different model parameter slices of the distributed training task respectively; The allocation unit is configured to allocate the model parameter slices updated by the first process and the gradient slices corresponding to the first process to the second process. The processing unit is configured to update the parameters in the model parameter slice corresponding to the first process through the second process based on the gradient slice corresponding to the first process.

12. An electronic device, characterized in that, include: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the model parameter processing method according to any one of claims 1-10.

13. A non-transitory computer-readable storage medium storing computer instructions, characterized in that, The computer instructions are used to cause the processor to execute the model parameter processing method according to any one of claims 1-10.

14. A computer program product, characterized in that, It includes a computer program, which, when executed by a processor, implements the model parameter processing method according to any one of claims 1-10.

Images