Method for determining fragmentation strategy of model training and related device

Abstract

The embodiment of the invention relates to a method for determining a fragmentation strategy for model training and a related device. A model can be trained through various fragmentation strategies. The method comprises the steps that firstly, the operation time of a plurality of processors in the model training process is determined, the processors can be divided according to a fragmentation strategy, and the fragmentation strategy indicates the division mode of data of a model in the processors; then, based on a fragmentation strategy, determining communication time of the plurality of processors in the process of training the model; furthermore, the larger one of the operation time and the communication time is selected, and when the larger one meets a time condition, the fragmentation strategy is determined as a target fragmentation strategy in the plurality of fragmentation strategies.

Description

Technical Field

[0001] Embodiments of the present specification generally relate to the field of computers, and more specifically to a method and related apparatus for determining a sharding strategy for model training. Background Art

[0002] Model training is a core part of machine learning and artificial intelligence. It optimizes the parameters of the model through algorithms and data so that it can perform specific tasks accurately and efficiently. Sharding technology plays an important role in model training. It can improve training efficiency, optimize memory usage, and enhance model scalability. It is one of the important means to accelerate model training.

[0003] Sharding technology can divide large data sets or models into smaller parts and distribute them to different computing devices for parallel processing, thereby significantly reducing the memory usage of a single device and improving training efficiency. Sharding technology is used in database management, big data processing and blockchain, and has great potential in improving system performance and scalability. Summary of the invention

[0004] The embodiments of this specification provide a method and related apparatus for determining a sharding strategy for model training.

[0005] In a first aspect of the present specification, a method for determining a sharding strategy for model training is provided, and the model can be trained by multiple sharding strategies. The method includes determining the operation time of multiple processors in the process of training the model, and the multiple processors are divided based on the sharding strategy, and the sharding strategy indicates how the data of the model is divided among the multiple processors. The method includes determining the communication time of the multiple processors in the process of training the model based on the sharding strategy. In addition, the method also includes determining that the sharding strategy is a target sharding strategy among multiple sharding strategies in response to the larger of the operation time and the communication time satisfying the time condition.

[0006] In a second aspect of the present specification, a device for determining a sharding strategy for model training is provided, and the model can be trained through multiple sharding strategies. The device includes an operation time determination module, which is configured to determine the operation time of multiple processors in the process of training the model, and the multiple processors are divided based on the sharding strategy, and the sharding strategy indicates how the data of the model is divided among the multiple processors. The device includes a communication time determination module, which is configured to determine the communication time of multiple processors in the process of training the model based on the sharding strategy. In addition, the device also includes a sharding strategy determination module, which is configured to determine the sharding strategy as the target sharding strategy among multiple sharding strategies in response to the larger of the operation time and the communication time satisfying the time condition.

[0007] In a third aspect of the present specification, a computing device is provided. The computing device includes a memory. The computing device also includes a memory coupled to a processor, the memory having instructions stored therein, and when the instructions are executed by the processor, the computing device executes the method according to the first aspect of the present specification.

[0008] In a fourth aspect of the present specification, a computer program product is provided, wherein the computer program product comprises a computer program, and the computer program is executed by a processor to implement the method according to the first aspect of the present specification.

[0009] In a fifth aspect of the present disclosure, a computer storage medium is provided. The computer-readable storage medium stores computer-executable instructions, wherein the computer-executable instructions are executed by a processor to implement the method provided according to the first aspect of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The above and other features, advantages and aspects of the embodiments of the present specification will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. In the accompanying drawings, the same or similar reference numerals represent the same or similar elements, wherein:

[0011] Figure 1 A schematic diagram illustrating an example environment in which some embodiments of the present specification may be implemented;

[0012] Figure 2 A flowchart of a method for determining a sharding strategy for model training in some embodiments of this specification is shown;

[0013] Figure 3 A flowchart showing a method for determining communication time according to some embodiments of the present specification;

[0014] Figure 4 A flowchart showing a method for determining a first communication time according to some embodiments of the present specification;

[0015] Figure 5 A flowchart showing a method for determining a second communication time according to some embodiments of the present specification; and

[0016] Figure 6 A schematic block diagram of a computing device is shown that illustrates some embodiments of the present description.

[0017] Throughout the drawings, the same or similar reference numbers denote the same or similar elements. DETAILED DESCRIPTION

[0018] The embodiments of the present specification will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present specification are shown in the accompanying drawings, it should be understood that the present specification can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present specification. It should be understood that the drawings and embodiments of the present specification are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0019] In the description of the embodiments of this specification, the term "including" and similar terms should be understood as open inclusion, that is, "including but not limited to". The term "based on" should be understood as "based at least in part on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The terms "first", "second", etc. may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0020] As mentioned above, there are a variety of distributed training strategies in large model training scenarios. Different sharding strategies have different video memory consumption and communication costs. Model size, the proportion of model parameters involved in training, communication costs between processors, and the number of machine resources all affect the performance of the sharding strategy. Some sharding strategy implementations are provided in related technologies. For example, ZeRO provides three distributed training strategies for Parameter, Gradient, and Optimizer State sharding. MiCS introduces the concept of grouping based on ZeRO-3, and shards within the group to increase video memory in exchange for reduced communication time. These solutions only provide some sharding strategy implementations, but for specific training tasks, there is no recommendation on which strategy to use, and it can only rely on experience or multiple attempts. Therefore, in a specific training scenario, how to effectively select a sharding strategy becomes an urgent problem to be solved.

[0021] To this end, a method for determining a sharding strategy for model training is provided in an embodiment of the present specification, and the model can be trained through multiple sharding strategies. First, the operation time of multiple processors in the process of training the model is determined, wherein the multiple processors can be divided according to the sharding strategy, and the sharding strategy can represent the way in which the data of the model is divided among multiple processors. Then, based on the sharding strategy, the communication time of multiple processors in the process of training the model is determined. Further, the larger of the operation time and the communication time is selected, and when the larger one meets the time condition, the sharding strategy is determined as the target sharding strategy among multiple sharding strategies.

[0022] In this way, it is possible to select targeted sharding strategies for large model training tasks based on different training tasks and the processor's computing time and communication time. This avoids the situation where, under specific training tasks, the sharding strategy cannot be selected flexibly and efficiently through experience or multiple attempts. Therefore, the present invention can improve the model training efficiency by selecting the sharding strategy through calculation under specific training tasks.

[0023] Figure 1 A schematic diagram of an example environment 100 is shown in which some embodiments of the present specification may be implemented. Figure 1 , the example environment 100 includes a computing device 102, and the computing device 102 includes but is not limited to a communication device, a computing chip, a computing system, a single server, a distributed server, or a cloud-based server. In some embodiments, the parameter set 104, the gradient set 106, and the optimizer state set 108 are data involved in the model training process, and the computing device 102 can shard the parameter set 104, the gradient set 106, and the optimizer parameter set 108 used for model training.

[0024] In some embodiments, the computing device 102 may adopt parameter set non-sharding, parameter set group intra-sharding, and parameter set global sharding for the parameter set 104. In some embodiments, the computing device 102 may adopt gradient set non-sharding, parameter set group intra-sharding, and parameter set global sharding for the gradient set 106. In some embodiments, the computing device 102 may adopt optimizer parameter set non-sharding, optimizer parameter set group intra-sharding, and optimizer parameter set global sharding for the optimizer parameter set 108.

[0025] In some embodiments, the computing device 102 may select and combine the parameter set sharding strategy, the gradient set sharding strategy, and the optimizer parameter set sharding strategy to form one of the sharding strategies in the sharding strategy 110, wherein the sharding strategy 110 includes multiple sharding strategies. For example, the computing device 102 selects the sharding strategy of non-sharding of the parameter set, sharding within the gradient set group, and global sharding of the optimizer parameter set for model training. In some embodiments, the computing device 102 selects the sharding strategy of non-sharding of the parameter set, global sharding of the gradient set, and sharding within the optimizer parameter set group for model training. The computing device 102 may also select sharding within the parameter set group, sharding within the gradient set group, and global sharding of the optimizer parameter set for model training. It can be understood that there are three sharding strategies for each of the parameter set 104, gradient set 106 and optimizer parameter set 108 involved in the model training process. There are 27 combinations of the parameter set sharding strategy, the gradient set sharding strategy and the optimizer parameter set sharding strategy, that is, there are 27 sharding strategies in the sharding strategy 110 to train the model.

[0026] In some embodiments, the computing device 102 selects a target sharding strategy from the sharding strategy 110 to train the model. It is understandable that different sharding strategies have their own advantages and applicability under different model scales and complexities, hardware resource limitations, communication overhead and bandwidth, training goals and requirements, and algorithm and framework support. In this way, the computing device 102 considers multiple factors when selecting a target sharding strategy and makes a selection based on the specific circumstances. By selecting a reasonable sharding strategy, the training efficiency of the model can be significantly improved, resource consumption can be reduced, and the iteration and optimization process of the model can be accelerated.

[0027] It should be understood that the architecture and functions in the example environment 100 are described for exemplary purposes only and do not imply any limitation on the scope of the present description. The embodiments of the present description may also be applied to other environments with different structures and / or functions.

[0028] The following will combine Figures 2 to 6 The process according to the embodiment of this specification is described in detail. For ease of understanding, the specific data mentioned in the following description are exemplary and are not intended to limit the scope of protection of the present disclosure. It should be understood that the embodiments described below may also include additional actions not shown and / or may omit the actions shown, and the scope of the present disclosure is not limited in this respect.

[0029] Figure 2 A flowchart of a method 200 for determining a sharding strategy for model training is shown in the specification. Figure 1 In the example environment 100 shown in , the method 200 can be performed by the computing device 102. It should be understood that although the following content is described with the computing device 102 as the execution subject, the method 200 can also be performed by other devices. The method 200 can also include additional actions not shown and / or can omit the actions shown, and the scope of the present disclosure is not limited in this respect.

[0030] At step 202, the operation time of multiple processors in the process of training the model is determined, and the multiple processors are divided based on a sharding strategy, and the sharding strategy indicates the way in which the data of the model is divided among the multiple processors. It is understood that in the process of training the model, the computing device 102 can use the sharding strategy to distribute the data in the model training process to the multiple processors. Each processor operates on the allocated model data according to the sharding strategy, thereby performing the operation in parallel.

[0031] In some embodiments, the computing device 102 may adopt a non-sharding strategy for the model data in the model training process. The non-sharding strategy is a sharding strategy in which no data or model sharding is performed in large model training, and all model data is stored on a single computing device or node. In some embodiments, the computing device 102 may adopt a global sharding strategy for the model data in the model training process. The global sharding strategy is a sharding strategy in which data and model parameters are distributed to multiple computing devices or nodes according to certain rules in large model training to achieve parallel training and accelerate the training process. It is understandable that the model data in the training process can be divided into multiple batches and processed in parallel on different processors. In some embodiments, the processor of this embodiment may be a graphics processing unit (GPU).

[0032] In some embodiments, an intra-group sharding strategy is adopted. The intra-group sharding strategy is a sharding strategy that divides model data with specific associations into the same group in large model training and performs data sharding within the group. It is understandable that the computing device 102 can perform sharding based on the association between the data or based on the characteristics of the data. In order to evaluate and optimize the performance of the entire training, the computing device 102 needs to determine the computing time consumed by each processor when performing its assigned tasks. In some embodiments, a group can be a computing node, such as a computer or a server, which can contain multiple processors.

[0033] At step 204, based on the sharding strategy, the communication time of multiple processors in the process of training the model is determined. In some embodiments, the communication time is the time required for the processors to transmit data or information, including sharing data, synchronizing states, and transferring calculation results. It is understandable that the granularity of the sharding, the number and configuration of processors, the speed and bandwidth of the network connection, and the efficiency of the communication protocol will affect the communication time. In some embodiments, finer-grained sharding may lead to more frequent communications, and coarse-grained sharding may reduce the number of communications but the amount of data in each communication will be larger.

[0034] At step 206, if the larger of the computing time and the communication time meets the time condition, the sharding strategy is determined to be the target sharding strategy among multiple sharding strategies. In some embodiments, the total time for training the model consists of the computing time and the communication time. It is understandable that in some efficient training frameworks and hardware architectures, computing and communication can be parallel to a certain extent, thereby reducing the training time. Based on this, the larger of the computing time and the communication time can be considered as the total time for training the model. In some embodiments, the time condition can be that the larger is lower than a time threshold, or that the larger reaches a minimum value. For example, under a certain sharding strategy, when the communication time is greater than the computing time, if the communication time is less than the time threshold, the sharding strategy can be used as the target sharding strategy among multiple sharding strategies.

[0035] In some embodiments, the processor can communicate with other processors while operating on the model data. In this case, the total time for training the model is the sum of the operation time and the communication time minus the overlap time, and the overlap time is between 0 and the smaller of the operation time and the communication time. By optimizing the training process, the smaller one can be used as the overlap time, so that the larger of the operation time and the communication time is used as the total time for training a model. For example, when the operation time is greater than the communication time, the overlap time can be equal to the communication time. In some embodiments, when the communication time is greater than the operation time, the overlap time can be equal to the operation time.

[0036] In the embodiments of the present disclosure, it is possible to select targeted sharding strategies for large model training tasks based on different training tasks and according to the processor's computing time and communication time. This avoids the situation where, under specific training tasks, the sharding strategy cannot be selected flexibly and efficiently through experience or multiple attempts. Therefore, the present invention can improve the model training efficiency by selecting the sharding strategy through calculation under specific training tasks.

[0037] Figure 3 A flow chart of a method 300 for determining communication time in this specification is shown. In some embodiments, the computing device 102 selects a data subset from a data set used for model training for input into a model for model training. Before each iteration, the computing device 102 selects a data subset from a data set used for training for the current iterative training. It is understandable that one iteration for a model is a basic unit in the model training process, and each iteration includes a complete forward propagation, a complete backward propagation, and a process of updating model parameters using calculated gradients. One iteration may involve communication between multiple processors. In some embodiments, it may be necessary to synchronize gradients or update parameters between processors, and this communication will increase the iteration time, and because multiple processors process data simultaneously, it will also accelerate the overall training process.

[0038] It is understandable that the data is passed through each layer of the model, and the time required to calculate the prediction result is the time associated with the forward propagation. The gradient is calculated based on the error between the prediction result obtained by the forward propagation and the actual label, and the time required to pass the gradient back to each layer of the model to update the weight is the time associated with the backward propagation. In some embodiments, forward propagation and backward propagation may involve communication between processors, such as when the model is divided into different processors, and intermediate results or gradients need to be transferred between processors to calculate the final prediction error and gradient. The computing device 102 determines the communication time (first communication time) associated with the forward propagation and the backward propagation by obtaining the time associated with the forward propagation and the backward propagation of the model in an iterative process.

[0039] In some embodiments, after the gradient calculation is completed, the computing device 102 uses multiple processors to operate on the model parameters in the model, such as model weights. It is understandable that the computing device 102 needs to synchronize the gradients in each processor. In some embodiments, the computing device 102 sends the gradients to a central processor or uses some form of distributed averaging algorithm. After the gradients are synchronized, the computing device 102 can use the gradients to update the model weights in the model through multiple processors. In some embodiments, the updated model weights may need to be synchronized between processors again to ensure that all processors use the same model weights for subsequent iterations. The updated parameters will be used for the forward propagation of the next iteration. This communication time related to the model update is the second communication time (t update In some embodiments, the computing device 102 determines the communication time of the processor in the process of training the model by the communication time associated with the forward propagation and the backward propagation (the first communication time) and the communication time associated with the model update (the second communication time).

[0040] Figure 4A flow chart of a method 400 for determining a first communication time in this specification is shown. In some embodiments, the computing device 102 divides a data set for training a model into a plurality of data subsets, each of which includes a number of data for each iteration in the training process. In an iterative process, after the data subset is input into the model, the computing device 102 performs forward propagation and backward propagation on the model to achieve training of the model, and the model parameters need to be transmitted between processors during the training process. In some embodiments, the model parameters may also need to be accessed or confirmed again during the backward propagation process (for example, to perform certain specific gradient calculations or regularization operations), which may also generate additional communication time. In some embodiments, the computing device 102 obtains the communication time associated with the first operation of the model parameters, such as the time consumed by performing an all-gather operation on the model parameters during the forward propagation and backward propagation processes.

[0041] In some embodiments, the computing device 102 obtains the communication time associated with the second operation of the model gradient in the backward propagation, such as the time consumed by the reduce-scatter operation used to accumulate the model gradient in the backward propagation process. In some embodiments, in order to improve communication efficiency, a gradient accumulation strategy is usually adopted, and the number of gradient accumulation steps directly affects the number of communications. The computing device 102 obtains the communication time associated with the model parameter operation (the communication time associated with the first operation t param ), the time associated with the model gradient operation during the back propagation process (the communication time associated with the second operation t gradient ) and the product of the number of gradient accumulation steps to determine the communication time (first communication time) associated with the forward propagation and the backward propagation. The second operation may be, for example, a reduce-scatter operation.

[0042] In some embodiments, during the model training process, for processing model parameters, a sharding strategy may be adopted among parameter non-sharding (first sharding strategy), parameter group sharding (second sharding strategy), and parameter global sharding (third sharding strategy). The computing device 102 determines the parameter sharding strategy. Under the parameter non-sharding strategy, the computing device 102 determines that the time associated with the model parameter operation (the communication time associated with the first operation) is 0, and t param =0. Under the parameter group intra-sharding strategy, the computing device 102 determines the time associated with the model parameter operation (the communication time associated with the first operation) according to the data volume of the model parameter, the number of processors in the group, and the communication bandwidth between the processors in the group. The specific formula is: param=2*ψ / m*(m-1) / B′, where ψ is the data volume of the model parameter, m is the number of processors in a group, and B′ is the communication bandwidth between processors in a group. Under the parameter global sharding strategy, the computing device 102 determines the time associated with the model parameter operation (the communication time associated with the first operation) according to the data volume of the model parameter, the number of processors, and the communication bandwidth between processors in the groups. param =2*ψ / n*(n-1) / B, where ψ is the data volume of the model parameters, n is the number of processors, and B is the communication bandwidth between processors in the group.

[0043] In some embodiments, during the training of the model, for the processing of the model gradient, the computing device 102 determines the gradient sharding strategy. In the gradient non-sharding mode (first sharding strategy), the model gradient exists in each processor, and the computing device 102 determines that the time associated with the model gradient operation during the back propagation (the communication time associated with the second operation) is 0, t gradient =0. In the parameter group intra-sharding (second sharding strategy), multiple processors are divided into multiple groups and the processors in the group share the model gradient equally. The computing device 102 determines the time associated with the model gradient operation (the second operation-related communication time) during the back propagation process according to the amount of data of the model parameters involved in the update, the number of processors in the group, and the communication bandwidth between the processors in the group. The specific formula is: gradient =ψ′ / m*(m-1) / B′, where ψ′ is the amount of data of the model parameters involved in the update, m is the number of processors in the group, and B′ is the communication bandwidth between processors in the group. In the parameter global sharding (third sharding strategy), multiple processors evenly divide the model gradient, and the computing device 102 determines the time related to the model gradient operation (the communication time related to the second operation) during the back propagation process according to the amount of data of the model parameters involved in the update, the number of processors, and the communication bandwidth between processors in the group. The specific formula is: t gradient =ψ′ / n*(n-1) / B, where ψ′ is the amount of data involved in updating the model parameters, n is the number of processors, and B is the communication bandwidth between processors in the group.

[0044] Figure 5 A flowchart of a method 500 for determining a second communication time of the present specification is shown. In some embodiments, the data subset includes multiple data, and the multiple data are processed sequentially during the training iteration. When the computing device 102 processes the last data in the process of training the model, the communication time required for the aggregation operation of the model gradient at this time (the communication time related to the third operation) can be determined, wherein the aggregation operation can be, for example, the above-mentioned reduce-scatter operation or all-reduce operation.

[0045] In some embodiments, after the gradient calculation with respect to the last data in the data subset is completed, the computing device 102 applies the gradient to the parameter update, and the updated parameters need to be transmitted back to each processor, and the updated parameters are applied to the next iteration. The computing device 102 obtains the communication time associated with the synchronization operation of the updated model in one iteration (the communication time associated with the fourth operation), such as the time consumed by the all-gather operation. The computing device 102 calculates the communication time associated with the operation of the model gradient associated with the last data in the data subset in one iteration (the communication time associated with the third operation t sync_g ) and the communication time associated with the synchronization operation of the updated model (the communication time associated with the fourth operation t sync_p ) determines the communication time associated with the model update (the second communication time t update ).

[0046] It can be understood that the communication time associated with the above model gradient operation in one iteration (the communication time associated with the third operation t sync_g ) is related to the gradient sharding strategy and the optimizer parameter sharding strategy. In some embodiments, the computing device 102 determines the gradient sharding strategy and the optimizer parameter sharding strategy. Under the gradient non-sharding and optimizer parameter non-sharding strategies, the computing device 102 determines the communication time related to the operation of the model gradient in one iteration (the communication time related to the third operation) according to the number of processors, the amount of data of the model parameters involved in the update, and the communication bandwidth between the processors of the groups. The specific formula is: sync_g =2*ψ′ / n*(n-1) / B. Where ψ′ is the amount of data involved in updating the model parameters, n is the number of processors, and B is the communication bandwidth between processors in the group.

[0047] Under the strategy of gradient non-sharding and optimizer parameter intra-group sharding, the computing device 102 determines the communication time related to the operation of the model gradient in one iteration (the communication time related to the third operation) according to the number of processors, the number of processors in the group, the amount of data of the model parameters involved in the update, the number of model gradients, and the communication bandwidth between processors in the groups. The specific formula is: sync_g =ψ′ / n*(n-1) / B+ψ′ / m / g*(g-1) / B, where ψ′ is the amount of data involved in the update of the model parameters, n is the number of processors, m is the number of processors in a group, B is the communication bandwidth between processors in different groups, and g is the number of model gradients.

[0048] Under the non-sharding strategy for gradients and the global sharding strategy for optimizer parameters, the computing device 102 determines the communication time (the communication time related to the third operation) associated with the operation of the model gradient in one iteration according to the number of processors, the amount of data of the model parameters involved in the update, and the communication bandwidth between processors in the groups. The specific formula is: sync_g =ψ′ / n*(n-1) / B, where ψ′ is the amount of data involved in updating the model parameters, n is the number of processors, and B is the communication bandwidth between processors in the group.

[0049] Under the strategy of gradient intra-group sharding and optimizer parameter intra-group sharding, the computing device 102 determines the communication time related to the operation of the model gradient in one iteration (the communication time related to the third operation) according to the number of processors in the group, the amount of data of the model parameters involved in the update, the number of model gradients, and the communication bandwidth between processors in the groups. The specific formula is: sync_g =2*ψ′ / m / g*(g-1) / B, where ψ′ is the amount of data involved in updating the model parameters, m is the number of processors in the group, g is the number of model gradients, and B is the communication bandwidth between processors in the groups.

[0050] Under the strategy of gradient group sharding and optimizer parameter global sharding, the computing device 102 determines the communication time related to the operation of the model gradient in one iteration (the communication time related to the third operation) according to the number of processors in the group, the amount of data of the model parameters involved in the update, the number of model gradients, and the communication bandwidth between processors in the groups. The specific formula is: sync_g =ψ′ / m / g*(g-1) / B, where ψ′ is the amount of data of the model parameters involved in the update, m is the number of processors in the group, g is the number of model gradients, and B is the communication bandwidth between processors in the group. It can be understood that, when a sharding strategy other than the above sharding strategy is adopted, the computing device 102 determines that the time associated with the model gradient operation (communication time associated with the third operation) during one iteration is 0, and t sync_g =0.

[0051] It can be understood that the communication time associated with the operation of the updated model (the communication time associated with the fourth operation t sync_p ) is related to the parameter sharding strategy and the optimizer parameter sharding strategy. In some embodiments, the computing device 102 determines the parameter sharding strategy and the optimizer parameter sharding strategy. Under the non-sharding of parameters and the sharding strategy within the optimizer parameter group, the computing device 102 determines the communication time related to the operation of the updated model (the communication time related to the fourth operation) according to the number of processors in the group, the amount of data of the model parameters involved in the update, and the communication bandwidth between the processors in the group. The specific formula is: sync_p=ψ′ / m*(m-1) / B′. Where, ψ′ is the amount of data involved in updating the model parameters, m is the number of processors in the group, and B′ is the communication bandwidth between processors in the group.

[0052] Under the strategy of no parameter sharding and global sharding of optimizer parameters, the computing device 102 determines the communication time related to the operation of the updated model (the communication time related to the fourth operation) according to the number of processors, the amount of data of the model parameters involved in the update, and the communication bandwidth between processors in the groups. The specific formula is: sync_p =ψ′ / n*(n-1) / B, where ψ′ is the amount of data involved in updating the model parameters, n is the number of processors, and B is the communication bandwidth between processors in the group.

[0053] Under the parameter group-internal sharding and optimizer parameter global sharding strategies, the computing device 102 determines the communication time associated with the operation of the updated model (the fourth operation-related communication time) according to the number of processors in the group, the amount of data of the model parameters involved in the update, the number of model gradients, and the communication bandwidth between processors in the groups. The specific formula is: sync_p =ψ′ / m / g*(g-1) / B, where ψ′ is the amount of data of the model parameters involved in the update, m is the number of processors in the group, g is the number of model gradients, and B is the communication bandwidth between processors in the group. It can be understood that, when a sharding strategy other than the above sharding strategy is adopted, the computing device 102 determines that the communication time associated with the operation of the updated model (the communication time associated with the fourth operation) is 0, and t sync_g =0.

[0054] In some embodiments, since a single one-way time can reflect the computational complexity of the model when processing a single micro-batch data sample, the computing time consumed by the processor when executing its assigned task can be estimated by the time of one forward propagation. The computing device 102 obtains the time required for the forward propagation of the data subset for one iteration of the model. The computing device 102 can determine the computing time for model training by determining the size of the data set, the number of data subsets, and the time required for forward propagation. It can be understood that the computing device 102 estimates the computing time based on the time required for a single forward propagation based on a linear relationship.

[0055] Figure 6 The block diagram of a computing device 600 suitable for implementing an embodiment of the present invention is schematically shown. The computing device 600 can be used to implement the computing device 102. The computing device 600 can be used to implement the execution Figure 2 The apparatus of the method 200 is shown. Figure 6As shown, the computing device 600 includes a processing unit (CPU) 601, which can perform various appropriate actions and processes according to computer program instructions stored in a read-only memory (ROM) 602 or computer program instructions loaded from a storage unit 608 into a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the computing device 600 can also be stored. The CPU 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0056] Multiple components in the computing device 600 are connected to the I / O interface 605, including: an input unit 606, an output unit 607, and a storage unit 608. The processing unit 601 performs the various methods and processes described above, such as executing the method 200. For example, in some embodiments, the various processes or operations described above can be implemented as a computer software program, which is stored in a machine-readable medium, such as the storage unit 608. In some embodiments, part or all of the computer program can be loaded and / or installed on the computing device 600 via the ROM 602 and / or the communication unit 609. When the computer program is loaded into the RAM 603 and executed by the CPU 601, the various methods and processes described above can be executed, such as executing one or more operations of the method 200. Alternatively, in other embodiments, the CPU 601 can be configured to execute the various methods and processes described above, such as executing one or more actions of the method 200, by any other appropriate means (for example, by means of firmware).

[0057] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0058] The computer program instructions for performing the operation of the present invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, such as Smalltalk, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. Computer-readable program instructions may be executed entirely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be personalized by utilizing the state information of the computer-readable program instructions, and the electronic circuit may execute the computer-readable program instructions, thereby realizing various aspects of the present invention.

[0059] These computer-readable program instructions can be provided to a processor in a voice interaction device, a general-purpose computer, a special-purpose computer, or a processing unit of other programmable data processing devices, thereby producing a machine, so that when these instructions are executed by the processing unit of a computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other device to work in a specific manner.

[0060] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0061] The embodiments of the present invention have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or technical improvements in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

[0062] The above are only optional embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for determining a sharding strategy for model training, wherein the model can be trained using multiple sharding strategies, the method comprising: Determining a computing time of a plurality of processors in a process of training the model, the plurality of processors being partitioned based on a sharding strategy indicating how data of the model is partitioned among the plurality of processors; Based on the sharding strategy, determining the communication time of the multiple processors during the process of training the model; as well as In response to the larger one of the operation time and the communication time satisfying a time condition, the sharding strategy is determined to be a target sharding strategy among the multiple sharding strategies.

2. The method according to claim 1, wherein based on the sharding strategy, determining the communication time of the plurality of processors during the process of training the model comprises: Obtaining a data subset from a data set used to train the model, wherein the data subset is used for one iteration of the model; Based on the data subset, determining a first communication time of the model associated with forward propagation and backward propagation during the one iteration, and a second communication time of the model associated with updating the model during the one iteration; as well as The communication time is determined based on the first communication time and the second communication time.

3. The method according to claim 2, wherein the data subset is divided into a plurality of data, and based on the data subset, determining a first communication time of the model associated with forward propagation and backward propagation during the one iteration, and a second communication time of the model associated with updating the model during the one iteration comprises: Determining, based on data from the plurality of data, a communication time associated with a first operation of a model parameter during the forward propagation and the backward propagation, and a communication time associated with a second operation of a model gradient during the backward propagation; The first communication time is determined based on a communication time associated with the first operation, a communication time associated with the second operation, and a number of gradient accumulation steps.

4. The method according to claim 3, wherein determining, based on data in the plurality of data, a communication time associated with a first operation of a model parameter during the forward propagation and the backward propagation, and a communication time associated with a second operation of a model gradient during the backward propagation comprises: In a case where the model parameter is a first sharding strategy, determining that the communication time associated with the first operation is 0, the first sharding strategy indicating that the model parameter exists in each processor; determining a communication time associated with the first operation based on a data volume of the model parameter, a number of processors in a group, and a communication bandwidth between processors in the group, in a case where the model parameter is a second sharding strategy, wherein the second sharding strategy indicates that the plurality of processors are divided into a plurality of groups and the processors in the groups equally share the model parameter; as well as In the case where the model parameters are the third sharding strategy, the communication time associated with the first operation is determined based on the data volume of the model parameters, the number of the multiple processors, and the communication bandwidth between the multiple processors between groups, and the third sharding strategy instructs the multiple processors to evenly divide the model parameters.

5. The method of claim 3, wherein determining, based on data in the plurality of data, a communication time associated with a first operation of a model parameter during the forward propagation and the backward propagation, and a communication time associated with a second operation of a model gradient during the backward propagation comprises: In a case where the model gradient is a first sharding strategy, determining that the communication time associated with the second operation is 0, the first sharding strategy indicating that the model gradient exists in each processor; In the case where the model gradient is a second sharding strategy, determining a communication time associated with the second operation based on the amount of data of the model parameters involved in the update, the number of processors in a group, and the communication bandwidth between the processors in the group, wherein the second sharding strategy indicates that the plurality of processors are divided into a plurality of groups and the processors in the groups evenly share the model gradient; and In the case where the model gradient is the third sharding strategy, the communication time associated with the second operation is determined based on the amount of data of the model parameters involved in the update, the number of the multiple processors, and the communication bandwidth between the multiple processors between groups. The third sharding strategy instructs the multiple processors to evenly divide the model gradient.

6. The method according to claim 2, wherein the data subset is divided into a plurality of data, and based on the data subset, determining a first communication time of the model associated with forward propagation and backward propagation during the one iteration, and a second communication time of the model associated with updating the model during the one iteration comprises: determining a communication time of the model associated with a third operation of a model gradient during the one iteration; determining a communication time of the model associated with a fourth operation of the updated model parameters during the one iteration; as well as The second communication time is determined based on the communication time associated with the third operation and the communication time associated with the fourth operation.

7. The method of claim 6, wherein determining the communication time of the model associated with the third operation of the model gradient during the one iteration comprises: Determining a sharding strategy for the model gradient and a sharding strategy for the optimizer parameters; as well as Based on the sharding strategy of the model gradient and the sharding strategy of the optimizer parameters, a communication time associated with the third operation is determined.

8. The method according to claim 7, wherein determining the communication time associated with the third operation based on the sharding strategy of the model gradient and the sharding strategy of the optimizer parameter comprises: When the model gradient is the first sharding strategy and the optimizer parameter is the first sharding strategy or the third sharding strategy, determining a communication time associated with the third operation based on the number of the plurality of processors, the amount of data of the model parameters involved in updating, and the communication bandwidth between the plurality of processors between groups; When the model gradient is a first sharding strategy and the optimizer parameter is a second sharding strategy, determining a communication time associated with the third operation based on the number of the plurality of processors, the number of processors in a group, the amount of data of the model parameters involved in the update, the number of the model gradients, and the communication bandwidth between the plurality of processors in the group; as well as When the model gradient is the second sharding strategy and the optimizer parameters are the second sharding strategy or the third sharding strategy, the communication time associated with the third operation is determined based on the number of processors in the group, the amount of data of the model parameters involved in the update, the number of model gradients, and the communication bandwidth between the multiple processors between groups.

9. The method of claim 6, wherein determining a communication time of the model associated with a fourth operation of the updated model parameters during the one iteration comprises: Determining a sharding strategy for the model parameters and a sharding strategy for the optimizer parameters; as well as Based on the sharding strategy of the model parameters and the sharding strategy of the optimizer parameters, a communication time associated with the fourth operation is determined.

10. The method according to claim 9, wherein determining the communication time associated with the fourth operation based on the sharding strategy of the model parameters and the sharding strategy of the optimizer parameters comprises: When the model gradient is a first sharding strategy and the optimizer parameter is a second sharding strategy, determining the communication time associated with the fourth operation based on the number of processors in the group, the amount of data of the model parameters involved in the update, and the communication bandwidth between the processors in the group; When the model gradient is the first sharding strategy and the optimizer parameter is the third sharding strategy, determining the communication time associated with the fourth operation based on the number of the plurality of processors, the amount of data of the model parameters involved in the update, and the communication bandwidth between the plurality of processors between groups; as well as When the model gradient is the second sharding strategy and the optimizer parameters are the third sharding strategy, the communication time associated with the fourth operation is determined based on the number of processors in the group, the amount of data of the model parameters involved in the update, the number of model gradients, and the communication bandwidth between the multiple processors between groups.

11. The method of claim 1 , wherein determining the operation time of a plurality of processors in the process of training the model comprises: Obtaining a data subset from a data set used to train the model, wherein the data subset is used for one iteration of the model; Determining a time for forward propagation of the model based on the data subset; as well as The operation time is determined based on the forward propagation time.

12. A device for determining a sharding strategy for model training, wherein the model can be trained using multiple sharding strategies, the device comprising: an operation time determination module configured to determine the operation time of a plurality of processors in the process of training the model, the plurality of processors being divided based on a sharding strategy indicating how the data of the model is divided among the plurality of processors; A communication time determination module, configured to determine the communication time of the plurality of processors in the process of training the model based on the sharding strategy; as well as The sharding strategy determination module is configured to determine the sharding strategy as a target sharding strategy among the multiple sharding strategies in response to the larger one of the operation time and the communication time satisfying a time condition.

13. A computing device comprising: processor; as well as A memory coupled to the processor, the memory having instructions stored therein, which, when executed by the processor, cause the computing device to perform the method according to any one of claims 1 to 11.

14. A computer program product comprising a computer program, the computer program being executed by a processor to implement the method according to any one of claims 1 to 11.

Images