Method, apparatus, electronic device, recording medium, and program for optimizing model training device performance

Abstract

To provide a method, an apparatus, an electronic device, a storage medium, and a program for optimizing model training device performance, in the field of artificial intelligence, in particular in the fields of deep learning, training of a large scale model, and a distributed parallel approach.SOLUTION: A specific solution includes determining a communication timing for a target model block at a target sort position of a current model training device so as to facilitate aggregated communication performed on a model block at the target sort position in synchronization with another model training device in a plurality of model training devices, and performing aggregated communication for a gradient in a reverse direction of the target model block at the communication timing.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present disclosure relates to the technical fields of artificial intelligence, in particular deep learning, training of large-scale models, distributed parallel methods, and the like. [Background technology]

[0002] A large-scale model is a machine learning model that has a large number of parameters and a complex structure. Large-scale models can process large amounts of data, which can improve the accuracy and generalization ability of machine learning models. Large-scale models are more complex and flexible, and can handle more complex problems. Large-scale models have more parameters and a more complex structure, which can more accurately represent data distributions and learn more complex features, which can improve the accuracy and performance of the model. Summary of the Invention [Problem to be solved by the invention]

[0003] Large-scale models have a wider range of application scenarios and higher expressive capabilities, but they need to process large amounts of data and parameters, take longer to train and infer, and consume more computing resources. [Means for solving the problem]

[0004] The present disclosure provides a method, apparatus, electronic device, storage medium and program for optimizing the performance of a model training device.

[0005] According to one aspect of the present disclosure, there is provided a method for optimizing performance of a model training device, the method comprising: Determining a communication timing of a current model training device for a target model block at a target sorting position, so as to facilitate synchronous aggregate communication with other model training devices in a plurality of model training devices for the model block at the target sorting position, wherein the current model training device is an arbitrary training device among the plurality of model training devices, the plurality of model training devices are used to train a same target model, the target model is divided into a plurality of model stages, each model stage includes a plurality of ordered model blocks, and in the process of training the target model using a distributed parallel method, bubbles are generated due to an increase in the calculation time length of the model training device caused by communication operations; and performing aggregate communication for the backward gradient of the target model block during communication timing.

[0006] According to another aspect of the present disclosure, there is provided an apparatus for optimizing performance of a model training device, the apparatus comprising: A determination module for determining a communication timing of a current model training device for a target model block at a target sorting position, so as to facilitate synchronous aggregate communication between other model training devices in the multiple model training devices for the model block at the target sorting position, where the current model training device is an arbitrary training device among the multiple model training devices, the multiple model training devices are used to train a same target model, the target model is divided into multiple model stages, each model stage includes multiple model blocks in order, and in the process of training the target model using a distributed parallel method, a bubble occurs due to an increase in the calculation time of the model training device caused by a communication operation; and an execution module for performing aggregate communication on the backward gradient of the target model block at the communication timing.

[0007] According to another aspect of the present disclosure, there is provided an electronic device, the device comprising: At least one processor; a memory in communication with the at least one processor; The memory stores instructions executable by the at least one processor, which, when executed by the at least one processor, cause a performance optimization method of any one of the model training device embodiments of the present disclosure to be performed.

[0008] According to another aspect of the present disclosure, there is provided a non-transitory computer-readable storage medium storing computer instructions for causing a computer to execute a model training device performance optimization method according to any one of the embodiments of the present disclosure.

[0009] According to another aspect of the present disclosure, there is provided a program for implementing a performance optimization method for a model training device according to any one of the embodiments of the present disclosure, when executed by a processor.

[0010] In the embodiment of the present disclosure, in multiple model training devices that train a target model in a distributed manner, a data parallel method combined with a 1F1B interleaved scheduling pipeline parallel method is used to realize synchronous communication between the multiple model training devices, thereby effectively reducing the bubble rate and improving the performance of the model training devices.

[0011] It should be understood that the contents described herein are not intended to describe key points or important features of the embodiments of the present disclosure, nor are they used to limit the scope of the present disclosure, other features of the present disclosure are facilitated by the following specification. The accompanying drawings are for better understanding of the solutions of the present disclosure, but are not intended to limit the present disclosure. [Brief description of the drawings]

[0012] [Figure 1] FIG. 2 is a schematic diagram illustrating a flowchart of a performance optimization method for a model training device according to an embodiment of the present disclosure. [Diagram 2] FIG. 13 is a timing diagram of a model training process according to one embodiment of the present disclosure. [Diagram 3] FIG. 13 is another timing diagram of another model training process according to another embodiment of the present disclosure. [Figure 4] FIG. 13 is another timing diagram of another model training process according to another embodiment of the present disclosure. [Diagram 5] FIG. 1 is a schematic diagram illustrating a configuration of a performance optimization apparatus for a model training device according to an embodiment of the present disclosure. [Figure 6] FIG. 2 is a block diagram of an electronic device for implementing a performance optimization method for a model training device according to an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] Hereinafter, exemplary embodiments of the present disclosure will be described with reference to the drawings. Various details of the embodiments of the present disclosure are included herein for ease of understanding, and they should be considered as merely exemplary. Therefore, those skilled in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, in the following description, descriptions of known functions and structures are omitted for clarity and simplicity.

[0014] The terms "first," "second," and "third," etc. in the embodiments and claims of the specification of this disclosure and in the drawings described above are intended to distinguish between similar objects without necessarily being used to describe a particular order or priority. Furthermore, the terms "comprise" and "have" and variations thereof are intended to be non-exclusive inclusions, e.g., inclusion of a series of steps or units. Methods, systems, products, or devices need not be limited to explicitly recited steps or units, but may include other steps or units not explicitly recited or inherent to these processes, methods, products, or devices.

[0015] With the advent of the big data era, the amount of data generated in people's lives continues to increase. In order to process large amounts of data, solve various complex problems, and meet users' requirements for high efficiency in deep learning tasks, excellent models are often obtained through complex training processes. In order to better complete the model training, the method of distributed parallel training is used to accelerate the training. In related technologies, this is usually realized in two ways: data parallelism and pipeline parallelism. The two methods are introduced below.

[0016] Among them, the principle of data parallelism is to split the data set into multiple parts, store the complete training parameters in each model training device, and process one sub-data set to accelerate the model training process.

[0017] Pipeline parallelism is one of the methods of model parallelism, whose principle is to reduce the memory consumption of a single model training device by dividing the model into different model training devices, and to realize ultra-large-scale model training by transmitting data between adjacent devices via communication links.

[0018] However, data parallelism requires the memory of the model training device to accommodate the complete model, which makes it poorly scalable. In addition, when splitting a model in pipeline parallelism, the load balancing and overall communication time of the model training device are not comprehensively considered. When using pipeline parallel training alone, the model training device is underutilized, and the efficiency of model training needs to be improved.

[0019] Therefore, in the related art, a large-scale model can be trained using a data parallel method and a pipeline parallel method. However, in this method, the overall bubble rate needs to be improved in the process of combining backward gradient communication. Bubbles lead to the waste of computational resources of the model training device. In order to improve the effect of model training and improve the performance of the large-scale model training device, this application proposes a performance optimization method for a model training device that combines data parallelism and pipeline parallelism to realize synchronous communication, improves the computation overlap rate of the model training device, reduces the overall bubble rate, and thereby improves the performance of the model training device. A specific implementation process of the performance optimization method for the model training device is shown in FIG. 1, for example, and includes the following:

[0020] In S101, a communication timing of the current model training device for a target model block at a target sorting position is determined so as to facilitate synchronous aggregate communication with other model training devices in the multiple model training devices for the model block at the target sorting position.

[0021] Here, the current model training device is any training device among a plurality of model training devices, and the plurality of model training devices are used to train the same target model, and the target model is divided into a plurality of model stages, and each model stage includes a plurality of ordered model blocks. In the process of training the target model using a distributed parallel method, bubbles occur due to an increase in the calculation time of the model training device caused by communication operations.

[0022] In S102, at the communication timing, aggregate communication is performed for the backward gradient of the target model block.

[0023] The target model in the embodiment of the present disclosure is a large-scale model. The large-scale model is a machine learning model with huge parameter scale and complexity. In the field of deep learning, the large-scale model usually refers to a neural network model with millions to billions of parameters. These models require a large amount of computational resources and storage space to train and store, and often require distributed computing and specialized hardware acceleration techniques.

[0024] Larger models are designed and trained to provide stronger and more accurate model performance for more complex and massive datasets and tasks. Larger models are usually able to learn finer patterns and rules and have stronger generalization and representation capabilities.

[0025] In the embodiment of the present disclosure, the communication timing of multiple model training devices is synchronized to maximize the parallel computing capabilities of the model training devices. By controlling the synchronous communication between multiple model training devices and rationally arranging the communication timing, the bubble rate of communication bubbles can be reduced, the overlap of calculations can be improved, the calculation time can be shortened, the communication efficiency can be improved, and the performance of the model training device can be improved.

[0026] When performing model training using the pipeline parallel approach, the target model is divided into multiple model stages according to the pipeline parallel approach. For example, as shown in FIG. 2, the target model is divided into four model stages, namely model stage 0, model stage 1, model stage 2, and model stage 3. Each model stage is deployed to a one-to-one corresponding model training device. Each model stage may be divided into multiple model blocks, and each model block includes one or more neural network layers. When including multiple neural network layers, the multiple neural network layers are consecutive. For example, when the target model has neural network layers numbered in sequence from 0 to 15, the target model has a total of 16 neural network layers. When deployed to four model training devices for training, four neural network layers can be deployed to each model training device. Suppose the corresponding target model can be divided into four model stages, and each model stage is divided into two model blocks, as shown in FIG. 2, the neural network layers with numbers 0, 1, 8, and 9 are included in the first model stage (i.e., model stage 0 in FIG. 2), which construct two model blocks with numbers <0, 1> and <8, 9>, respectively; the neural network layers with numbers 2, 3, 10, and 11 are included in the second model stage (i.e., model stage 1 in FIG. 2), which construct two model blocks with numbers <2, 3> and <10, 11>, respectively; the neural network layers with numbers 4, 5, 12, and 13 are included in the third model stage (i.e., model stage 2 in FIG. 2), which construct two model blocks with numbers <4, 5> and <12, 13>, respectively; and the neural network layers with numbers 6, 7, 14, and 15 are included in the fourth model stage (i.e., model stage 3 in FIG. 2), which construct two model blocks with numbers <6, 7> and <14, 15>, respectively.

[0027] The description will be continued based on the model stages shown in FIG. 2. During training, first, forward calculation is performed using the first model block included in the first model stage, and after the forward calculation of the first model block included in the first model stage is completed, the forward calculation of the first model block included in the second model stage is performed, and the forward calculation of the first model block included in the fourth model stage is sequentially propagated downward until the forward calculation of the first model block included in the fourth model stage is completed. The forward calculation of the second model block included in the first model stage is started, the calculation result is propagated downward, and after the forward calculation of the second model block is completed by propagating to the fourth model stage, the forward calculation and the backward calculation are started to be performed alternately. Then, the backward calculation result is sequentially propagated upward until all the backward calculations of the first model block included in the first model stage are completed, that is, until the backward propagation of the 16-layer neural network is completed.

[0028] In an embodiment of the present disclosure, the target model is trained using a data parallel method combined with a pipeline parallel method of 1F1B interleaved scheduling, and the target model is divided into a number of model stages according to the pipeline parallel method, and each model stage is assigned to a corresponding model training device for training, and in the data parallel method, each batch (mini batch) of training data is divided into a number of micro batches. As shown in FIG. 2, the training data is divided into a number of mini batches, and each mini batch is divided into four micro batches. In FIG. 2, the same micro batch is numbered with the same numeral. A represents forward calculation for the first model block included in each model stage, and B represents forward calculation for the second model block included in each model stage. b represents backward gradient calculation for the second model block included in each model stage. a represents backward gradient calculation for the first model block included in each model stage. Each micro batch goes through the processing of each model block in sequence, which corresponds to completing the forward processing operation in the order of the neural network layers of the target model. In the backward direction, the update order of the model blocks is exactly the opposite of the forward process.

[0029] For example, continuing to take the model partitioning method of Figure 2 as an example, the forward calculation of the first model block included in each model stage during the forward process in the timing diagram of Figure 2 is represented by a rectangular box filled with gray, and the forward calculation of the second model block included in each model stage is represented by a rectangular box filled with white. The backward calculation of the second model block included in each model stage during the backward process in Figure 2 is represented by a rectangular frame filled with black, and the backward calculation of the first model block included in each model stage is represented by a rectangular frame filled with dot shading (all such rectangular frames contain the letter a). The forward process of the micro batch is carried out through the first model block <0, 1> in model stage 0, then propagates the result to the first model block <2, 3> in model stage 1, then propagates the result to the first model block <4, 5> in model stage 2, then propagates the result to the first model block <6, 7> in model stage 3, then propagates the result to the second model block <8, 9> in model stage 0, then propagates the result to the second model block <10, 11> in model stage 1, then propagates the result to the second model block <12, 13> in model stage 2, and finally propagates the result to the second model block <14, 15> in model stage 3. Thus, the process of forward propagation of the entire 16-layer neural network is completed by the cooperation of different model stages and different models. The processing order of the backward process is the exact opposite of the forward process. For example, referring to the black rectangular frame including the letter b in FIG. 2, the backward calculation of the second model block included in model stage 3 is first performed, then the backward calculation of the second model block included in model stage 2 is performed, then the backward calculation of the second model block included in model stage 1 is performed, and then the backward calculation of the second model block included in model stage 0 is performed. After the backward calculation of the second model block is completed, referring to the dot-shaded rectangular frame including the letter a in FIG. 2, the backward calculations of the first model block included in model stage 3, model stage 2, model stage 1, and model stage 0 are performed in that order.

[0030] In the embodiment of the present disclosure, a 1F1B interleaved scheduling scheme is used so that each model training device can alternately perform forward and backward processes. In 1F1B interleaved scheduling, each device can perform calculations on a subset of layers (i.e., model blocks) rather than a set of consecutive layers.

[0031] The 1F1B mode is a method in which forward and backward calculations are crossed. In the 1F1B mode, forward and backward calculations are crossed, and unnecessary intermediate variables can be released in a timely manner.

[0032] In the interleaved scheduling method of 1F1B, a target model is trained using multiple mini batches, and each mini batch is divided into multiple micro batches. A dependency relationship is established between the computation tasks of multiple micro batches of data on the same model training device. For example, each model training device performs forward computation on the i-th micro batch data until the target number of backward computations is completed, and then performs backward computation directly on the result of the forward computation of the i-th micro batch. After the target number of backward computations are completed, aggregate communication is performed, making it easy to update the model parameters based on the gradient obtained in the backward computation.

[0033] As shown in FIG. 2, the gray-filled rectangular box containing the letter A and the white-filled rectangular box containing the letter B are the forward calculation process, the black-filled rectangular box is the backward calculation process for the white rectangular box, and the dot-shaded rectangular box containing the letter a is the backward calculation process for the gray rectangular box. Each mini batch is divided into four micro batches. The backward calculation process is introduced by taking the second model block as an example. After the second model block included in the model stage 0 in FIG. 2 completes the forward calculation in the model stage 3, it performs backward gradient calculation for the second model block <14, 15> included in the model stage 3 (for example, 1b in the model stage 3 in FIG. 2), and propagates the calculation result to the model training device of the model stage 2, and the model training device performs backward gradient calculation for the second model block <12, 13> included in the model stage 2 (for example, 1b in the model stage 2 in FIG. 2), and propagates the processing result to the model training device of the model stage 1. The model training device of the model stage 1 performs backward gradient calculation for the second model block <10, 11>, and then propagates the processing result to the model training device of the model stage 0. The model training device of the model stage 0 performs backward gradient calculation for the second model block <8, 9>. Then, the processing result is sequentially propagated to the model stage 3, and the backward gradient calculation for the first model block <6, 7> included in the model stage 3 is completed, and the processing result is propagated to the model stage 2, and the backward gradient calculation for the first model block <4, 5> included in the model stage 2 is completed, and the processing result is propagated to the model stage 1, and the backward gradient calculation for the first model block <2, 3> included in the model stage 1 is completed, and the processing result is propagated to the model stage 0, and the backward gradient calculation for the first model block <0, 1> included in the model stage 0 is completed. Thus, through the cooperation of different model stages and different models, the backward propagation process of the entire 16-layer neural network is completed.

[0034] In the embodiment of the present disclosure, a data parallel method combined with a 1F1B interleaved scheduling pipeline parallel method is used to perform distributed training, and forward and backward calculations are alternately performed, thereby freeing up unnecessary intermediate variables and reducing memory resource consumption. In addition, by training a model using multiple batches and dividing each batch into multiple microbatches, memory efficiency can be improved and the model training speed can be increased. For a large training data set, loading the entire batch into memory at once can cause memory shortage problems, but training a model using multiple batches can be loaded and processed one by one, making effective use of memory resources. By dividing multiple batches into multiple microbatches, they can be processed in parallel, thereby increasing the training speed.

[0035] In some other examples, each model block may include one neural network layer as well as multiple neural network layers. For example, in FIG. 3, the corresponding target model is divided into four model stages, and each model stage includes four model blocks arranged in sequence. Here, each model training device performs calculations for four model blocks, and each model block includes one neural network layer. As shown in FIG. 3, model stage 0 has four neural network layers with 0, 4, 8, and 12, which are also model blocks and neural network layers; model stage 1 has four neural network layers with 1, 5, 9, and 13, which are also model blocks and neural network layers; model stage 2 has four neural network layers with 2, 6, 10, and 14, which are also model blocks and neural network layers; and model stage 3 has four neural network layers with 3, 7, 11, and 15, which are also model blocks and neural network layers. Here, in the first model block 0 included in the model stage 0, a rectangular frame with diagonal shading and filled with five-pointed stars represents the execution of a forward process, and the processing result is sequentially propagated to the first model block 3 included in the model stage 3. The processing result is then sequentially propagated to the second model block 4 included in the model stage 0. The second model block represents the execution of a forward process using a rectangular frame with vertical stripes and filled with squares in FIG. 3, and the processing result is sequentially propagated to the second model block 7 included in the model stage 3. The processing result is then sequentially propagated to the third model block 8 included in the model stage 0, and in the third model block, a forward process is executed on the data represented by a rectangular frame with horizontal stripes and filled with circles, and the result is sequentially propagated to the third model block 11 included in the model stage 3. The processing result is then sequentially propagated to the fourth model block 12 included in model stage 0, where a rectangular box with dot stripes and arrows filled in represents the execution of the forward process, and the result is sequentially propagated to the fourth model block 15 included in model stage 3.The backpropagation process starts to run after the forward propagation process has finished running, and the order of updating the backpropagation process and the forward propagation model block is exactly the opposite.

[0036] According to this solution, each model training device in the pipeline is assigned multiple pipeline stages, and the computational complexity of each pipeline stage is reduced.

[0037] In addition, the number of model stages into which the target model can be divided, the number of model blocks included in each model stage, and the number of neural network layers included in each model block may be determined according to actual needs, and are not limited thereto in the embodiments of the present disclosure.

[0038] Both Fig. 2 and Fig. 3 can be understood as timing diagrams of the model training process. The horizontal axis in Fig. 2 and Fig. 3 represents time, and the rectangular boxes stacked vertically represent the types of operations performed at the corresponding times. Among them, the operation types are divided into two parts including calculation and communication. As shown in Fig. 3, the part including the relatively dense rectangular boxes on each model training device is the calculation part. In Fig. 3, the rectangular boxes filled with five-pointed stars in diagonal shading, squares in vertical shading, circles in horizontal shading, and arrows in dot shading respectively represent the forward calculation part, and the rectangular boxes filled with water droplets in light gray shading, hearts in black shading, diamonds in white shading, and triangles in dark gray shading respectively represent the backward calculation part. The part including the rectangular box starting with the alphabet A at the bottom is the communication part.

[0039] Both Fig. 2 and Fig. 3 show the 1F1B interleaved scheduling method. However, in the scheduling process based on the 1F1B mode, each model training device uses asynchronous communication, i.e., the backward gradient communication process of the model block at the same sorting position is completed in a distributed manner at different times. Asynchronous communication leads to the generation of extra bubbles in the calculation process, which increases the calculation time, which causes a "cumulative effect" and increases the overall end-to-end bubble rate. As shown in Fig. 3, the communication of each model stage (e.g., A0, B0, C0, D0) is performed in a distributed manner at different times, and in the process of backpropagation, the bubbles between the rectangular box filled with a water drop shape in the light gray shading and the rectangular box filled with a heart shape in the black shading become gradually larger, which causes a "cumulative effect", i.e., multiple bubbles are generated cumulatively, which increases the calculation time length and increases the overall end-to-end bubble rate.

[0040] In order to eliminate such a "cumulative effect", in the embodiment of the present disclosure, in the 1F1B interleaved scheduling of the parallel pipeline parallel method, it is necessary to control all model training devices participating in the pipeline calculation to synchronize the data parallel gradient communication. As shown in FIG. 4, each model stage communicates at the same communication timing, and with the process of the backward calculation result propagating upward, compared with FIG. 3, the bubble between the rectangular frame filled with a water drop shape in light gray shading and the rectangular frame filled with a heart shape in black shading becomes gradually smaller, and the bubble is obviously reduced during the entire model training process, and the calculation time during the backpropagation process is reduced. The model training efficiency can be improved, and the bubble rate can be reduced, and the overall performance of the model training device can be improved.

[0041] In the embodiment of the present disclosure, there are two implementation methods for the current model training device to determine the communication timing for the target model block at the target sorting position:

[0042] In method 1, each model training device can determine the communication timing of the aggregate communication of each model block according to the communication method.

[0043] In the embodiment of the present disclosure, synchronous communication may be realized by a communication mechanism between different model training devices to enable synchronous aggregate communication to eliminate bubbles. It can be implemented as follows: in response to receiving a synchronization message, determine that the next calculation stage is the communication timing for the target model block, and the synchronization message is issued by the model training device of the first model stage of the multiple model stages to the model block at the target sorting position.

[0044] It should be understood that this synchronization message can be issued after the model training device of the first model stage completes the backward gradient calculation for the model block at the target sorting position, so that multiple model training devices can easily perform backward communication for the model block at the same sorting position. For example, in Figure 4, after the communication timing of the model stage 0 is determined, the communication timing of the model stages 1, 2, and 3 are automatically shifted backward by a rank (calculation stage) step to synchronize with the communication timing of the model stage 0.

[0045] In the embodiments of the present disclosure, multiple model training stages correspond to the first model stage of one model training device, and the multiple model training devices perform synchronous communication, thereby improving the overlap of calculations, reducing the bubble rate, and improving the performance of the model training device.

[0046] As shown in FIG. 4, in the first model stage, A0 is a part that performs aggregate communication for the backward gradient of the model block 12 whose sorting position is the fourth, and the time when the aggregate communication is started is the communication timing. B0 is a part that performs aggregate communication for the backward gradient of the model block 8 whose sorting position is the third. C0 is a part that performs aggregate communication for the backward gradient of the model block 4 whose sorting position is the second. D0 is a part that performs aggregate communication for the backward gradient of the model block 0 whose sorting position is the first. From this, it can be seen that the order of aggregate communication is reversed to the order of forward calculation of the model block. Therefore, the same is true for the parts that perform aggregate communication for each target model block in model stages 1, 2, and 3, and the target model block and its aggregate communication part may correspond one-to-one, and this disclosure will omit details thereof.

[0047] In method 2, each model training device can determine the communication timing of the aggregate communication of each model block by itself.

[0048] This can be realized as follows: if the target constraint is satisfied, determine that the next calculation step is the communication timing for the target model block; The goal constraint is 1) The number of times that the current model training device calculates the backward gradient for the target model block is equal to or greater than the target number; and 2) At the communication timing, in a first model stage among the multiple model stages, calculation of the backward gradient of the model block at the target sorting position is completed.

[0049] As shown in FIG. 2, the first model stage is model stage 0, and the model blocks in the first model stage include two model blocks, <0, 1> and <8, 9>. In model stage 0 in FIG. 2, the gray rectangular frame and the white rectangular frame represent the forward calculation process of the model blocks at different sorting positions. The black rectangular frame is the backpropagation process for the white rectangular frame, and the dot-shaded rectangular frame is the backpropagation process for the gray rectangular frame. As can be seen from FIG. 2, the backward gradient calculation of the model block at the target sorting position in the first model stage is completed, which can be understood by taking the second model block as an example, that the backward gradient calculation for the black rectangular frame of the same mini batch is completed in model stage 0.

[0050] In the embodiment of the present disclosure, synchronous communication is performed according to the target constraint conditions, and each model training device can determine the timing of the synchronous communication by itself, so that the timing of communication can be controlled more accurately and communication can occur at the required timing. Accurately controlling the communication timing according to the target constraint conditions can shorten the calculation time and reduce the bubble rate, thereby improving the performance of the model training device and thus improving the model training efficiency.

[0051] In order to determine whether the target constraint condition is satisfied, the embodiment of the present disclosure exemplarily provides the following two implementation methods.

[0052] In the method for determining the target constraints 1), Determining that the target constraint is satisfied can be achieved as follows: determine an accumulated amount of backward gradient to be synchronized of the target model block; and determine that the target model block satisfies the target constraint if the accumulated amount is greater than or equal to a target number and the model block at the target sorting position can start aggregate communication in a first model stage among multiple model stages.

[0053] In implementation, the number of backward calculations for each model block can be tallied individually to easily determine whether the model block meets the communication requirements.

[0054] Whether the model block at the target sorting position in the first model stage can start aggregate communication can be estimated from the time length required for forward calculation and backward calculation. A specific estimation method can be a neural network, or can be obtained by inference according to the calculation complexity of the corresponding model block. Although a specific inference process is not described in this disclosure, any inference method that can obtain the above solution can be used in this disclosure.

[0055] In the embodiments of the present disclosure, gradient accumulation is a very simple computational method, and using this method to count the number of backward calculations can save computational resources, reduce communication overhead, and improve the efficiency of model training.

[0056] In the method for determining the target constraints 2), Determining that the target constraint is satisfied can be achieved as follows: for the number of backward gradient calculations by the current model training device, perform an accumulative count based on a default value; and if the accumulative count is greater than or equal to a target threshold, and it is determined that an integer number of batches of backward gradient calculations have been completed based on the accumulative count, then determine that the target model block at the target sorting position satisfies the target constraint.

[0057] In this, a target threshold is determined based on a target number of times such that the number of times the current model training device calculates a backward gradient for the target model block is equal to or greater than the target number of times.

[0058] In the embodiment of the present disclosure, in the scheduling process, the model training device performs one backward gradient calculation for each micro batch and one gradient accumulation count. If the target number Acc_Step is 8, aggregate communication is performed when the accumulated count of the backward gradient calculation performed for one model block is 8 or more. In FIG. 2, after performing the backward gradient calculations indicated by 1b, 2b, 3b, and 4b shown in black rectangular frames for the second model block included in model stage 0, the backward gradient calculations accumulated later are for the first model block, and the backward gradients for the second model block can be accumulated up to 8 times before performing aggregate communication for the second model block, so that the same model block can complete the Acc_Step number of times.

[0059] In the embodiment of the present disclosure, the current model training device determines the communication timing of the target model block included therein without being affected by other model training devices. Each model training device identifies the communication timing of the backward gradient calculation result of each model block according to the constraint condition, and interacts the communication timing between the devices. In addition, the determination of the cumulative count and the target preset in the entire process is a very simple calculation operation, and the consumption of calculation resources is small and can be neglected, thus improving the efficiency of model training.

[0060] In the embodiment of the present disclosure, different solutions are provided for identifying communication timing based on different settings of the default value of the cumulative count. The setting method of the default value is obtained by experimental inference according to bubbles that need to be removed for synchronous communication, and the inference process is not described here. The communication timing in different default setting methods will be described below.

[0061] (1) Identify the timing of aggregate communication based on the first type default value

[0062] In an embodiment of the present disclosure, when the default value is NULL, the target threshold is determined based on the number of backward gradient calculations, the number of model stages, and the number of model blocks contained in each model stage.

[0063] For example, the default value is set to NULL, and the target threshold determined based on the number of backward gradient calculations, the number of model stages, and the number of model blocks included in each model stage is (Acc_Step / / PP - 1) * PP *VP, where Acc_Step represents the number of backward gradient calculations, PP represents the number of divided model stages, VP represents the number of model blocks included in each model stage, and / / represents rounding down.

[0064] In the embodiment of the present disclosure, the default value of the number of backward gradient calculations by the current model training device is set to NULL, so that the memory occupation amount can be reduced. In addition, the target threshold is determined based on the number of backward gradient calculations, the number of model stages, and the number of model blocks included in each model stage, so that the model training device can timely determine the communication timing, reduce the bubble rate, and improve the efficiency of model training.

[0065] In some embodiments, determining that an integer number of batches of backward gradients have been computed based on the cumulative count may be implemented as follows:

[0066] The cumulative count and the difference between the model stage number of the current model training device and the target threshold are determined, and if the difference is divisible by the target number, it is determined that an integer number of batches of backward gradient calculations have been completed.

[0067] For example, a counter is set for each model training device, and if the default value is NULL, counter=0.

[0068] In the embodiment of the present disclosure, the model training device completes the training of the target model based on the 1F1B interleaved scheduling method, performs 1F1B interleaved scheduling of VP, performs one backward calculation, and adds 1 to the count of the counter, so that counter=counter+1.

[0069] When the counter of the rank-th computing device (i.e., the rank-th model stage) satisfies the following condition, VP-(counter-rank-(Acc_Step / / PP - 1) * PP *VP) / / PP-th model block is determined as the target model block at the target sorting position, and communication is made to the target model block in parallel with the computation flow, where rank is the model stage number of the current model training device.

[0070] Condition a), counter is greater than (Acc_Step / / PP - 1) * PP *VP.

[0071] Condition b), counter-rank - (Acc_Step / / PP - 1) * PP *VP is divisible by Acc_Step.

[0072] Here, condition b is to determine the difference between the cumulative count and the model stage number and the target threshold of the current model training device, and if the difference is divisible by the target number, determine that an integer number of batches of backward gradient calculations have been completed.

[0073] In the embodiments of the present disclosure, by combining the difference between the cumulative count and the model stage number held by the current model training device with a target threshold, the communication timing of the target model block can be flexibly controlled according to the model stage number held by the current model training device; and by determining that the backward gradient calculation of an integer number of batches has been completed, the process of model training can be better controlled, memory space usage can be saved, and the efficiency of model training can be improved.

[0074] For example, as shown in FIG. 4, PP is set to 4, VP is set to 4, and Acc_Step is set to 8, that is, the target model is divided into four model stages, each mini batch is divided into four micro batches, and the backward gradient accumulation count is 8. When the default value is NULL, counter=0, the target threshold is 16. Take model stage 0 as an example, each micro batch performs one backward calculation on each of the four model training devices until the counter count reaches 16 after the backward calculation of the target model block is completed, and the backward gradient accumulation count is performed once after the backward calculation of each micro batch is completed, and the counter count is increased by 1. At this time, the value of counter-rank - (Acc_Step / / PP - 1) * PP *VP is 0, which can be divided by Acc_Step, and the counter count is greater than 16 when the first backward calculation is performed on the micro batch of the next model block. Since it can be determined that this is the communication timing of the target model block, the VP-(counter-rank-(Acc_Step / / PP - 1) * PP *VPP) / / PP-th model block is subjected to aggregate communication. Here, the VP-(counter-rank-(Acc_Step / / PP - 1) * PP *VPP) / / PP-th model block is the target model block at the target sorting position.

[0075] (2) Identifying the timing of aggregate communication based on the second type default value

[0076] In an embodiment of the present disclosure, it is determined to set the target threshold to a predetermined value when the default value is determined based on the number of backward gradient calculations, the number of model stages, and the number of model blocks included in each model stage.

[0077] For example, the default value determined based on the number of backward gradient calculations, the number of model stages, and the number of model blocks included in each model stage is -(Acc_Step / / PP - 1) * PP *VP, and the target threshold is set to a predetermined value of 0. Here, Acc_Step represents the number of backward gradient calculations, PP represents the number of divided model stages, and VP represents the number of model blocks included in each model stage.

[0078] In the embodiment of the present disclosure, the default value of the number of backward gradient calculations is determined according to the number of backward gradient calculations, the number of model stages, and the number of model blocks included in each model stage, so as to better meet the limitations of the model training device, so as to make the model training process more stable and improve the model performance. In addition, the target threshold is set to a predetermined value, so as to reduce the complexity of parameter adjustment and improve the efficiency of model training.

[0079] In the embodiment of the present disclosure, a counter is provided for each model training device, and the initial value is counter=-(Acc_Step / / PP - 1) * PP * VP. 1F1B scheduling of VPP is executed, a reverse calculation is executed once, and the count by counter is incremented by 1, so that counter=counter+1.

[0080] If the counter of the rank-th computing device satisfies the following condition, the VP-(counter-rank) / / PP-th model block is determined as the target model block at the target sorting position, and communication is performed with the target model block in parallel with the computation flow.

[0081] Condition a), counter is greater than 0.

[0082] Condition b): Counter-rank is divisible by Acc_Step.

[0083] From the above, it can be understood that in both method (1) and method (2), condition b determines the difference between the cumulative count and the model stage number of the current model training device and the target threshold. If the difference is divisible by the target number, it is determined that the backward gradient calculation for an integer number of batches is completed. In the embodiment of the present disclosure, by combining the difference between the cumulative count and the model stage number of the current model training device with the target threshold, the communication timing of the target model block can be flexibly controlled according to the model stage number of the current model training device, and by determining that the backward gradient calculation for an integer number of batches is completed, the process of model training can be better controlled, the use of memory space can be saved, and the efficiency of model training can be improved.

[0084] In addition, in both method (1) and method (2), the target model block at the target sorting position is determined according to the cumulative count, the model stage number, the target threshold, the number of divided model stages, and the number of model blocks included in each model stage. In the embodiment of the present disclosure, the target model block is dynamically determined, so that flexible model adjustment can be performed according to different sorting goals, and the needs of different sorting tasks can be met; the target model block is determined as the target model block at the target sorting position, and the backward gradient calculation and parameter update are performed only for the target model block, so that unnecessary calculation and memory overhead can be reduced, and the efficiency of model training can be improved.

[0085] For example, when PP is 4, VP is 4, and Acc_Step is 8, the default value is -16 and the target threshold is 0. In the fourth model block 12 included in the model stage 0 in FIG. 4, a forward process is performed on the data represented by a rectangular box filled with an arrow in dot shading, and a backward process is performed on a rectangular box filled with a water drop shape in gray shading. When the target model completes the calculation of all backward gradients of the fourth model block included in the model stage 0, the backward gradient accumulation count is 4. When the backpropagation process of one mini batch of all model blocks included in the model stage 0 is completed, the backward gradient accumulation count reaches 16 times and counter=0. When the count result by the counter reaches the target threshold and the difference between the model stage number of the model training device and the target threshold can be divided by the target number Acc_Step, the next calculation stage is the communication timing of the target model block.

[0086] Regardless of whether the above-mentioned method 1 or method 2 is used, in the embodiments of the present disclosure, the method for performing aggregate communication may be ALL Reduce (global reduction), Broadcast, Scatter, Gather, etc., and the present disclosure is not limited thereto.

[0087] When all the model blocks included in the model stage of the current model training device have completed the aggregate communication, the parameters of the model stage are updated based on the optimizer.

[0088] As shown in Fig. 4, when all four model blocks included in the model stage of each model training device have completed aggregate communication, each optimizer S updates the parameters for the corresponding model stage, and the parameter updates are performed based on the result of backward gradient calculation. Here, the optimization unit can be a momentum optimizer, an adaptive learning rate optimizer, a regularization optimizer, etc., but is not limited thereto in the present disclosure.

[0089] In the embodiments of the present disclosure, the model performance can be improved by using an optimizer to update the model-stage parameters, and the parameter update method and speed can be adjusted to allow the model to better adapt to the training data and optimization goals.

[0090] In an embodiment of the present disclosure, the target model performs at least one of the following tasks:

[0091] The system performs at least one task of natural language processing, visual information processing, multimodal information processing, or protein structure prediction.

[0092] In some embodiments, the target model may perform natural language processing. First, the target model receives a text to be processed, which may be a query, a command, or a statement. The target model converts the preprocessed text to be processed into a machine understandable representation, typically converting the text to be processed into a text vector representation. The model performs corresponding actions according to specific task requirements, for example, performing a question and answer task or a translation task based on the text vector. After completing the task execution, the model generates a corresponding output. In the question and answer task, the output is the answer. In the translation task, the output is the translation result.

[0093] In some other embodiments, the target model may perform visual information processing. First, the target model receives visual input, such as an image or a video. For an image, the target model converts the image into a computer-understandable format, such as a pixel matrix. For a video, the target model segments the video into a series of image frames. The model performs a corresponding action according to the specific task requirements. For example, the target model may perform a classification or regression task on the image. After completing the task execution, the model generates a corresponding output. For a classification task, the output of the target model may be a classification label. For a regression task, the output of the target model may be a target detection frame. The target model may output, for example, an image segmentation result, i.e., accomplish other viewpoint information processing tasks, including classification and regression tasks.

[0094] In some other embodiments, the target model may perform multimodal information processing. Multimodal information processing involves the combination of multiple input modalities and multiple feature extraction methods. First, the target model receives multimodal inputs, such as images and text, images and audio, etc. The target model extracts features for each input modality, and once the features of each modality input are extracted, the target model fuses them and integrates the information of different modalities. According to the specific task needs, the target model performs corresponding operations, such as multimodal sentiment analysis tasks, prediction of input sentiment categories, multimodal question-answering tasks, etc., to answer questions related to the input. After the task execution is completed, the model generates corresponding outputs, and for classification tasks, the output may be classification labels, for question-answering tasks, the output may be answers, for recommendation tasks, the output may be recommendation results, etc.

[0095] In some other embodiments, the target model may perform protein structure prediction, where the target model first receives the amino acid sequence of the protein as input, predicts the interactions and spatial relationships between amino acids, and based on the result of the sequence prediction, generates an initial structure of the protein using a structural modeling technique, and the target model can then use an optimization algorithm to adjust the conformation of the original structure to minimize energy and satisfy physical constraints.

[0096] It should be noted that the target model of the present disclosure is not limited to the above-mentioned tasks and application scenarios, but may be extended to other single-modal or multi-modal tasks, and the specifics can be determined according to the actual application scenario.

[0097] In the embodiments of the present disclosure, by performing multiple tasks, the model can learn relevant knowledge and associations between different tasks, and such knowledge sharing can promote mutual understanding and interaction between different tasks and contribute to improving the overall performance and understanding of the model.

[0098] According to the same technical idea, an embodiment of the present disclosure also provides a performance optimization device 500 for a model training device, as shown in FIG. 5, the device includes: A determination module 501 for determining a communication timing of a current model training device for a target model block at a target sorting position, so as to facilitate synchronous aggregate communication between other model training devices in a plurality of model training devices for the model block at the target sorting position, where the current model training device is an arbitrary training device among the plurality of model training devices, the plurality of model training devices are used to train a same target model, the target model is divided into a plurality of model stages, each model stage includes a plurality of ordered model blocks, and in the process of training the target model using a distributed parallel method, a bubble occurs due to an increase in the calculation time of the model training device caused by a communication operation; and an execution module 502 for performing aggregate communication for the backward gradient of the target model block at the communication timing.

[0099] In some embodiments, the decision module comprises: A first determination submodule is provided for determining that the next calculation step is a communication timing for the target model block when the target constraint condition is satisfied; The goal constraint is the number of backward gradient calculations for the target model block by the current model training device is equal to or greater than the target number; The communication timing includes completing calculation of a backward gradient of a model block at a target sorting position in a first model stage of the plurality of model stages.

[0100] In some embodiments, the decision module comprises: determining an accumulation of backward gradients for a target model block to be synchronized; and determining that the target model block satisfies the target constraint if the cumulative amount is greater than or equal to the target number of times and, in a first model stage of the multiple model stages, the model block at the target sorting position is capable of starting aggregate communication.

[0101] In some embodiments, the decision module comprises: A cumulative count is performed on the number of times the current model training device has performed a backward gradient calculation based on a default value. determining that the target model block at the target sort position satisfies the target constraint if the cumulative count is greater than or equal to the target threshold and determining that an integer number of batches of backward gradients have been calculated based on the cumulative count; determining a target threshold value based on a target number of calculations of a backward gradient for the target model block by the current model training device, the target number being equal to or greater than the target number; and a third determining sub-module for determining whether the target constraint is satisfied based on a method including:

[0102] In some embodiments, the decision unit comprises: and a fourth determining sub-module for determining a target threshold value based on the number of backward gradient calculations, the number of model stages, and the number of model blocks included in each model stage when the default value is NULL.

[0103] In some embodiments, the decision module comprises: The method further includes a fifth determination submodule for determining to set the target threshold to a predetermined value when the default value is determined based on the number of backward gradient calculations, the number of model stages, and the number of model blocks included in each model stage.

[0104] In some embodiments, the third determination submodule is specifically: determining a difference between the cumulative count and the model stage number and the target threshold of the current model training device; and determining that an integer number of batches of backward gradients have been computed if the difference is divisible by the target number.

[0105] In some embodiments, the third determination submodule is specifically: It is used to determine a target model block at a target sorting position based on the cumulative count, the model stage number, the target threshold, the number of divided model stages, and the number of model blocks contained in each model stage.

[0106] In some embodiments, the decision module comprises: A sixth determination sub-module is provided for determining that the next calculation stage is a communication timing for the target model block in response to receiving a synchronization message, where the synchronization message is issued by the model training device of the first model stage of the multiple model stages to the model block at the target sorting position.

[0107] In some embodiments, a performance optimization unit for the model training device comprises: The model training device further includes an update module for updating parameters for the model stage based on the optimizer when all of the model blocks included in the model stage of the current model training device have completed the aggregation communication.

[0108] In some embodiments, the target model performs at least one task of natural language processing, visual information processing, multimodal information processing, or protein structure prediction.

[0109] In some embodiments, the target model is trained using a data parallel approach combined with a pipeline parallel approach of a 1F1B interleaved scheduling scheme, and according to the pipeline parallel approach, the target model is divided into a plurality of model stages, and each model stage is assigned to a corresponding model training device to perform training; In a data parallel approach, each batch of training data is divided into multiple micro-batches.

[0110] In the technical solution disclosed herein, the acquisition, storage, and application of users' personal information comply with the provisions of relevant laws and regulations and do not violate public order and morals.

[0111] The specific functions and exemplary descriptions of each module and sub-module of the apparatus according to the embodiments of the present disclosure may refer to the relevant descriptions of the corresponding steps in the above-mentioned method embodiments, and will not be repeated here.

[0112] According to an embodiment of the present disclosure, the present disclosure further provides an electronic device, a non-transitory computer-readable storage medium, and a program product.

[0113] 6 is a block diagram of an electronic device 600 for implementing an embodiment of the present disclosure. The electronic device refers to any type of digital computer, such as a laptop computer, a desktop computer, a workstation, a personal digital assistant, a server, a blade server, a mainframe computer, and other suitable computers. The electronic device also refers to any type of mobile device, such as a personal digital assistant, a cellular phone, an intelligent phone, a wearable device, and other similar computing devices. The components, their connections, and functions described in this disclosure are merely exemplary and are not intended to limit the implementation of what is described and specified in this disclosure.

[0114] 6, device 600 includes a computing unit 601 that can perform various appropriate operations and processes based on computer program instructions stored in a read-only memory (ROM) 602 or loaded from a storage unit 608 into a random access memory (RAM) 603. The RAM 603 can further store various programs and data required for the operation of device 600. The computing unit 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0115] The components in the device 600 are connected to an I / O interface 605, which includes input units 606 such as a keyboard and a mouse, output units 607 such as various displays and speakers, storage units 608 such as a magnetic disk and an optical disk, and a communication unit 609 such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 609 allows the device 600 to exchange information / data with other devices via a computer network such as the Internet and / or various carrier networks.

[0116] The computing unit 601 may be various general-purpose and / or dedicated processing components having processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, computing units that execute various machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 601 executes each of the methods and processes described above, such as the thread stripping method. For example, in some embodiments, the thread stripping method can be realized as a computer software program tangibly included in a machine-readable medium such as the storage unit 608. In some embodiments, some or all of the computer program can be loaded and / or installed in the device 600 via the ROM 602 and / or the communication unit 609. When the computer program is loaded into the RAM 603 and executed by the computing unit 601, one or more steps of the thread stripping method described above can be performed. Additionally, in other embodiments, the computing unit 601 may be configured to perform the thread stripping method in any other suitable manner (eg, firmware).

[0117] Various embodiments of the systems or techniques described in this disclosure may be implemented as digital electronic circuitry systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. Each of these embodiments may include implementation by one or more computer programs executed and / or interpreted by a programmable system including at least one programmable processor, which may be a special purpose or general purpose programmable processor capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transferring data and instructions to the storage system, the at least one input device, and the at least one output device.

[0118] The program code for carrying out the methods of the present disclosure can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general purpose computer, special purpose computer or other programming data processing apparatus such that when the program code is executed by the processor or controller, it can perform the functions / operations specified in the flowcharts and / or block diagrams. The program code can be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a separate software package, or entirely on a remote machine or server.

[0119] In the description of this disclosure, a machine-readable medium may be a tangible medium, which contains or stores a program for use by or in conjunction with an instruction execution system, device or apparatus. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. Further examples of machine-readable storage media include one or more wired electrical connections, portable computer disk cartridges, hard disks, random access memories (RAMs), read-only memories (ROMs), erasable programmable read-only memories (EPROMs or flash memories), optical fibers, portable compact disk read-only memories (CD-ROMs), optical storage devices, magnetic storage devices, or any combination of the foregoing.

[0120] To provide for interaction with a user, the systems and techniques described herein can be implemented on a computer that can include a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, etc.) for displaying information to a user, and a keyboard and pointing device (e.g., a mouse or trackball, etc.) for the user to provide input to the computer. Other types of devices can also be used to provide for interaction with a user, for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, haptic feedback, etc.) and input from the user can be received in any form (e.g., acoustic input, voice input, tactile input, etc.).

[0121] The systems and techniques described herein may be implemented in a computing system that includes a background component (e.g., as a data server), or a middleware component (e.g., an application server), or a front-end component (e.g., a user computer having a graphical user interface or a network browser through which a user can interact with embodiments of the systems and techniques described herein), or any combination of such background, middleware, or front-end components. The components of the system may be connected to each other via any form or medium of digital data communication (e.g., a communications network). Examples of communications networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0122] The computer system may include a client and a server. Typically, the client and the server are remote from each other and generally interact with each other via a communication network. The client-server relationship is created by a computer program that operates on a corresponding computer and has a client-server relationship. The server may be a cloud server, or a server of a distributed system, or a server incorporating a blockchain, etc.

[0123] It should be understood that steps can be reordered, added, or removed using the various aspects of the flow shown above. For example, each step described in this disclosure can be performed in parallel, sequentially, or in a different order. As long as the technical solutions disclosed in this disclosure can achieve the desired results, this disclosure is not limited thereto.

[0124] The above specific examples do not constitute limitations on the scope of protection of the present disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions are possible depending on design matters and other factors. Any modifications, equivalent replacements, improvements, etc. within the spirit and principles of the present disclosure should be included in the scope of protection of the present disclosure.

Claims

1. 1. A method for optimizing performance of a model training device, comprising: Determining a communication timing of a current model training device for a target model block at a target sorting position, so as to facilitate synchronous aggregate communication with other model training devices in a plurality of model training devices for the model block at the target sorting position, wherein the current model training device is an arbitrary training device among the plurality of model training devices, the plurality of model training devices are used to train a same target model, the target model is divided into a plurality of model stages, each model stage includes a plurality of ordered model blocks, and in the process of training the target model using a distributed parallel method, bubbles are generated due to an increase in the calculation time length of the model training device caused by communication operations; and performing aggregate communication on the gradient of the backward direction of the target model block at the communication timing. A method for optimizing the performance of a model training device.

2. Determining a communication timing for a target model block at the target sorting position of the current model training device includes: determining that the next calculation step is a communication timing for the target model block if the target constraint is satisfied; The target constraint is: the number of backward gradient calculations for the target model block by the current model training device is equal to or greater than a target number; At the communication timing, a backward gradient calculation of the model block at the target sorting position is completed in a first model stage of the plurality of model stages. A method for optimizing performance of a model training device according to claim 1.

3. Determining that the target constraint is satisfied includes: determining an accumulation of backward gradients to be synchronized for the target model block; determining that the target model block satisfies the target constraint if the cumulative amount is equal to or greater than the target number of times and if, in a first model stage of the plurality of model stages, a model block at the target sorting position is capable of initiating aggregate communication; A method for optimizing performance of a model training device according to claim 2.

4. Determining that the target constraint is satisfied includes: performing a cumulative count of the number of backward gradient calculations by the current model training device based on a default value; determining that the target model block at the target sorting position satisfies the target constraint if the cumulative count is greater than or equal to a target threshold and if determining that an integer number of batches of backward gradients have been computed based on the cumulative count; determining the target threshold based on a target number of calculations of a backward gradient for the target model block by the current model training device, the target number being equal to or greater than the target number; A method for optimizing performance of a model training device according to claim 2.

5. The method for optimizing the performance of the model training device includes: and if the default value is NULL, determining the target threshold based on a number of backward gradient calculations, the number of model stages, and a number of model blocks included in each model stage. A method for optimizing performance of a model training device according to claim 4.

6. The method for optimizing the performance of the model training device includes: determining, when the default value is determined based on a number of backward gradient calculations, the number of model stages, and a number of model blocks included in each model stage, to set the target threshold to a predetermined value. A method for optimizing performance of a model training device according to claim 4.

7. determining that an integer number of batches of backward gradients have been computed based on the cumulative counts, determining a difference between the cumulative count and the current model training device's model stage number and the target threshold; determining that an integer number of batches of backward gradients have been computed if the difference is divisible by the target number. A method for optimizing performance of a model training device according to claim 4.

8. Determining the target model block at the target sorting position comprises: determining a target model block at the target sorting position based on the cumulative count, a model stage number, the target threshold, the number of divided model stages, and the number of model blocks included in each model stage; A method for optimizing performance of a model training device according to claim 4.

9. Determining a communication timing for a target model block at the target sorting position of a current model training device includes: determining that the next calculation step is a communication timing for the target model block in response to receiving a synchronization message, where the synchronization message is issued by a model training device of a first model stage of the plurality of model stages to a model block at the target sorting position; A method for optimizing performance of a model training device according to claim 1.

10. The method for optimizing the performance of the model training device includes: Further comprising: when each model block included in the model stage of the current model training device has completed the aggregate communication, updating parameters for the model stage based on an optimizer; A method for optimizing performance of a model training device according to claim 1.

11. The target model performs at least one task of natural language processing, visual information processing, multimodal information processing, or protein structure prediction. A method for optimizing performance of a model training device according to claim 1.

12. The target model is trained using a data parallel technique combined with a pipeline parallel technique of 1F1B interleaved scheduling, and according to the pipeline parallel technique, the target model is divided into the plurality of model stages, and each model stage is assigned to a corresponding model training device to perform training; In the data parallel approach, each batch of training data is divided into multiple micro-batches. A method for optimizing performance of a model training device according to claim 1.

13. 1. An apparatus for optimizing performance of a model training device, comprising: A determination module for determining a communication timing of a current model training device for a target model block at a target sorting position, so as to facilitate synchronous aggregate communication between other model training devices in a plurality of model training devices for the model block at the target sorting position, wherein the current model training device is any training device among the plurality of model training devices, the plurality of model training devices are used to train a same target model, the target model is divided into a plurality of model stages, each model stage includes a plurality of ordered model blocks, and in the process of training the target model using a distributed parallel method, a bubble occurs due to an increase in the calculation time of the model training device caused by a communication operation; and an execution module for performing aggregate communication on the backward gradient of the target model block at the communication timing. Model training device performance optimization device.

14. The decision module: a first determination submodule for determining that the next calculation step is a communication timing for the target model block if a target constraint condition is satisfied; The target constraint is: the number of backward gradient calculations for the target model block by the current model training device is equal to or greater than a target number; At the communication timing, a backward gradient calculation of the model block at the target sorting position is completed in a first model stage of the plurality of model stages.

14. Apparatus for optimizing performance of a model training device according to claim 13.

15. The decision module: determining an accumulation of backward gradients to be synchronized for the target model block; and determining that the target model block satisfies the target constraint condition if the cumulative amount is equal to or greater than the target number of times and if, in a first model stage of the plurality of model stages, a model block at the target sorting position can start aggregate communication.

15. Apparatus for optimizing performance of a model training device according to claim 14.

16. The decision module: performing a cumulative count of the number of backward gradient calculations by the current model training device based on a default value; determining that the target model block at the target sorting position satisfies the target constraint if the cumulative count is greater than or equal to a target threshold and if determining that an integer number of batches of backward gradients have been computed based on the cumulative count; determining the target threshold based on a target number of backward gradient calculations for the target model block by the current model training device, the target number being equal to or greater than the target number; and a third determining submodule for determining whether the target constraint is satisfied based on a method including:

15. Apparatus for optimizing performance of a model training device according to claim 14.

17. The determination unit is a fourth determination submodule for determining the target threshold value based on a number of backward gradient calculations, the number of model stages, and a number of model blocks included in each model stage when the default value is NULL; 17. Apparatus for optimizing performance of a model training device according to claim 16.

18. The decision module: a fifth determination submodule for determining to set the target threshold to a predetermined value when the default value is determined based on the number of backward gradient calculations, the number of model stages, and the number of model blocks included in each model stage; 17. Apparatus for optimizing performance of a model training device according to claim 16.

19. The third determination submodule specifically includes: determining a difference between the cumulative count and the current model training device's model stage number and the target threshold; determining that an integer number of batches of backward gradients have been computed if the difference is divisible by the target number.

17. Apparatus for optimizing performance of a model training device according to claim 16.

20. The third determination submodule specifically includes: used for determining a target model block at the target sorting position based on the cumulative count, the model stage number, the target threshold, the number of divided model stages, and the number of model blocks included in each model stage; 17. Apparatus for optimizing performance of a model training device according to claim 16.

21. The decision module: a sixth determination submodule for determining that the next calculation stage is a communication timing for the target model block in response to receiving a synchronization message, where the synchronization message is issued by a model training device of a first model stage of the plurality of model stages to a model block at the target sorting position; 14. Apparatus for optimizing performance of a model training device according to claim 13.

22. The performance optimization apparatus for the model training device includes: Further, an update module is provided for updating parameters of the model stage based on an optimizer when all the model blocks included in the model stage of the current model training device have completed the aggregate communication. Apparatus for optimizing the performance of a model training device according to any one of claims 13 to 21.

23. The target model performs at least one task of natural language processing, visual information processing, multimodal information processing, or protein structure prediction. Apparatus for optimizing the performance of a model training device according to any one of claims 13 to 21.

24. The target model is trained using a data parallel technique combined with a pipeline parallel technique of 1F1B interleaved scheduling, and according to the pipeline parallel technique, the target model is divided into the plurality of model stages, and each model stage is assigned to a corresponding model training device to perform training; In the data parallel approach, each batch of training data is divided into multiple micro-batches. Apparatus for optimizing the performance of a model training device according to any one of claims 13 to 21.

25. At least one processor; a memory in communication with the at least one processor; An electronic device, wherein the memory stores instructions executable by the at least one processor, the instructions, when executed by the at least one processor, causing the at least one processor to perform a control method according to any one of claims 1 to 12.

26. A non-transitory computer-readable storage medium storing computer instructions for causing a computer to carry out the control method according to any one of claims 1 to 12.

27. A program for implementing the control method according to any one of claims 1 to 12 when executed by a processor in a computer.

Images