Performance optimization method, device and equipment for model training equipment

Abstract

The present disclosure provides a performance optimization method, apparatus and equipment for a model training device, which relates to the field of artificial intelligence technology, and in particular to technical fields such as deep learning, large model training, and distributed parallel strategies. The specific implementation scheme is: determining the communication timing of the current model training device for the target model block at the target sorting position, so that it can synchronize and aggregate communication with other model training devices in multiple model training devices for the model block at the target sorting position; at the communication timing, aggregate communication is performed on the reverse gradient of the target model block. In the embodiment of the present disclosure, by making the communication timing of multiple model training devices basically consistent, some bubbles are eliminated as much as possible to improve the performance of the device, and then the parallel computing capabilities of the model training device can be maximized.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, and in particular to technical fields such as deep learning, large model training, and distributed parallel strategies. Background Art

[0002] Large models are machine learning models with numerous parameters and complex structures. Large models can process massive amounts of data, thereby improving the accuracy and generalization capabilities of machine learning models. Large models possess greater complexity and flexibility, enabling them to handle more complex problems. With more parameters and a more complex structure, large models can more accurately represent data distributions and learn more complex features, thereby improving model accuracy and performance.

[0003] Large models have a wider range of application scenarios and higher performance. However, large models need to process a large amount of data and parameters, and the training and inference time is longer, which requires more computing resources. Summary of the Invention

[0004] The present disclosure provides a performance optimization method, apparatus, and device for a model training device.

[0005] According to one aspect of the present disclosure, a performance optimization method for a model training device is provided, comprising:

[0006] Determine the communication timing of the current model training device for the target model block at the target sorting position, so as to facilitate synchronous modulation and aggregation communication with other model training devices among the multiple model training devices for the model block at the target sorting position; wherein the current model training device is any training device among the multiple model training devices; the multiple model training devices are used to train the same target model; the target model is divided into multiple model stages, each model stage includes multiple model blocks arranged in sequence; in the process of training the target model using a distributed parallel strategy, bubbles are generated due to the increase in the calculation time of the model training device due to communication operations;

[0007] At the communication opportunity, aggregate communication is performed on the backward gradients of the target model block.

[0008] According to another aspect of the present disclosure, a performance optimization apparatus for a model training device is provided, comprising:

[0009] A determination module is used to determine the communication timing of the current model training device for the target model block at the target sorting position, so as to facilitate synchronous modulation and aggregation communication with other model training devices in multiple model training devices for the model block at the target sorting position; wherein the current model training device is any training device in the multiple model training devices; the multiple model training devices are used to train the same target model; the target model is divided into multiple model stages, each model stage includes multiple model blocks arranged in sequence; bubbles are generated in the process of training the target model using a distributed parallel strategy because the communication operation increases the calculation time of the model training device;

[0010] The execution module is used to perform aggregate communication on the reverse gradient of the target model block at the communication opportunity.

[0011] According to another aspect of the present disclosure, there is provided an electronic device, comprising:

[0012] at least one processor; and

[0013] a memory communicatively connected to the at least one processor; wherein,

[0014] The memory stores instructions that can be executed by the at least one processor. The instructions are executed by the at least one processor to enable the at least one processor to perform the method of any embodiment of the present disclosure.

[0015] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute the method according to any embodiment of the present disclosure.

[0016] According to another aspect of the present disclosure, a computer program product is provided, comprising a computer program, which implements the method according to any embodiment of the present disclosure when executed by a processor.

[0017] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The accompanying drawings are provided to facilitate a better understanding of the present invention and do not constitute a limitation of the present disclosure.

[0019] Figure 1 is a flowchart of a performance optimization method for a model training device according to an embodiment of the present disclosure;

[0020] Figure 2 is a timing diagram of a model training process provided according to an embodiment of the present disclosure;

[0021] Figure 3 is a timing diagram of another model training process provided according to another embodiment of the present disclosure;

[0022] Figure 4 is a timing diagram of another model training process provided according to another embodiment of the present disclosure;

[0023] Figure 5 is a schematic structural diagram of a performance optimization device for a model training device according to an embodiment of the present disclosure;

[0024] Figure 6 It is a block diagram of an electronic device used to implement the performance optimization method of the model training device of an embodiment of the present disclosure. DETAILED DESCRIPTION

[0025] The following description of exemplary embodiments of the present disclosure is made in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be appreciated by those skilled in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0026] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features being referred to. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of such features. Throughout the present disclosure, "plurality" means two or more, unless otherwise specifically defined.

[0027] With the advent of the big data era, people are likely to generate vast amounts of data in their daily lives. To process this massive amount of data, solve various complex problems, and meet user demands for high efficiency in deep learning tasks, a complex training process is often required to obtain a high-quality model. To effectively train models, distributed parallel training methods are often used to accelerate training. Related technologies typically employ data parallelism and pipeline parallelism. The following describes each of these approaches.

[0028] The principle of data parallelism is that the data set is divided into multiple parts, each model training device saves the complete training parameters and processes a sub-data set to accelerate the model training process.

[0029] Pipeline parallelism is a method of model parallelism. Its principle is: by dividing the model into different model training devices, the video memory consumption of a single model training device is reduced, and data is transmitted between adjacent devices through communication links to achieve ultra-large-scale model training.

[0030] However, data parallelism requires that the memory of the model training device be able to accommodate the entire model, resulting in poor scalability. Furthermore, when sharding the model using pipeline parallelism, load balancing across the model training device and overall communication time are not considered. Using pipeline parallel training alone fails to fully utilize the model training device, and model training efficiency still needs to be improved.

[0031] Therefore, data parallel strategies and pipeline parallel strategies can be used in related technologies to train large models. However, in this way, in the process of combining reverse gradient communication, the overall bubble rate (bubble rate) still needs to be improved. Bubbles will cause the model training equipment to waste computing resources. In order to improve the model training effect and achieve performance improvement of large-scale model training equipment, this application proposes a performance optimization method for model training equipment. The performance optimization method combines data parallelism and pipeline parallelism to achieve synchronous communication, thereby improving the overlap (overlap rate) of the model training equipment calculation and reducing the overall bubble rate, thereby improving the performance of the model training equipment. The specific implementation process of the performance optimization method of the model training equipment is as follows. Figure 1 Shown, including:

[0032] S101, determining a communication timing of a current model training device for a target model block at a target sorting position, so as to enable synchronous modulation and aggregation communication with other model training devices in a plurality of model training devices for the model block at the target sorting position.

[0033] The current model training device is any training device among the multiple model training devices; the multiple model training devices are used to train the same target model; the target model is divided into multiple model stages, each model stage includes multiple model blocks arranged in sequence; bubbles are generated during the process of training the target model using a distributed parallel strategy due to increased computing time of the model training device caused by communication operations;

[0034] S102 , at a communication opportunity, performing aggregate communication on the reverse gradient of the target model block.

[0035] The target model in the embodiments of this disclosure is a large model. A large model refers to a machine learning model with a large parameter size and complexity. In the field of deep learning, a large model typically refers to a neural network model with millions to billions of parameters. These models require a large amount of computing resources and storage space to train and store, and often require distributed computing and special hardware acceleration technologies.

[0036] Large models are designed and trained to provide more powerful and accurate model performance to handle more complex and larger datasets or tasks. Large models are typically able to learn more subtle patterns and regularities, and have stronger generalization and expressive capabilities.

[0037] In the disclosed embodiments, by aligning the communication timings of multiple model training devices, the parallel computing capabilities of the model training devices can be maximized. By controlling the synchronization of communications between multiple model training devices and properly arranging communication timings, the bubble rate can be reduced, computational overlap can be increased, computation time can be shortened, communication efficiency can be improved, and the performance of the model training devices can be enhanced.

[0038] When using pipeline parallel strategy for model training, the target model is divided into multiple model stages according to the pipeline parallel strategy. For example, Figure 2 The target model is divided into four model stages, namely model stage 0, model stage 1, model stage 2, and model stage 3. Each model stage is deployed to a one-to-one corresponding model training device. Each model stage can be divided into multiple model blocks, and each model block includes one or more neural network layers. In the case of including multiple neural network layers, the multiple neural network layers are continuous. For example, the target model has neural network layers numbered 0-15 in sequence, then the target model has a total of 16 neural network layers. If deployed to 4 model training devices for training, each model training device can deploy 4 neural network layers. The corresponding target model can be divided into 4 model stages, and each model stage is assumed to be divided into two model blocks. Then, Figure 2 As shown, the neural network layers numbered 0, 1, 8, and 9 form the first model stage (i.e. Figure 2 The two model blocks of the model stage 0 in the figure are <0, 1>, <8, 9> respectively; the neural network layers numbered 2, 3, 10, and 11 constitute the second model stage (i.e. Figure 2 The two model blocks of the model stage 1) are <2, 3>, <10, 11>; the neural network layers numbered 4, 5, 12, and 13 are constructed into the third model stage (i.e. Figure 2 The two model blocks of the model stage 2 in the figure are <4, 5> and <12, 13> respectively; the neural network layers numbered 6, 7, 14, and 15 constitute the fourth model stage (i.e. Figure 2 The two model blocks in the model stage 3) are <6, 7> and <14, 15> respectively.

[0039] Continue in Figure 2The following explanation is based on the model stages shown. During training, forward calculations are performed on the first model block of the first model stage. After the forward calculations for the first model block of the first model stage are completed, the forward calculations for the first model block of the second model stage are performed, and this process is passed down in sequence until the forward calculations for the first model block of the fourth model stage are completed. Forward calculations are then performed on the second model block of the first model stage, and the results are passed down. After the forward calculations for the second model block of the fourth model stage are completed, forward and backward calculations are performed alternately. The results of the backward calculations are then passed up in sequence. Backpropagation of the 16-layer neural network is completed until all backward calculations for the first model block of the first model stage are completed.

[0040] In the embodiment of the present disclosure, the target model is trained using a data parallel strategy combined with a pipeline parallel strategy of 1F1B interleaved scheduling, and the target model is divided into multiple model stages according to the pipeline parallel strategy, and each model stage is assigned to a corresponding model training device for training; in the data parallel strategy, each batch (minibatch) of training data is divided into multiple micro batches (micro batch). Figure 2 As shown in the figure, the training data is divided into multiple minibatches, and each mini batch is divided into 4 micro batches. Figure 2 In the figure, identically numbered microbatches represent the same microbatch. A represents the forward computation of the first model block in each model stage, and B represents the forward computation of the second model block in each model stage. b represents the reverse gradient computation of the second model block in each model stage. a represents the reverse gradient computation of the first model block in each model stage. Each microbatch is processed sequentially by each model block, equivalent to completing the forward processing operation in the order of the target model's neural network layers. During the reverse process, the model blocks are updated in the opposite order to the forward process.

[0041] For example, continue with Figure 2 As an example, Figure 2 In the timing diagram of the forward process, the forward calculation of the first model block of each model stage is represented by a gray-filled rectangular box, and the forward calculation of the second model block of each model stage is represented by a white-filled rectangular box. Figure 2In the reverse process, the reverse calculation of the second model block of each model stage is represented by a black-filled rectangular box; the reverse calculation of the first model block of each model stage is represented by a dotted-background-filled rectangular box (this type of rectangular box all contains the character a). The forward process of the micro batch is that it is first processed by the first model block <0, 1> of model stage 0, and then passed to the first model block <2, 3> of model stage 1, and then the processing result is passed to the first model block <4, 5> of model stage 2, and then the processing result is passed to the first model block <6, 7> of model stage 3, and then the processing result is passed to the second model block <8, 9> of model stage 0, and then the processing result is passed to the second model block <10, 11> of model stage 1, and then the processing result is passed to the second model block <12, 13> of model stage 2, and finally the processing result is passed to the second model block <14, 15> of model stage 3. In this way, the forward transmission process of the entire 16-layer neural network is completed through the cooperation of different model stages and different models. The processing order of the reverse process is exactly the opposite of the forward process. For example, refer to Figure 2 The black rectangle containing the character b first performs the reverse calculation of the second model block of model stage 3, then performs the reverse calculation of the second model block of model stage 2, then performs the reverse calculation of the second model block of model stage 1, and then performs the reverse calculation of the second model block of model stage 0. After the reverse calculation of the second model block is completed, refer to Figure 2 For the dotted shading rectangle containing the character a, the reverse calculation of model stage 3, model stage 2, model stage 1, and the first model block of model stage 0 are performed in sequence.

[0042] In the disclosed embodiments, a 1F1B interleaved scheduling approach is used, allowing each model training device to alternately execute the forward and reverse processes. In 1F1B interleaved scheduling, each device can compute a subset of multiple layers (i.e., a model block) rather than a continuous set of layers.

[0043] The 1F1B mode alternates forward and backward computations, freeing up unnecessary intermediate variables.

[0044] In the 1F1B interleaved scheduling approach, the target model is trained using multiple mini-batches, with each mini-batch divided into multiple micro-batches. Dependencies are established between the computational tasks for multiple micro-batches on the same model training device. For example, after each model training device performs forward computation on the i-th micro-batch, it directly performs reverse computation on the i-th micro-batch's forward computation results. After completing the target number of reverse computations, aggregate communication is performed to update the model parameters based on the gradients obtained from the reverse computations.

[0045] like Figure 2 As shown, the gray-filled rectangle containing character A and the white-filled rectangle containing character B represent the forward computation process. The black-filled rectangle represents the reverse computation process for the white rectangle, and the dot-filled rectangle containing character a represents the reverse computation process for the gray rectangle. Each mini-batch is divided into four microbatches. The reverse computation process is described using the second model block as an example. Figure 2 The second model block of model stage 0 performs reverse gradient calculation for the second model block <14, 15> of model stage 3 after the forward calculation is completed in model stage 3 (e.g. Figure 2 1b) of model stage 3 in the model, the calculation result is passed to the model training device of model stage 2, and the model training device performs reverse gradient calculation for the second model block <12, 13> of model stage 2 (such as Figure 2 1b) in model stage 2, and then pass the processing result to the model training device of model stage 1. The model training device of model stage 1 performs reverse gradient calculation on the second model block <10, 11>, and then passes the processing result to the model training device of model stage 0. The model training device of model stage 0 performs reverse gradient calculation on the second model block <8, 9>. The processing result is then passed to model stage 3 in sequence to complete the reverse gradient calculation for the first model block <6, 7> of model stage 3, and then the processing result is passed to model stage 2 to complete the reverse gradient calculation for the first model block <4, 5> of model stage 2, and then the processing result is passed to model stage 1 to complete the reverse gradient calculation for the first model block <2, 3> of model stage 1, and then the processing result is passed to model stage 0 to complete the reverse gradient calculation for the first model block <0, 1> of model stage 0. In this way, the reverse transfer process of the entire 16-layer neural network is completed through the cooperation of different model stages and different models.

[0046] In the disclosed embodiment, a pipeline parallel strategy using a 1F1B staggered scheduling method is combined with a data parallel strategy for distributed training, and forward calculations and reverse calculations are performed alternately, which can release unnecessary intermediate variables and reduce the consumption of memory resources. In addition, using multiple batch training models, each batch is divided into multiple micro-batches to improve memory efficiency and accelerate model training speed. When the training data set is large, loading the entire batch into the memory at one time may cause insufficient memory. Using multiple batch training models can load and process them one by one, effectively utilizing memory resources. Dividing multiple batches into multiple micro-batches can be processed in parallel, thereby accelerating the training speed.

[0047] In other examples, each model block may include not only multiple neural network layers, but also one neural network layer. For example, Figure 3 In the example, the target model is divided into four model stages, each of which includes four model blocks arranged in sequence. Each model training device performs calculations on four model blocks, and each model block has a neural network layer. Figure 3 As shown, model stage 0 has four neural network layers, namely 0, 4, 8, and 12, which are both model blocks and neural network layers; model stage 1 has four neural network layers, namely 1, 5, 9, and 13, which are both model blocks and neural network layers; model stage 2 has four neural network layers, namely 2, 6, 10, and 14, which are both model blocks and neural network layers; and model stage 3 has four neural network layers, namely 3, 7, 11, and 15, which are both model blocks and neural network layers. Among them, the first model block 0 of model stage 0 is represented by a rectangular box with a five-pointed star embedded in a slash background, which represents the execution of the forward process, and the processing results are sequentially passed down to the first model block 3 of model stage 3. Then the processing results are sequentially passed to the second model block 4 of model stage 0. The second model block is in Figure 3 The rectangular box with a square embedded in vertical stripes represents the execution of the forward process, and the result is passed down in sequence to the second model block 7 of model stage 3. The processing result is then passed to the third model block 8 of model stage 0 in sequence. The third model block uses a rectangular box with a circle embedded in horizontal stripes to represent the data execution forward process, and the result is passed down in sequence to the third model block 11 of model stage 3. The processing result is then passed to the fourth model block 12 of model stage 0 in sequence. The fourth model block uses a rectangular box with an arrow embedded in dotted stripes to represent the execution of the forward process, and the result is passed down in sequence to the fourth model block 15 of model stage 3. After the forward propagation process is completed, the backward propagation process begins. The order of updating the backward propagation model blocks is exactly the opposite.

[0048] With this approach, each model training device in the pipeline is assigned multiple pipeline stages, and each pipeline stage has less computational effort.

[0049] It should be noted that the number of model stages into which the target model can be divided, the number of model blocks contained in each model stage, and the number of neural network layers contained in each model block can be determined according to actual needs, and the embodiments of the present disclosure do not limit this.

[0050] Figure 2 and Figure 3 Both can be understood as timing diagrams of the model training process. Figure 2 and Figure 3 The horizontal axis represents time, and the vertically stacked rectangular boxes represent the types of operations performed at the corresponding time. The operation types are divided into two parts, including calculation and communication. Figure 3 As shown, for each model training device, the part containing a denser rectangular box is the computing part. Figure 3 In the figure, rectangles with a five-pointed star embedded in a diagonal shading, a square embedded in a vertical shading, a circle embedded in a horizontal shading, and an arrow embedded in a dotted shading represent the forward calculation part. Rectangular boxes with a teardrop embedded in a light gray shading, a heart embedded in a black shading, a diamond embedded in a white shading, and a triangle embedded in a dark gray shading represent the reverse calculation part. The part below that, containing a rectangle beginning with the letter A, represents the communication part.

[0051] Figure 2 and Figure 3 Both reflect the staggered scheduling method of 1F1B. However, in the process of scheduling based on the 1F1B mode, each model training device uses asynchronous communication, that is, the reverse gradient communication process of the model blocks in the same sorting position is completed at different times. Asynchronous communication will cause redundant bubbles in the calculation process, increase the calculation time, and thus produce a "cumulative effect", resulting in an increase in the entire end-to-end bubble rate. Figure 3 As shown in the figure, the communication of each model stage (such as A0, B0, C0, D0) is performed at different times. During the backpropagation process, the bubbles between the teardrop-shaped rectangular box embedded in the light gray background and the heart-shaped rectangular box embedded in the black background become larger and larger, resulting in a "cumulative effect", that is, multiple bubbles are accumulated, which prolongs the calculation time and increases the bubble rate of the entire end-to-end process.

[0052] In order to eliminate this "cumulative effect", in the embodiment of the present disclosure, it is necessary to control all model training devices participating in the pipeline calculation to synchronize and perform data parallel gradient communication in the 1F1B staggered scheduling of the parallel pipeline parallel strategy. Figure 4 As shown, each model stage communicates at the same communication time. As the reverse calculation results propagate upward, relative to Figure 3For example, the bubbles between the teardrop-shaped rectangles within the light gray background and the heart-shaped rectangles within the black background are getting smaller and smaller. This significantly reduces the number of bubbles throughout the model training process, which reduces the computational time during backpropagation. This improves model training efficiency while also reducing the bubble rate, thereby improving the performance of the entire model training device.

[0053] In the embodiment of the present disclosure, there are two implementation methods for determining the communication timing of the target model block of the target sorting position of the current model training device, including:

[0054] Method 1: Each model training device can determine the communication timing of the aggregate communication of each model block through the communication method.

[0055] To synchronize aggregated communication and eliminate bubbles, in embodiments of the present disclosure, synchronized communication can be achieved through a communication mechanism between different model training devices. This can be implemented by: in response to receiving a synchronization message, determining the communication timing for the target model block in the next computational stage, where the synchronization message is sent by the model training device in the first model stage of multiple model stages for the model block in the target sorting position.

[0056] It can be understood that the model training device of the first model stage can send the synchronization message after completing the reverse gradient calculation for the model block at the target sorting position, so as to facilitate the reverse communication of multiple model training devices for the model block at the same sorting position. Figure 4 In the process, after the communication timing of model stage 0 is determined, the communication timing of model stages 1, 2, and 3 are automatically moved back by rank (calculation stage) steps to synchronize with the communication timing of model stage 0.

[0057] In the embodiment of the present disclosure, multiple model training stages correspond to the first model stage of a model training device, enabling multiple model training devices to communicate synchronously, improving calculation overlap, reducing bubble rate, and improving the performance of the model training device.

[0058] like Figure 4As shown, in the first model stage, A0 is the part that performs aggregate communication on the reverse gradient of the model block 12 whose sorting position is the 4th, and the time when the aggregate communication starts is the communication opportunity. B0 is the part that performs aggregate communication on the reverse gradient of the model block 8 whose sorting position is the 3rd. C0 is the part that performs aggregate communication on the reverse gradient of the model block 4 whose sorting position is the 2nd. D0 is the part that performs aggregate communication on the reverse gradient of the model block 0 whose sorting position is the 1st. It can be seen that the order of aggregate communication is opposite to the order of forward calculation of the model blocks. Therefore, the same is true for the part that performs aggregate communication on each target model block in model stages 1, 2, and 3. The target model blocks and their aggregate communication parts can be corresponded one by one, and the present disclosure will not go into details here.

[0059] Method 2: Each model training device can independently determine the communication timing of the aggregate communication of each model block.

[0060] The method can be implemented as follows: determining the next calculation phase as the communication timing of the target model block when the target constraint condition is satisfied;

[0061] The target constraints include:

[0062] 1) The number of reverse gradient calculations performed by the current model training device on the target model block is greater than or equal to the target number;

[0063] 2) At the communication timing, the reverse gradient calculation of the model block at the target sorting position in the first model stage among the multiple model stages is completed.

[0064] like Figure 2 As shown in the figure, the first model stage is model stage 0, and the model blocks in the first model stage include two model blocks, <0, 1> and <8, 9>. Figure 2 In model stage 0, gray and white rectangular boxes are used to represent the forward calculation process of model blocks in different sorting positions. The black rectangular box is the back propagation process for the white rectangular box, and the rectangular box with dotted background is the back propagation process for the gray rectangular box. Figure 2 It can be seen that the reverse gradient calculation of the model block at the target sorting position in the first model stage is completed, which can be understood as: taking the second model block as an example, the reverse gradient calculation of the black rectangular box for the same minibatch in model stage 0 is completed.

[0065] In the disclosed embodiments, synchronization is performed based on target constraints. The timing of this synchronization can be determined independently by each model training device. This allows for more precise control of communication timing, ensuring that communication occurs when needed. Precisely controlling communication timing through target constraints can shorten computation time and reduce bubble rates, thereby improving the performance of the model training device and, consequently, the efficiency of model training.

[0066] In order to determine whether the target constraint conditions are met, the following two implementation methods are exemplified in the embodiments of the present disclosure, including:

[0067] Method for determining target constraints 1):

[0068] Determining whether the target constraint condition is satisfied can be implemented as follows: determining the cumulative amount of reverse gradients to be synchronized of the target model block; when the cumulative amount is greater than or equal to the target number of times and it is determined that the model block at the target sorting position in the first model stage of multiple model stages can start aggregate communication, determining that the target model block satisfies the target constraint condition.

[0069] During implementation, the number of reverse calculations can be counted for each model block to determine whether the model block meets the communication requirements.

[0070] Whether the model block at the target ranking position in the first model stage can initiate aggregate communication can be estimated based on the time required for forward and backward computations. This estimation can be achieved using a neural network or by reasoning based on the computational complexity of the corresponding model block. The specific reasoning process is not further elaborated in this disclosure; any reasoning method that can achieve the above solution can be used in this disclosure.

[0071] In the disclosed embodiment, gradient accumulation is a very simple calculation method. Using this method to count the number of reverse calculations can save computing resources, reduce communication overhead, and improve the efficiency of model training.

[0072] Method 2 for determining target constraints):

[0073] Determining that the target constraint is satisfied can be implemented as follows: cumulatively counting the number of reverse gradient calculations of the current model training device based on the default value; when the cumulative count is greater than or equal to the target threshold, and an integer number of batches of reverse gradient calculations are determined to have been completed based on the cumulative count, determining that the target model block at the target sorting position satisfies the target constraint.

[0074] The target threshold is determined based on a target number of times, so that the number of reverse gradient calculations performed by the current model training device on the target model block is greater than or equal to the target number of times.

[0075] In the embodiment of the present disclosure, during the scheduling process, the model training device performs a reverse gradient calculation for each micro batch and performs a gradient accumulation count. Assuming that the target number Acc_Step is 8, aggregate communication is performed when the cumulative number of reverse gradient calculations completed for a model block is greater than or equal to 8. Figure 2In the figure, for the second model block of model stage 0, after executing the reverse gradient calculations shown in 1b, 2b, 3b, and 4b as shown in the black rectangle, the subsequent accumulated reverse gradient calculations are for the first model block. However, for the second model block, the reverse gradient number needs to be accumulated to 8 times before the aggregate communication for the second model block can be performed to ensure that the same model block can complete the Acc_Step number.

[0076] In the disclosed embodiments, the current model training device determines the communication timing of the target model block it contains, unaffected by other model training devices. Each model training device can identify the communication timing of the reverse gradient calculation results of each model block based on constraints, relative to the communication timing between devices. Furthermore, the entire process of cumulative counting and target preset determination is a very simple computational operation, requiring negligible computational resources, thereby improving the efficiency of model training.

[0077] In the disclosed embodiments, different schemes for identifying communication opportunities are provided based on different default values for the cumulative count. The default value settings can be derived through experimental reasoning based on the bubbles required to eliminate synchronous communication. The reasoning process is not detailed here. The following describes the communication opportunities under different default value settings.

[0078] (1) Based on the first default value, identify the communication timing of aggregate communication

[0079] In the embodiment of the present disclosure, when the default value is empty, the target threshold is determined based on the number of reverse gradient calculations, the number of model stages, and the number of model blocks included in each model stage.

[0080] For example, if the default value is set to empty, the target threshold determined based on the number of reverse gradient calculations, the number of model stages, and the number of model blocks contained in each model stage is (Acc_Step / / PP–1)*PP*VP; where Acc_Step represents the number of reverse gradient calculations; PP represents the number of split model stages; VP represents the number of model blocks contained in each model stage; and / / represents rounding down.

[0081] In the disclosed embodiment, setting the default value of the number of reverse gradient calculations for the current model training device to null can reduce memory usage. Furthermore, determining a target threshold based on the number of reverse gradient calculations, the number of model stages, and the number of model blocks contained in each model stage allows the model training device to determine communication opportunities in a timely manner, reducing the bubble rate and improving model training efficiency.

[0082] In some embodiments, determining that an integer number of batches of reverse gradient calculations have been completed based on the cumulative count can be implemented as follows: determining the difference between the cumulative count and the model stage number and target threshold of the current model training device; and determining that an integer number of batches of reverse gradient calculations have been completed when the difference is divisible by the target number of times.

[0083] For example, each model training device sets a counter counter, and when the default value is empty, counter = 0.

[0084] In the embodiment of the present disclosure, the model training device completes the training of the target model based on the 1F1B interleaved scheduling method. 1F1B interleaved scheduling of VP is executed, a reverse calculation is performed, and the counter count is increased by 1, then counter=counter+1.

[0085] When the counter of the rank-th computing device (i.e., the rank-th model stage) meets the following conditions, the model block at VP-(counter-rank-(Acc_Step / / PP–1)*PP*VP) / / PP is determined as the target model block at the target ranking position, and communication with this target model block is performed concurrently with the computing flow. Rank is the model stage number of the current model training device.

[0086] Condition a), counter is greater than (Acc_Step / / PP–1)*PP*VP

[0087] Condition b), counter-rank–(Acc_Step / / PP–1)*PP*VP is divisible by Acc_Step.

[0088] Condition b is the difference between the cumulative count and the model stage number and target threshold of the current model training device. If the difference is divisible by the target number, it is determined that an integer number of batches of reverse gradient calculations have been completed.

[0089] In the embodiment of the present disclosure, the difference between the cumulative count and the model stage number of the current model training device is combined with the target threshold, and the communication timing of the target model block can be flexibly controlled according to the model stage number of the current model training device. By determining the completion of an integer number of batches of reverse gradient calculations, the model training process can be better controlled, the use of memory space can be saved, and the efficiency of model training can be improved.

[0090] For example, Figure 4As shown, PP is 4, VP is 4, and Acc_Step is set to 8. This means the target model is divided into four model stages, each mini batch is divided into four micro batches, and the reverse gradient accumulation count is 8. When the default value is left empty, counter = 0, and the target threshold is 16. Taking model stage 0 as an example, each micro batch performs a reverse calculation on each of the four model training devices. After each micro batch reverse calculation, a reverse gradient accumulation count is performed, and the counter count is incremented by 1 until the reverse gradient calculation counter for the target model block reaches 16. At this point, the value of counter-rank – (Acc_Step / / PP–1)*PP*VP is 0, which is divisible by Acc_Step. When the first reverse calculation is performed on the next model block's micro batch, the counter count is greater than 16. This indicates that this is the communication opportunity for the target model block, so the model block at VP-(counter-rank – (Acc_Step / / PP–1)*PP*VPP) / / PP performs aggregate communication. Among them, the model block of VP-(counter-rank-(Acc_Step / / PP–1)*PP*VPP) / / PP is the target model block of the target ranking position.

[0091] (2) Based on the second default value, identify the communication timing of aggregate communication

[0092] In an embodiment of the present disclosure, when a default value is determined based on the number of reverse gradient calculations, the number of model stages, and the number of model blocks included in each model stage, the target threshold is determined to be set to a specified value.

[0093] For example, the default value determined based on the number of reverse gradient calculations, the number of model stages, and the number of model blocks contained in each model stage is -(Acc_Step / / PP–1)*PP*VP, and the target threshold is set to the specified value 0; where Acc_Step represents the number of reverse gradient calculations; PP represents the number of split model stages; and VP represents the number of model blocks contained in each model stage.

[0094] In the disclosed embodiments, determining a default value for the reverse gradient count based on the number of reverse gradient calculations, the number of model stages, and the number of model blocks contained in each model stage can better accommodate the limitations of model training equipment, making the model training process more stable and improving model performance. Furthermore, setting the target threshold to a specified value can reduce the complexity of parameter adjustment and improve the efficiency of model training.

[0095] In the disclosed embodiment, each model training device sets a counter, with an initial value of counter = -(Acc_Step / / PP–1)*PP*VP. 1F1B scheduling of VPP is executed, and a reverse calculation is performed. The counter count is incremented by 1, counter = counter+1.

[0096] When the counter of the rank computing device meets the following conditions, the model block of the VP-(counter-rank) / / PP is determined as the target model block of the target sorting position, and communication is performed on the target model block concurrently with the computing flow.

[0097] Condition a) counter is greater than 0.

[0098] Condition b) counter-rank can be divided by Acc_Step.

[0099] In summary, regardless of method (1) or method (2), condition b can be understood as determining the difference between the cumulative count and the model stage number of the current model training device and the target threshold; when the difference is divisible by the target number, it is determined that an integer number of batches of reverse gradient calculations have been completed. In the embodiment of the present disclosure, the difference between the cumulative count and the model stage number of the current model training device is combined with the target threshold, and the communication timing of the target model block can be flexibly controlled according to the model stage number of the current model training device. By determining that an integer number of batches of reverse gradient calculations have been completed, the model training process can be better controlled, the use of memory space can be saved, and the efficiency of model training can be improved.

[0100] Moreover, in both method (1) and method (2), the target model block at the target sorting position can be determined based on the cumulative count, the model stage number, the target threshold, the number of segmented model stages, and the number of model blocks contained in each model stage. In the embodiment of the present disclosure, by dynamically determining the target model block, flexible model adjustment can be performed according to different sorting targets, which can meet the needs of different sorting tasks. By determining the target model block as the target model block at the target sorting position, only reverse gradient calculation and parameter update are performed on the target model block, which can reduce unnecessary calculation and storage overhead and improve the efficiency of model training.

[0101] For example, when PP is 4, VP is 4, and Acc_Step is 8, the default value is -16 and the target threshold is 0. Figure 4The fourth model block 12 of the model stage 0 performs the forward process on the data represented by the rectangular box with an embedded arrow in the dotted background, and the reverse process is explained as an example by the rectangular box with a gray background and a teardrop shape. When the target model completes all the reverse gradient calculations of the fourth model block of the model stage 0, the cumulative reverse gradient count is 4. When the backward propagation process of a mini batch of all model blocks in the model stage 0 is completed, the cumulative reverse gradient count reaches 16 times, and counter = 0. When the counter count result reaches the target threshold and the difference between the model stage number of the model training device and the target threshold is divisible by the target number Acc_Step, the next calculation stage is the communication time of the target model block.

[0102] Regardless of which of the above-mentioned methods 1 or 2 is adopted, in the embodiment of the present disclosure, the method of performing aggregate communication can be ALL Reduce (global reduction), or it can be Broadcast (broadcast), Scatter (division), Gather (aggregation), etc., and the present disclosure does not impose any restrictions on this.

[0103] When all model blocks of the model stage of the current device complete aggregate communication, the parameters of the model stage are updated based on the optimizer.

[0104] like Figure 4 As shown, when all four model blocks in the model stage of each model training device complete aggregate communication, an optimizer S is used to update the parameters of the corresponding model stage, and the parameter update is performed based on the results of the reverse gradient calculation. The optimizer can use a momentum optimizer, an adaptive learning rate optimizer, a regularization optimizer, etc., which is not limited in this disclosure.

[0105] In the disclosed embodiment, using an optimizer to update the parameters of the model stage can improve the model performance. By adjusting the update method and speed of the parameters, the model can better adapt to the training data and optimization goals.

[0106] In the embodiment of the present disclosure, the target model performs at least one of the following tasks:

[0107] Natural language processing, visual information processing, multimodal information processing, protein structure prediction

[0108] In some embodiments, the target model can perform natural language processing. First, the target model receives the text to be processed, which can be a question, a command, or a statement. The target model converts the pre-processed text to be processed into a machine-understandable representation, usually converting the text to be processed into a text vector representation. According to the specific task requirements, the model will perform corresponding operations. For example, a question-answering task or a translation task is completed based on the text vector. After the task is completed, the model will generate the corresponding output. In the question-answering task, the output is the answer; in the translation task, the output is the translation result.

[0109] In other embodiments, the target model can perform visual information processing. First, the target model receives visual input, such as an image or video. For images, the target model converts them into a computer-understandable format, such as a pixel matrix. For videos, the target model splits them into a series of image frames. Depending on the specific task requirements, the model will perform corresponding operations. For example, the target model can perform classification or regression tasks on images. After the task is completed, the model will generate corresponding outputs. In classification tasks, the output of the target model can be a classification label; in regression tasks, the output of the target model can be a target detection box. The target model can also complete other perspective information processing tasks, such as outputting image segmentation results, etc., which includes classification and regression tasks.

[0110] In other embodiments, the target model can perform multimodal information processing. Multimodal information processing involves a combination of multiple input modalities and multiple feature extraction methods. First, the target model receives multimodal input, such as images and text, images and audio, etc. The target model extracts features for each input modality. Once the features of each modal input are extracted, the target model fuses them and integrates information from different modalities. According to the specific task requirements, the target model will perform corresponding operations, such as multimodal sentiment analysis tasks to predict the sentiment category of the input, and multimodal question-answering tasks to answer questions related to the input. After the task is completed, the model will generate corresponding outputs. For classification tasks, the output can be a classification label; for question-answering tasks, the output can be an answer; for recommendation tasks, the output can be a recommendation result, etc.

[0111] In other embodiments, the target model can perform protein structure prediction. First, the target model receives the protein's amino acid sequence as input and predicts the interactions and spatial relationships between the amino acids. Based on the sequence prediction results, the target model uses structural modeling techniques to generate an initial protein structure. The target model can also use optimization algorithms to adjust the conformation of the initial structure to minimize energy and satisfy physical constraints.

[0112] It should be noted that the target model disclosed in the present invention is not limited to the above-mentioned tasks and application scenarios, but can also be extended to other single-modal or multi-modal tasks, which can be determined according to the actual application scenario.

[0113] In the disclosed embodiments, by performing multiple tasks, the model can learn relevant knowledge and connections between different tasks. This knowledge sharing can promote mutual understanding and interaction between different tasks, and help improve the overall performance and understanding ability of the model.

[0114] Based on the same technical concept, the embodiment of the present disclosure also provides a performance optimization device 500 for a model training device, such as Figure 5 Shown, including:

[0115] Determination module 501, used to determine the communication timing of the current model training device for the target model block at the target sorting position, so as to facilitate synchronous modulation and aggregation communication with other model training devices among multiple model training devices for the model block at the target sorting position; wherein the current model training device is any training device among the multiple model training devices; the multiple model training devices are used to train the same target model; the target model is divided into multiple model stages, each model stage includes multiple model blocks arranged in sequence; bubbles are generated during the process of training the target model using a distributed parallel strategy due to the increase in the calculation time of the model training device due to communication operations;

[0116] The execution module 502 is configured to perform aggregate communication on the reverse gradient of the target model block at a communication opportunity.

[0117] In some embodiments, the determining module includes:

[0118] A first determining submodule is used to determine the communication timing of the target model block in the next calculation phase when the target constraint condition is satisfied;

[0119] The target constraints include:

[0120] The number of reverse gradient calculations performed by the current model training device on the target model block is greater than or equal to the target number;

[0121] At the communication timing, the reverse gradient calculation of the model block at the target sort position in the first model stage of the multiple model stages is completed.

[0122] In some embodiments, the determination module further includes a second determination submodule configured to determine whether the target constraint condition is satisfied based on the following method:

[0123] Determine the cumulative amount of reverse gradients to be synchronized for the target model block;

[0124] When the cumulative amount is greater than or equal to the target number and it is determined that the model block at the target ranking position in the first model stage of the multiple model stages can start aggregate communication, it is determined that the target model block meets the target constraint condition.

[0125] In some embodiments, the determination module further includes a third determination submodule configured to determine whether the target constraint condition is satisfied based on the following method:

[0126] The number of reverse gradient calculations for the current model training device is accumulated based on the default value;

[0127] When the cumulative count is greater than or equal to the target threshold and an integer number of batches of reverse gradient calculations are determined to be completed based on the cumulative count, determining that the target model block at the target sort position satisfies the target constraint condition;

[0128] The target threshold is determined based on the target number of times so that the number of reverse gradient calculations of the current model training device for the target model block is greater than or equal to the target number of times.

[0129] In some embodiments, the determination module further includes a fourth determination submodule, specifically configured to:

[0130] When the default value is empty, the target threshold is determined based on the number of backward gradient calculations, the number of model stages, and the number of model blocks contained in each model stage.

[0131] In some embodiments, the determination module further includes a fifth determination submodule, specifically configured to:

[0132] When a default value is determined based on the number of backward gradient calculations, the number of model stages, and the number of model blocks included in each model stage, the target threshold is determined to be set to a specified value.

[0133] In some embodiments, the third determining submodule is specifically configured to:

[0134] Determine the difference between the cumulative count and the model stage number and target threshold value of the current model training device;

[0135] When the difference is divisible by the target number, it is determined that an integer number of batches of reverse gradient calculations have been completed.

[0136] In some embodiments, the third determining submodule is specifically configured to:

[0137] Determines the target model nugget for the target sort position based on the cumulative count, the model stage number, the target threshold, the number of split model stages, and the number of model nuggets contained in each model stage.

[0138] In some embodiments, the determining module includes:

[0139] The sixth determination submodule is used to determine the communication timing of the target model block in the next calculation stage in response to receiving a synchronization message, where the synchronization message is issued by the model training device of the first model stage among multiple model stages for the model block in the target sorting position.

[0140] In some embodiments, further comprising:

[0141] The update module is used to update the parameters of the model stage based on the optimizer when all model blocks in the model stage of the current model training device complete the aggregate communication.

[0142] In some embodiments, the target model performs at least one of the following tasks:

[0143] Natural language processing, visual information processing, multimodal information processing, protein structure prediction.

[0144] In some embodiments, the target model is trained using a data parallel strategy combined with a pipeline parallel strategy using a 1F1B interleaved scheduling method, and the target model is divided into multiple model stages according to the pipeline parallel strategy, and each model stage is assigned to a corresponding model training device for training;

[0145] In the data parallel strategy, each batch of training data is divided into multiple micro-batches.

[0146] Of course, in the technical solution disclosed herein, the acquisition, storage and application of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0147] For the description of specific functions and examples of each module and submodule of the device in the embodiment of the present disclosure, please refer to the relevant description of the corresponding steps in the above method embodiment, which will not be repeated here.

[0148] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0149] Figure 6 A schematic block diagram of an example electronic device 600 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are provided as examples only and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0150] like Figure 6 As shown, the device 600 includes a computing unit 601, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 602 or a computer program loaded from a storage unit 608 into a random access memory (RAM) 603. Various programs and data required for the operation of the device 600 can also be stored in the RAM 603. The computing unit 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0151] Various components in device 600 are connected to I / O interface 605, including: an input unit 606, such as a keyboard, mouse, etc.; an output unit 607, such as various types of displays, speakers, etc.; a storage unit 608, such as a magnetic disk, optical disk, etc.; and a communication unit 609, such as a network card, modem, wireless communication transceiver, etc. The communication unit 609 allows device 600 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0152] The computing unit 601 can be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units that run machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 601 performs the various methods and processes described above, such as the performance optimization method of the model training device. For example, in some embodiments, the performance optimization method of the model training device can be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 608. In some embodiments, part or all of the computer program can be loaded and / or installed on the device 600 via the ROM 602 and / or the communication unit 609. When the computer program is loaded into the RAM 603 and executed by the computing unit 601, one or more steps of the performance optimization method of the model training device described above can be performed. Alternatively, in other embodiments, the computing unit 601 may be configured to execute the performance optimization method of the model training device in any other appropriate manner (e.g., by means of firmware).

[0153] Various embodiments of the systems and techniques described above can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip systems (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0154] The program code for implementing the method of the present disclosure can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device so that when the program code is executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0155] In the context of the present disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, device or equipment. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0156] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0157] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0158] A computer system may include a client and a server. The client and server are generally remote from each other and typically interact through a communication network. The client-server relationship arises through computer programs running on the respective computers and having a client-server relationship with each other. The server may be a cloud server, a server in a distributed system, or a server integrated with a blockchain.

[0159] According to an embodiment of the present disclosure, the electronic device may be integrated with the communication component, the display screen, and the information collection device, or may be separately provided with the communication component, the display screen, and the information collection device.

[0160] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved. This is not limited herein.

[0161] The above specific embodiments do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the principles of this disclosure shall be included within the scope of protection of this disclosure.

Claims

1. A performance optimization method for a model training device, comprising: Determine the communication timing of the current model training device for the target model block at the target sorting position, so that it can synchronize and aggregate communication with other model training devices among the multiple model training devices for the model block at the target sorting position; wherein, the current model training device is any training device among the multiple model training devices; the multiple model training devices are used to train the same target model; the target model is divided into multiple model stages, and each model stage includes multiple model blocks arranged in sequence; in the process of training the target model using a distributed parallel strategy, bubbles are generated due to the increase in the calculation time of the model training device due to the communication operation; wherein, synchronized and aggregate communication means that the target model blocks at the target sorting position of the current model training device and the other model training devices perform aggregate communication at the same communication timing; performing aggregate communication on the reverse gradient of the target model block at the communication opportunity; The determining of a communication timing of a target model block of a target sorting position by a current model training device includes: Under the condition that the target constraint condition is satisfied, determining the next calculation phase as the communication timing of the target model block; The target constraints include: The number of reverse gradient calculations performed by the current model training device on the target model block is greater than or equal to the target number; At the communication timing, the reverse gradient calculation of the model block at the target sorting position in the first model stage of the multiple model stages is completed.

2. The method according to claim 1, wherein Determining whether the target constraint conditions are met includes: Determining the cumulative amount of reverse gradients to be synchronized for the target model block; When the cumulative amount is greater than or equal to the target number and it is determined that the model block at the target sorting position in the first model stage of the multiple model stages can start aggregate communication, it is determined that the target model block meets the target constraint condition.

3. The method according to claim 1, wherein Determining whether the target constraint conditions are met includes: Accumulate the number of reverse gradient calculations of the current model training device based on the default value; When the cumulative count is greater than or equal to a target threshold and an integer number of batches of reverse gradient calculations are determined to be completed based on the cumulative count, determining that the target model block at the target sort position satisfies the target constraint condition; The target threshold is determined based on the target number of times so that the number of reverse gradient calculations performed by the current model training device on the target model block is greater than or equal to the target number of times.

4. The method according to claim 3, further comprising: When the default value is empty, the target threshold is determined based on the number of reverse gradient calculations, the number of model stages, and the number of model blocks included in each model stage.

5. The method according to claim 3, further comprising: When the default value is determined based on the number of reverse gradient calculations, the number of model stages, and the number of model blocks included in each model stage, the target threshold is determined to be set to a specified value.

6. The method according to claim 3, wherein: The determining, based on the cumulative count, that an integer number of batches of reverse gradient calculations have been completed includes: Determine a difference between the cumulative count and a model stage number of the current model training device and the target threshold; When the difference is divisible by the target number of times, it is determined that an integer number of batches of reverse gradient calculations are completed.

7. The method according to claim 3, wherein: The target model block for determining the target sorting position includes: The target model block at the target sorting position is determined based on the cumulative count, the model stage number, the target threshold, the number of segmented model stages, and the number of model blocks included in each model stage.

8. The method according to claim 1, wherein The determining of a communication timing of a target model block of a target sorting position by a current model training device includes: In response to receiving a synchronization message, determining that the next calculation stage is the communication opportunity of the target model block, the synchronization message is issued by the model training device of the first model stage among the multiple model stages for the model block at the target sorting position.

9. The method according to claim 1, further comprising: When all model blocks of the model stage of the current model training device complete aggregate communication, parameters of the model stage are updated based on the optimizer.

10. The method according to claim 1, wherein the target model performs at least one of the following tasks: Natural language processing, visual information processing, multimodal information processing, protein structure prediction.

11. The method according to any one of claims 1 to 10, wherein the target model is trained using a data parallel strategy combined with a pipeline parallel strategy using a 1F1B interleaved scheduling method, and the target model is divided into the plurality of model stages according to the pipeline parallel strategy, and each model stage is assigned to a corresponding model training device for training; In the data parallel strategy, each batch of training data is divided into multiple micro-batches.

12. A performance optimization device for a model training device, comprising: A determination module is used to determine the communication timing of the current model training device for the target model block at the target sorting position, so that it can synchronize and aggregate communication with other model training devices among multiple model training devices for the model block at the target sorting position; wherein, the current model training device is any training device among the multiple model training devices; the multiple model training devices are used to train the same target model; the target model is divided into multiple model stages, and each model stage includes multiple model blocks arranged in sequence; in the process of training the target model using a distributed parallel strategy, bubbles are generated due to the increase in the calculation time of the model training device due to the communication operation; wherein, synchronized aggregate communication means that the target model blocks at the target sorting position of the current model training device and the other model training devices perform aggregate communication at the same communication timing; An execution module, configured to perform aggregate communication on the reverse gradient of the target model block at the communication opportunity; The determining module includes: A first determining submodule is configured to determine that the next calculation phase is a communication opportunity of the target model block when the target constraint condition is satisfied; The target constraints include: The number of reverse gradient calculations performed by the current model training device on the target model block is greater than or equal to the target number; At the communication timing, the reverse gradient calculation of the model block at the target sorting position in the first model stage of the multiple model stages is completed.

13. The device according to claim 12, wherein The determining module further includes a second determining submodule, configured to determine whether the target constraint condition is satisfied based on the following method: Determining the cumulative amount of reverse gradients to be synchronized for the target model block; When the cumulative amount is greater than or equal to the target number and it is determined that the model block at the target sorting position in the first model stage of the multiple model stages can start aggregate communication, it is determined that the target model block meets the target constraint condition.

14. The device according to claim 12, wherein The determining module further includes a third determining submodule, configured to determine whether the target constraint condition is satisfied based on the following method: Accumulate the number of reverse gradient calculations of the current model training device based on the default value; When the cumulative count is greater than or equal to a target threshold and an integer number of batches of reverse gradient calculations are determined to be completed based on the cumulative count, determining that the target model block at the target sort position satisfies the target constraint condition; The target threshold is determined based on the target number of times so that the number of reverse gradient calculations performed by the current model training device on the target model block is greater than or equal to the target number of times.

15. The apparatus according to claim 14, wherein the determining module further comprises a fourth determining submodule, specifically configured to: When the default value is empty, the target threshold is determined based on the number of reverse gradient calculations, the number of model stages, and the number of model blocks included in each model stage.

16. The apparatus according to claim 14, wherein the determining module further comprises a fifth determining submodule, specifically configured to: When the default value is determined based on the number of reverse gradient calculations, the number of model stages, and the number of model blocks included in each model stage, the target threshold is determined to be set to a specified value.

17. The device according to claim 14, wherein in, The third determining submodule is specifically configured to: Determine a difference between the cumulative count and a model stage number of the current model training device and the target threshold; When the difference is divisible by the target number, it is determined that an integer number of batches of reverse gradient calculations are completed.

18. The device according to claim 14, wherein The third determining submodule is specifically configured to: The target model block at the target sorting position is determined based on the cumulative count, the model stage number, the target threshold, the number of segmented model stages, and the number of model blocks included in each model stage.

19. The device according to claim 12, wherein The determining module includes: The sixth determination submodule is used to determine that the next calculation stage is the communication timing of the target model block in response to receiving a synchronization message, wherein the synchronization message is issued by the model training device of the first model stage among the multiple model stages for the model block at the target sorting position.

20. The apparatus according to claim 12, further comprising: An updating module is used to update parameters of the model stage based on an optimizer when all model blocks of the model stage of the current model training device complete aggregate communication.

21. The apparatus of claim 12, wherein the target model performs at least one of the following tasks: Natural language processing, visual information processing, multimodal information processing, protein structure prediction.

22. The apparatus according to any one of claims 12 to 21, wherein the target model is trained using a data parallel strategy combined with a pipeline parallel strategy using a 1F1B interleaved scheduling method, and the target model is divided into the plurality of model stages according to the pipeline parallel strategy, and each model stage is assigned to a corresponding model training device for training; In the data parallel strategy, each batch of training data is divided into multiple micro-batches.

23. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 11.

24. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-11.

25. A computer program product comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 11.

Images