A model training method, device, apparatus, system and storage medium

Abstract

A model training method, device, equipment, system and storage medium are disclosed, relating to the technical field of machine learning. The management device obtains a gradient transmission strategy adaptive to the data transmission performance of multiple computing devices according to the execution information of the model, so that when the computing device trains the model based on the gradient transmission strategy, the communication overhead of the computing device will not increase due to the small amount of gradient data transmitted each time. In addition, in the case that the data transmission performance of the computing device is relatively good, the gradient transmission strategy can increase the amount of gradient data transmitted each time to improve the utilization of the data transmission performance of the system. The gradient transmission strategy effectively utilizes the data transmission performance of the computing device, so that the last group of data to be transmitted by each computing device can also be adapted to the data transmission performance of the computing device, which is beneficial to reduce the gradient tailing delay of each communication of the computing device and the management device, thereby reducing the training time of the model and improving the training efficiency of the model.

Description

[0001] This application is a divisional application, the original application number is 202310208984.1, the original application date is February 24, 2023, and the entire contents of the original application are incorporated herein by reference. TECHNICAL FIELD

[0002] The present application relates to the technical field of machine learning, in particular to a model training method, device, equipment, system and storage medium. BACKGROUND

[0003] Neural network models (hereinafter referred to as "models") are widely used in computer vision (CV), speech recognition, natural language processing (NLP) and other fields. In order to improve the training efficiency of the model, a distributed data parallel training method is often used to train the model. In this training method, the distributed training system includes a management device and multiple computing devices. Each computing device maintains the same model parameters and the same computing task, that is, each computing device deploys the same model to be trained, but different computing devices process different training data. After each computing device completes its own data calculation process, it obtains the gradient determined in the back propagation process of the model and sends the gradient to a device for performing gradient reduction of the model, such as a management device or other computing devices. Taking the management device performing gradient reduction as an example, the management device obtains update information after processing all gradients, and updates the model to obtain a target model according to the update information.

[0004] In the training process of the model, the communication of the gradient occupies a large amount of time. In order to reduce the transmission time of the gradient, the management device uses a fixed capacity gradient fusion method: the computing device packs the gradients generated by multiple tasks in the model and transmits them, and the amount of packed gradients is less than or equal to the preset fixed capacity. However, if the fixed capacity is too small, the communication overhead between the management device and the computing device will increase and the bandwidth utilization rate will be low; if the fixed capacity is too large, the amount of the last transmitted gradient in the computing device will be large, resulting in a large communication tail length in the distributed training system. As a result, the training time of the model increases, and the efficiency of the model training is affected. SUMMARY

[0005] The present application provides a model training method, device, equipment, system and storage medium, which solves the problems of low bandwidth utilization rate and large communication tail length caused by the computing device transmitting the gradient according to the fixed capacity.

[0006] In a first aspect, the present application provides a model training method. The model training method is applied to a distributed training system, the distributed training system comprising a management device and a plurality of computing devices, each computing device being deployed with a model to be trained, and the model comprising a plurality of operators. The model training method comprises: the management device obtaining execution information of the model and data transmission performance of the plurality of computing devices. The execution information comprises: execution order of the plurality of operators, execution duration, and data volume of a plurality of gradients, one gradient corresponding to one operator. And the management device determines a gradient transmission strategy of the model according to the execution information and the data transmission performance of the plurality of computing devices, and applies the gradient transmission strategy to training of the model. The gradient transmission strategy is used to indicate a manner of gradient fusion transmission of each computing device when training the model.

[0007] In the embodiment, the management device can obtain the gradient transmission strategy that is adapted to the data transmission performance of the plurality of computing devices according to the execution information of the model. In the case that the gradient transmission strategy is applied to the training process of the model, the data volume of the gradient transmitted by each computing device each time can match the data transmission performance of the computing device. Thus, the data transmission performance of the computing device will not cause the problem of increased communication overhead due to the small data volume of the gradient transmitted each time. And in the case that the data transmission performance of the computing device is good, the gradient transmission strategy can increase the data volume of the gradient transmitted each time to improve the utilization ability of the data transmission performance of the distributed training system. Moreover, since the gradient transmission strategy effectively utilizes the data transmission performance of the computing device, the last group of data to be transmitted by each computing device can also adapt to the data transmission performance of the computing device, which is beneficial to reduce the gradient tailing delay of each communication of the computing device and the management device, thereby reducing the training time of the model and improving the training efficiency of the model.

[0008] For example, the gradient transmission strategy causes the gradient in any computing device to be fused at least once for a communication operator, and the transmission duration of the fused data of the k gradients generated in any computing device in sequence is the shortest, k being a positive integer.

[0009] In an optional implementation, the management device determining the gradient transmission strategy of the model according to the execution information and the data transmission performance of the plurality of computing devices can comprise: the management device predicting one or more candidate gradient transmission strategies that meet a first optimization condition according to the execution information and the data transmission performance of the plurality of computing devices. And the management device determines the gradient transmission strategy that meets a second optimization condition from the one or more candidate gradient transmission strategies. The first optimization condition is used to indicate that the transmission duration of the fused data of the k gradients generated in the computing device in sequence is the shortest, k being a positive integer; and the second optimization condition is used to indicate that the number of times of transmission of the gradient of the distributed training system is the least.

[0010] When the gradient transfer strategy is applied to the model training process by the management device, the data transfer time of the last gradient data transmitted in each computing device is the shortest, that is, the gradient tail time is minimized, which helps to reduce the training time of the model and improve the training efficiency of the model.

[0011] In one alternative implementation, the management device predicts one or more candidate gradient transfer strategies that meet the first optimization condition based on execution information and the data transfer performance of multiple computing devices. This includes: the management device constructing an objective function based on the execution information and the data transfer performance of multiple computing devices, solving the objective function according to the first optimization condition, and obtaining one or more candidate gradient transfer strategies.

[0012] For example, the objective function includes: a first function, a second function, a third function, a fourth function, and a fifth function. The first function minimizes the sum of the generation time and transmission time of each gradient. The second function selects the maximum value between the transmission end time of the first gradient and the generation time of the second gradient; the generation order of the first gradient and the generation order of the second gradient are adjacent, and the generation order of the first gradient precedes the generation order of the second gradient. The third function determines the transmission time of each gradient based on the data volume of each gradient and the data transmission performance of the computing device where each gradient resides. The fourth function determines the sum of the generation time and communication time of the first gradient as the transmission end time of the first gradient. The fifth function determines the sum of the data volume of multiple communications as the data volume of all gradients in the model.

[0013] In this embodiment, the aforementioned gradient transfer strategy minimizes the data transfer time of the k operators whose gradient generation order is later (i.e., whose execution order is earlier). Therefore, when the management device applies the gradient transfer strategy to the model training process, the data transfer time of the last gradient data transmitted in each computing device is minimized, i.e., the gradient tailing time is minimized, which helps to reduce the training time of the model and improve the training efficiency of the model.

[0014] It is worth noting that gradient tailing is an unavoidable part of the data communication process between different computing devices. In this embodiment, the optimization conditions adopted by the management device can be used to reduce the number of data transmissions indicated by the training strategy. When the gradient tailing time of each communication remains constant or has little difference, the fewer the number of data transmissions, the smaller the total gradient tailing time required for model training. This reduces the gradient transmission time between different computing devices, thereby reducing the model training time and improving the model training efficiency.

[0015] In one optional implementation, the management device acquires the data transmission performance of multiple computing devices, including: the management device performing communication processing on the multiple computing devices to acquire communication profiles of the multiple computing devices, and acquiring the data transmission performance of the multiple computing devices based on the communication profiles. The communication processing includes at least one of communication modeling or communication simulation, and the communication profile is used to indicate the communication latency for each computing device transmitting different amounts of data.

[0016] In this embodiment, the management device can use the acquired data transmission performance to optimize the gradient transmission process of the model, so as to obtain a gradient transmission strategy that maximizes the utilization of data transmission performance and minimizes the gradient tail duration, thereby improving the training efficiency of the model.

[0017] In one alternative implementation, the management device acquires the model's execution information, including: instructing multiple computing devices to perform at least one round of training on the model, acquiring an intermediate representation (IR) graph and performance data of the model; the performance data is used to indicate the execution order and execution duration of multiple operators. Furthermore, the management device acquires the data volume and generation order of each gradient in the model based on the IR graph, and acquires the aforementioned execution information based on the performance data and the data volume and generation order of each gradient in the model.

[0018] Compared to training the model on a single computing device, the management device distributes multiple operators of the model to be trained to different computing devices, which helps to improve the training efficiency of the model.

[0019] Moreover, the management device obtains information (such as operator gradients and execution order) after at least one round of training of the model, and uses this information to obtain the execution information required for the optimization strategy. This is beneficial for the management device to perform training optimization based on the actual operation of the model on multiple computing devices, thereby improving the accuracy and effectiveness of model training.

[0020] In one alternative implementation, the gradient transfer strategy described above includes: during model training, a set or more sets of gradients to be fused in any computing device, the set of gradients including at least two sequentially adjacent gradients, the at least two gradients corresponding to at least two adjacent operators among a plurality of operators; and the set of gradients being transferred in a single communication, wherein the end time of the transfer of the set of gradients is less than or equal to the end time of the transfer of the second gradient, the second gradient being the last gradient generated before the fusion of the at least two gradients in the passing operator.

[0021] For example, the management device applies a gradient transfer strategy to model training, including: after synchronizing the gradient transfer strategy to multiple computing devices, the management device receives all gradients sent by the multiple computing devices after running the model, including gradients fused by communication operators according to the gradient transfer strategy. The management device then processes all gradients to obtain model parameter update information and updates the model parameters based on the parameter update information. The gradient processing includes at least one of weighted or averaged gradients.

[0022] In this embodiment, the management device uses the gradients fed back by multiple computing devices to update the parameters of the model, so that each computing device can participate in the model training process. This avoids the problem of reduced model accuracy caused by some computing devices not participating in model training, and helps to improve the accuracy of model training.

[0023] In one alternative implementation, after the management device applies the gradient transfer strategy to the model training, the model training method provided in this embodiment further includes: the management device acquiring the trained model, and if the trained model meets the preset model convergence conditions, the management device outputting the trained model.

[0024] In one example, the model's output can be achieved by the management device deploying the model to the user's device (or terminal).

[0025] In another example, the model's output can be as follows: the management device stores relevant information about the model (such as model parameters and deployment methods) in a designated storage device. This storage device provides a user interface, through which the user (or terminal) can call the trained model stored in the storage device to realize the model's corresponding functions.

[0026] Secondly, this application provides a model training apparatus. This apparatus is applied to the management device of a distributed training system, which further includes multiple computing devices, each deploying a model to be trained. The model includes multiple operators. Exemplarily, the model training apparatus includes: an acquisition module, a gradient transfer optimization module, and a training module. The acquisition module is used to acquire the execution information of the model and the data transmission performance of the multiple computing devices; the execution information includes: the execution order of the multiple operators, the execution duration, and the data volume of multiple gradients, with one gradient corresponding to one operator. The gradient transfer optimization module is used to determine the gradient transfer strategy of the model based on the execution information and the data transmission performance of the multiple computing devices. The gradient transfer strategy is used to indicate the method of gradient fusion transfer among the computing devices during model training. The training module is used to apply the gradient transfer strategy to the training of the model.

[0027] In one alternative implementation, the gradient transfer optimization module is specifically configured to: predict one or more candidate gradient transfer strategies that meet a first optimization criterion based on execution information and the data transfer performance of multiple computing devices; and determine a gradient transfer strategy that meets a second optimization criterion from the one or more candidate gradient transfer strategies. The first optimization criterion indicates that the transfer time of the fused data generating the k gradients in the last order in the computing devices is minimized, where k is a positive integer. The second optimization criterion indicates that the number of gradient transfers in the distributed training system is minimized.

[0028] In one alternative implementation, the gradient transfer optimization module is specifically used to: construct an objective function based on execution information and the data transfer performance of multiple computing devices; and solve the objective function according to the first optimization condition to obtain one or more candidate gradient transfer strategies.

[0029] In one optional implementation, the objective function includes: a first function, a second function, a third function, a fourth function, and a fifth function. The first function minimizes the sum of the generation time and transmission time of each gradient. The second function selects the maximum value between the transmission end time of the first gradient and the generation time of the second gradient; the generation order of the first gradient and the generation order of the second gradient are adjacent, and the generation order of the first gradient precedes the generation order of the second gradient. The third function determines the transmission time of each gradient based on the data volume of each gradient and the data transmission performance of the computing device where each gradient resides. The fourth function determines the transmission end time of the first gradient as the sum of its generation time and communication time. The fifth function determines the total data volume of all gradients in the model as the sum of the data volumes from multiple communications.

[0030] In one alternative implementation, the acquisition module is specifically used to: perform communication processing on multiple computing devices to acquire communication profiles of the multiple computing devices; and, based on the communication profiles of the multiple computing devices, acquire the data transmission performance of the multiple computing devices. The communication processing includes at least one of communication modeling or communication simulation, and the communication profiles are used to indicate the communication latency for each computing device transmitting different amounts of data.

[0031] In one alternative implementation, the acquisition module is specifically used to: instruct multiple computing devices to perform at least one round of training on the model, and acquire the model's IR plot and performance data. The performance data is used to indicate the execution order and execution duration of multiple operators. The acquisition module is also specifically used to: acquire the data volume and generation order of each gradient in the model based on the IR plot; and, based on the performance data and the data volume and generation order of each gradient in the model, acquire execution information.

[0032] In one alternative implementation, the gradient transfer strategy includes: during model training, one or more sets of gradients to be fused in any computing device, wherein a set of gradients includes at least two gradients that are sequentially adjacent, and the at least two gradients correspond to at least two adjacent operators among a plurality of operators; and, a set of gradients is transferred in a single communication, and the end time of the transfer of a set of gradients is less than or equal to the end time of the transfer of a second gradient, the second gradient being the last gradient generated before the fusion of the at least two gradients in the passing operator.

[0033] In one optional implementation, the training module is specifically used to: synchronize the gradient transfer strategy to multiple computing devices; and receive all gradients sent by the multiple computing devices after running the model, including gradients fused by communication operators according to the gradient transfer strategy. The training module is also specifically used to: process all gradients to obtain the model's parameter update information; and update the model's parameters based on the parameter update information. The processing includes at least one of weighted or averaged methods.

[0034] Thirdly, this application provides an electronic device. The electronic device includes a memory and a processor. The memory stores a set of computer instructions; the processor invokes the set of computer instructions to implement the method of the first aspect or any implementation thereof.

[0035] For example, the electronic device may refer to the management device in the first aspect, or other devices used to implement the functions of the management device in the first aspect.

[0036] Fourthly, this application provides a distributed training system. The distributed training system includes a management device and multiple computing devices, each computing device deploying a model to be trained, the model including multiple operators. The management device acquires the model's execution information and the data transmission performance of the multiple computing devices; the execution information includes the execution order of the multiple operators, execution duration, and the data volume of multiple gradients, with one gradient corresponding to one operator. Furthermore, based on the execution information and the data transmission performance of the multiple computing devices, the management device determines a gradient transfer strategy for the model, which instructs the method of gradient fusion and transfer among the computing devices during model training. Subsequently, the management device synchronizes the gradient transfer strategy to the multiple computing devices. The multiple computing devices train the model according to the gradient transfer strategy, acquiring all gradients of the model, including gradients fused and transferred according to the gradient transfer strategy. Thus, the management device processes all gradients, acquires the model's parameter update information, and updates the model's parameters based on the parameter update information. The gradient processing includes at least one of weighted or averaged gradients.

[0037] Fifthly, this application provides a computer-readable storage medium. The computer-readable storage medium includes: computer software instructions. When the computer software instructions are invoked by a computer, a method is provided to implement the first aspect or any implementation thereof.

[0038] Sixthly, this application provides a computer program product. When the computer program product is run on a computer, the computer executes the method of the first aspect or any implementation thereof.

[0039] The beneficial effects of aspects two through six can be found in the description of any implementation in aspect one, and will not be repeated here. Based on the implementations provided in the above aspects, this application can be further combined to provide even more implementations. Attached Figure Description

[0040] Figure 1A This is a schematic diagram of the structure of a model training system provided in an embodiment of this application;

[0041] Figure 1B This is a schematic diagram of the structure of a distributed training system provided in an embodiment of this application;

[0042] Figure 2A A software architecture diagram for model training provided in this application embodiment;

[0043] Figure 2B A schematic diagram illustrating the data transmission performance of multi-GPU training as provided in an embodiment of this application;

[0044] Figure 3 A schematic flowchart of a model training method provided in an embodiment of this application;

[0045] Figure 4 A schematic flowchart of a model training method provided in an embodiment of this application is shown in Figure 2.

[0046] Figure 5 A schematic diagram illustrating the temporal distribution information of a ResNet18 model provided in an embodiment of this application;

[0047] Figure 6 A schematic diagram illustrating the gradient distribution information of a ResNet18 model provided in an embodiment of this application;

[0048] Figure 7 A flowchart illustrating a model training method provided in this application embodiment. Figure 3 ;

[0049] Figure 8A A schematic diagram illustrating the optimization of the gradient transfer strategy provided in this application embodiment;

[0050] Figure 8B Schematic diagram 2 showing the optimization of the gradient transfer strategy provided in the embodiments of this application;

[0051] Figure 9 A flowchart illustrating a model training method provided in this application embodiment. Figure 4 ;

[0052] Figure 10 This is a schematic diagram of the structure of a model training device provided in an embodiment of this application. Detailed Implementation

[0053] This application provides a model training method. The method includes: a management device acquiring a gradient transfer strategy adapted to the data transmission performance of multiple computing devices based on the model's execution information. When this gradient transfer strategy is applied to the model training process, the amount of gradient data transmitted by each computing device in each transmission matches the data transmission performance of that device. Therefore, the data transmission performance of the computing devices does not suffer from increased communication overhead due to smaller gradient data transmission amounts per transmission. Furthermore, when the data transmission performance of the computing devices is superior, the gradient transfer strategy can increase the amount of gradient data transmitted per transmission, thereby improving the distributed training system's utilization of data transmission performance. Moreover, because the gradient transfer strategy effectively utilizes the data transmission performance of the computing devices, the last set of data to be transmitted by each computing device can also be adapted to its data transmission performance, which helps reduce gradient tail latency in each communication between the computing devices and the management device, thereby reducing model training time and improving model training efficiency.

[0054] To make the following embodiments clearer, a description of the relevant technologies is given first.

[0055] Deep learning is a type of machine learning technique based on deep neural network algorithms. Its main characteristic is the use of multiple nonlinear transformations to process and analyze data. Deep learning is mainly applied in perception and decision-making scenarios in the field of artificial intelligence (AI), such as image and speech recognition, natural language translation, and computer games.

[0056] Distributed parallel training: Due to the increasing size of models and training datasets, training a model using a single computing device is too time-consuming. Therefore, the training task is often divided into multiple processes (or multiple computing devices). Each process maintains the same model parameters and the same computational tasks, but different processes are used to process different training data (mini-batch data). After each process completes its data computation, data communication occurs between the computing devices to reduce the gradients. Then, the training device (such as a management device or a computing device) updates the model parameters based on the reduced gradients. If, after multiple gradient reductions, the model parameters meet the preset convergence conditions, the training device outputs the trained model.

[0057] Operators: The operators included in a model refer to the computational units within the model, such as convolutional layers, pooling layers, or fully connected layers in a deep learning model. Different processing layers have different processing logic; therefore, computational units with different processing logics are also called different types of operators. Typically, a model includes multiple different types of operators, and the support for these different types of operators varies across computing devices. For example, this difference in support indicates the difference in the computational precision of the operators run on the computing device. If the computing device has poor support for an operator, the computational precision of that operator will be low; if the computing device has good support for an operator, the computational precision of that operator will be high.

[0058] Communication operator fusion: Since each communication between different devices incurs a fixed network latency overhead, computing devices may choose to reduce the number of communications to mitigate the impact of this latency overhead. Therefore, computing devices accumulate data that would otherwise require multiple communication operators to a certain volume and then use a single communication operator for communication, thereby improving bandwidth utilization. In some optional scenarios, communication operator fusion is also referred to as model-integrated communication.

[0059] Gradient fusion: Store at least two gradients of the model in a buffer in sequence, and then integrate the buffer to obtain the integrated data.

[0060] Gradient fusion transmission: In a distributed training system, computing devices transmit the integrated data (including at least two sequentially adjacent gradients) to other devices in the same data communication process.

[0061] It's worth noting that during model training, one gradient corresponds to one operator. In gradient fusion transmission, multiple gradients of the model are transmitted in a single communication process. Therefore, the multiple operators corresponding to these gradients only need to execute a single output process; that is, these multiple operators can communicate using a single communication operator. In other words, gradient fusion transmission and communication operator fusion describe things from different perspectives. Gradient fusion transmission describes the process of transmitting the fused gradients (integrated data) from a single device, while communication operator fusion describes the process of generating the fused gradients within a single device.

[0062] Figure 1A This is a schematic diagram of the structure of a model training system provided in an embodiment of this application. Figure 1A As shown, the model training system includes a data center and multiple terminals (such as...). Figure 1A As shown in terminals 111 to 113, the data center can communicate with the terminals via a network, which can be the Internet or other networks. This network can include one or more network devices, such as routers or switches.

[0063] A data center includes one or more servers, such as Figure 1A The server 110 shown is, for example, an application server that supports application services, which can provide video services, image services, and other AI processing services based on video or images. In an alternative scenario, server 110 refers to a cluster of multiple computing devices deployed thereon. This cluster may have racks that can establish communication between the multiple servers via wired connections, such as Universal Serial Bus (USB) or Peripheral Component Interconnect Express (PCIe) high-speed buses.

[0064] Server 110 can also acquire data from terminals, process the data using AI, and then send the AI ​​processing results to the corresponding terminals. This AI processing can refer to using AI models to perform object recognition, target detection, or acquiring AI models that meet specific requirements based on samples collected by the terminals.

[0065] In addition, the data center may also include other physical devices with AI processing capabilities, such as mobile phones, tablets, or other devices.

[0066] A terminal can also be called a terminal device, user equipment (UE), mobile station (MS), or mobile terminal (MT). A terminal can be a mobile phone (such as...).Figure 1A The terminal 111 shown), and facial recognition payment devices with mobile payment functions (such as...) Figure 1A The terminal 112 shown), and the camera device (such as) with data (such as image or video) acquisition and processing functions are shown. Figure 1A The terminal shown (113) can also be a tablet computer, a computer with wireless transceiver capabilities, a virtual reality (VR) terminal device, an augmented reality (AR) terminal device, a wireless terminal in industrial control, a wireless terminal in self-driving, a wireless terminal in a smart grid, a wireless terminal in transportation safety, a wireless terminal in a smart city, a wireless terminal in a smart home, and so on. The embodiments of this application do not limit the specific technology or device form used in the terminal device.

[0067] It is worth noting that users can access AI models and other data stored on server 110 via their terminals. For example, this AI model could be a model that performs operations such as object detection, object recognition, or classification on the data.

[0068] For example, user 1 uses the AI ​​model deployed in terminal 111 to realize functions such as facial recognition and fingerprint recognition on mobile phones.

[0069] For example, user 2 can use the AI ​​model deployed in terminal 112 to realize functions such as facial recognition payment and object classification (such as product classification).

[0070] For example, the AI ​​model deployed in terminal 113 can be used to implement functions such as object detection.

[0071] Figure 1A These are merely illustrative images and should not be construed as limiting the scope of this application. The embodiments of this application do not limit the application scenarios for terminals and servers.

[0072] The following example, using a data center as a distributed training system, illustrates the hardware architecture of a distributed training system. Figure 1B This is a schematic diagram of the structure of a distributed training system provided in an embodiment of this application. The distributed training system 100 can be used to implement... Figure 1A The distributed training system 100 includes a management device 130 and multiple servers, such as the function of the server 110. Figure 1BServers 111 to 113 are included. Different servers can communicate through switch 114. In some optional cases, switch 114 can also be other network devices that support data communication, which is not limited in this application.

[0073] Management device 130 is used to assign different tasks to multiple computing devices, which may be located on the same server, such as computing device 1 and computing device 2 located on server 111.

[0074] Optionally, the aforementioned computing devices may also be located on different servers. For example, the multiple computing devices may include computing device 112B and computing device 113B, with computing device 112B located on server 112 and computing device 113B located on server 113.

[0075] Taking model training as an example, the management device 130 can distribute multiple operators in the model to different computing devices, with each computing device used to perform operations on a portion of the multiple operators. In some optional implementations, the training data required for training the model can be stored on a dedicated storage device in the distributed training system 100, such as the memory included in the management device 130, or a shared storage space accessible to multiple computing devices, etc., which is not limited in this application.

[0076] In terms of hardware implementation, the management device 130 may include a memory 1310 and at least one processor 1320, which can implement the model training method provided in the above embodiments, and the memory 1310 is used to store the software instructions corresponding to the above model training method. As an optional implementation, in terms of hardware implementation, the management device 130 may refer to a chip or chip system that encapsulates one or more processors 1320.

[0077] In an optional configuration, the management device 130 may further include a communication interface 1330, which can be used to send and receive data. For example, the communication interface 1330 is used to receive or send model parameters, intermediate model data, network load information of the distributed training system, execution information of the model, etc.; the communication interface 1330 can be implemented through interface circuits included in the management device 130.

[0078] In the embodiments of this application, the communication interface 1330, processor 1320, and memory 1310 can be connected via a bus 1340, which can be divided into an address bus, a data bus, a control bus, etc. The bus 1340 can be a PCIe bus, or an extended industry standard architecture (EISA) bus, a unified bus (Ubus or UB), a compute express link (CXL), a cache coherent interconnect for accelerators (CCIX), etc.

[0079] Processor 1320 is the computing and control core of management device 130. Processor 1320 can be a very large-scale integrated circuit. The processor 1320 contains an operating system and other software programs, enabling it to access memory and various PCIe devices. Processor 1320 includes one or more processor cores. These cores can be, for example, central processing units (CPUs) or other application-specific integrated circuits (ASICs). Processor 1320 can also be other general-purpose processors, digital signal processors (DSPs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. In practical applications, management device 130 may also include multiple processors.

[0080] The memory 1310 can be a storage medium used to store execution information and other information. The memory 1310 can be volatile memory or non-volatile memory, or a combination of both. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous linked dynamic random access memory (SLDRAM), and direct rambus RAM (DR RAM).

[0081] The memory 1310 can be used to store execution information of the model to be trained by the distributed training system, such as the execution timing of each operator in the model, the amount of gradient data corresponding to the operator, and the computing device to which the operator belongs.

[0082] Taking server 111 as an example, for Figure 1B The hardware implementation of the server is illustrated by way of example. The server 111 includes a network card 111A, a computing device 1 and a computing device 2. The network card and the computing device, as well as different computing devices, can communicate with each other through PCIe bus, EISA bus, UB, CXL or CCIX, etc.

[0083] It is worth noting that the data transmission bandwidth between different computing devices can be the same or different. Here, data transmission bandwidth refers to the maximum amount of data transmitted per unit time; alternatively, it can be called the maximum amount of communication that can be transmitted per unit time. With a constant amount of communication, the larger the data transmission bandwidth, the shorter the time (or communication latency) required for data exchange or communication between computing devices performing tasks (or operators).

[0084] CombinationFigure 1B The computing devices shown and Table 1 below illustrate the data transfer bandwidth between multiple computing devices. The unit of data transfer bandwidth is: million bytes per second (MB / s).

[0085] Table 1

[0086] Units: MB / s Computing device 1 Computing device 2 Computing device 112B Computing device 112C Computing device 1 - 1000 500 500 Computing device 2 - - 500 500 Computing device 112B - - - 1000 Computing device 112C - - - -

[0087] Depending on the server to which the computing devices belong, communication in a distributed training system includes communication between devices within the server (local communication) and communication between multiple servers (cross-server communication). Cross-server communication, besides requiring data transmission within the server to the network interface card (NIC), also necessitates coordination between the NICs of multiple servers and the switch, resulting in higher communication overhead than local communication. As the cluster size and the number of servers continue to increase, cross-server communication will become a bottleneck for high-performance cluster computing tasks.

[0088] Referring to Table 1, when different computing devices are located on the same server, the communication method between the computing devices is local communication: for example, the data transmission bandwidth between computing device 1 and computing device 2 is 1000MB / s, and the data transmission bandwidth between computing device 112B and computing device 112C is 1000MB / s.

[0089] When different computing devices are located on different servers, the communication method between computing devices is cross-server communication: for example, the data transmission bandwidth between computing device 1 and computing devices 112B and 112C is 500MB / s; the data transmission bandwidth between computing device 2 and computing devices 112B and 112C is also 500MB / s.

[0090] The computing device described above can be a processor, which includes one or more processor cores. A processor core can be, for example, a CPU or an ASIC. A processor can also be other general-purpose processors, graphics processing units (GPUs), neural network processing units (NPUs), DSPs, FPGAs, or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. In practical applications, a computing device may also include multiple processors.

[0091] The above description of the server hardware implementation is based on server 111 as an example. Servers 112 and 113 may also include hardware required for performing model training. For example, server 112 includes network card 112A, computing device 112B and computing device 112C, and server 113 includes network card 113A, computing device 113B and computing device 113C.

[0092] Figure 1B Only the operators (operators 1 to m) deployed in server 111 are shown, but servers 112 and 113 may also deploy operators from the model, thereby enabling the distributed training system 100 to train the model.

[0093] It is worth noting that, Figure 1B The management device 130 and the server are different physical devices, but in some alternative implementations, the functions of the management device 130 and the server can be integrated into a single physical device. For example, a specific computing device in the server can be used to implement the functions of the management device 130 in the distributed training system 100, and this application does not limit this.

[0094] In the above embodiments, the process of training the model by the distributed training system 100 is implemented on multiple servers. However, in some optional cases, the process of training the model by the distributed training system 100 can be implemented on a single server, such as server 111. One computing device is designated in server 111 to implement the functions of management device 130, while other computing devices are used to execute tasks for different operators in the model.

[0095] Below Figure 1A and Figure 1B Based on this, this embodiment provides an architecture for model training, such as Figure 2A As shown, Figure 2A This application provides a software architecture diagram for model training. The software architecture 200 includes: a communication performance predictor 210, a model gradient information parser 220, a model runtime information parser 230, and a gradient fusion and transfer strategy optimizer 240.

[0096] The communication performance predictor 210 is used for the training environment (e.g., Figure 1B The distributed training system 100 performs communication modeling or communication simulation to obtain the data transmission performance of the computing devices in the training environment. For example, the communication performance predictor 210 constructs a performance predictor for the communication task, taking the communication volume (the amount of data transmitted) of each communication as input and outputting the communication latency.

[0097] For example, the communication performance predictor 210 can obtain the data transmission performance of multiple computing devices by performing communication processing on the multiple computing devices, obtaining a communication profile of the multiple computing devices, and obtaining the data transmission performance of the multiple computing devices based on the communication profile. Here, the communication profile is used to indicate the communication latency of each computing device transmitting different amounts of data. It is worth noting that the communication processing here includes at least one of communication modeling or communication simulation.

[0098] like Figure 2B As shown, Figure 2B This diagram illustrates the data transmission performance of multi-GPU training as provided in an embodiment of this application. "Multi-GPU" refers to multiple computing devices, with each GPU representing a single computing device. Physically, a GPU can be an accelerator card used for model training or other hardware cards with model training capabilities. Taking eight GPUs as an example, the AllReduce algorithm is used to fuse communication operators between the different GPUs to transmit gradients or other data. Figure 2B (1) shows the communication latency of these 8 cards at different communication volumes. As the communication volume increases, the communication latency of these 8 cards increases. Figure 2B Figure (2) shows that for different communication volumes, the data transmission bandwidth provided by these 8 cards for data transmission first increases and then decreases as the communication volume increases. That is, as the communication volume gradually increases, the bandwidth utilization first increases and then decreases.

[0099] In this embodiment, the communication performance predictor 210 can use the acquired data transmission performance to optimize the gradient transmission process of the model, so as to obtain a gradient transmission strategy that maximizes the utilization of data transmission performance and minimizes the gradient tail duration, thereby improving the training efficiency of the model.

[0100] In one alternative approach, the data transmission performance can be represented by data transmission bandwidth and other metrics, such as the inherent latency of the computing device performing the communication task or the latency associated with the type of communication task. For a description of data transmission bandwidth, please refer to the relevant content in Table 1 above, which will not be repeated here.

[0101] The model gradient information parser 220 is used to obtain information such as the gradient size distribution (e.g., gradient generation time and data size for each gradient), the dependencies (or calling relationships, etc.) of the operators that generate gradients, and the operators that generate gradients in the trained model. One gradient corresponds to one operator. It is worth noting that in some cases, some operators in the model have no model parameters; these operators are only used to pass data, etc., and therefore, these operators do not have corresponding gradients.

[0102] The model runtime information parser 230 is used to obtain the timing information of the computation tasks of different operators during the training of the model, such as the execution order of the operators, the execution duration of each operator and the start and end times (e.g., transmission start time and transmission end time).

[0103] The gradient fusion transfer strategy optimizer 240 is used to construct the objective function for communication operator fusion and solve the objective function according to preset optimization conditions, thereby optimizing the gradient transfer strategy during model training. Possible implementations of this objective function can be found in Equations 1 to 6 below, and will not be elaborated upon here.

[0104] The software components of the above software architecture 200 can be provided with corresponding operating resources (such as computing resources, storage resources, or network resources) by the computing devices included in the server. It is worth noting that... Figure 2A The possible implementations for model training provided in this embodiment are merely illustrative and should not be construed as limiting this application. Depending on the hardware resources included in the server, the above software may be divided into more or fewer modules (or units), etc., and this application does not limit this.

[0105] To address the problems raised in the background section, the following will be discussed... Figure 1A , Figure 1B and Figure 2A Based on the architecture shown, embodiments of this application provide a model training method. For example... Figure 3 As shown, Figure 3 This is a schematic flowchart illustrating a model training method provided in an embodiment of this application. This model training method can be applied to... Figure 1A The server shown is 110 or the data center, or Figure 1B The distributed training system 100 shown is or any server.

[0106] The model training method is performed by the management device, such as the management device being... Figure 1B The management device 130 in the system, or the management device is a computing device in any server used to implement the functions of the management device 130. The model training method provided in this application embodiment includes the following steps S310 to S330.

[0107] S310, the management device responds to training requests for a model, obtains the model's execution information and the data transmission performance of multiple computing devices.

[0108] In one possible example, the training request is an instruction from the user received by the management device.

[0109] In another possible example, the training request is an instruction generated by the management device for a specific model after the training time of the model in S310 has been reached, based on the training status of multiple models in the distributed training system.

[0110] Optionally, the management device can also acquire execution information and data transmission performance during model training without responding to training requests, and can then begin subsequent gradient transfer optimization and model training processes.

[0111] In S310, the execution information includes: the execution order of multiple operators, the execution time, and the data volume of multiple gradients, with one gradient corresponding to one operator.

[0112] by Figure 1B Taking the model in the example as an example, the model includes operators 1 to m, and the index of each operator is the execution order of that operator. Table 2 below shows the execution time of different operators and the amount of gradient data corresponding to each operator.

[0113] Table 2

[0114]

[0115] Wherein, operator 3 is an operator used to pass intermediate results in the model (i.e., operator 3 does not have model parameters), and therefore, operator 3 does not have gradients during the backpropagation process of the model.

[0116] It is worth noting that the content shown in Table 2 above is only an example provided in this embodiment and should not be construed as limiting this application.

[0117] Regarding the method for obtaining execution information, this embodiment provides a possible implementation method, such as... Figure 4 As shown, Figure 4 This is a schematic flowchart of a model training method provided in an embodiment of this application. Step S310 above may include steps S311 to S313.

[0118] S311, The management device instructs multiple computing devices to perform at least one round of training on the model to obtain the model's intermediate representation (IR) graph and performance data.

[0119] One round of training refers to all computing devices fully utilizing the training data they can access to run the model and obtain its output. After comparing the model's output with the data labels of the training data, the model's loss value can be determined. The magnitude of this loss value can be used to determine the model's training direction (i.e., the gradient).

[0120] The IR graph of a model is used to indicate: the control flow in the model and the dependencies between operators and training data, and to provide an interface for graph-level optimization. The IR graph can also contain rich semantic information for compilation and provide extensible edit semantics for custom operators.

[0121] This performance data indicates the execution order and execution time of multiple operators. Details regarding the execution order and execution time can be found in Table 2 and will not be repeated here.

[0122] In one possible example, the performance data could be runtime sequence information of the model obtained by the management device through parsing the profiling trace file during model training, thereby obtaining the execution order, start and end times, etc., of the computational tasks corresponding to each gradient. (See below.) Figure 5 As shown, Figure 5 This is a schematic diagram of the temporal distribution information of a ResNet18 model provided in an embodiment of this application. Figure 5 The horizontal axis represents the execution order of the operators, and the vertical axis represents the time interval distribution of the generation times of two adjacent gradients.

[0123] S312. The management device obtains the amount of data and the generation order of each gradient in the model based on the IR diagram.

[0124] For example, for multiple operators in the model, the management device constructs a model gradient information parser (e.g., based on the model's IR graph) Figure 2B The model gradient information parser (in the model) is used to obtain the gradient distribution information of the model. Figure 6 As shown, Figure 6 This diagram illustrates the gradient distribution information of a ResNet18 model provided in an embodiment of this application. Different operators in the ResNet18 model have different amounts of gradient data, and some operators do not have model parameters; therefore, these operators do not have gradients.

[0125] S313. The management device obtains execution information based on performance data and the amount of data and generation order of each gradient in the model.

[0126] Compared to training the model on a single computing device, the management device distributes multiple operators of the model to be trained across different computing devices, which improves the training efficiency. Furthermore, the management device acquires information about the model after at least one round of training (operator gradients and execution order, etc.) and uses this information to obtain the execution information needed for optimization strategies. This allows the management device to perform training optimization based on the actual operation of the model across multiple computing devices, thereby improving the accuracy and effectiveness of model training.

[0127] For details on how to obtain data transmission performance, please refer to [link / reference]. Figure 2A The relevant descriptions in Table 1 will not be repeated here.

[0128] Please continue to refer to Figure 3 The model training method provided in this embodiment also includes the following steps S320 and S330.

[0129] S320, the management device determines the gradient transfer strategy of the model based on the execution information and the data transmission performance of multiple computing devices.

[0130] The gradient transfer strategy is used to indicate how gradients are transferred between computing devices during model training.

[0131] In one alternative implementation, the gradient transfer strategy enables gradients in any computing device to be fused at least once using a communication operator, and the transmission time of the fused data of the k gradients generated in the order of the last gradient in any computing device is minimized, where k is a positive integer.

[0132] If the execution order of an operator in the model is earlier, then the gradient generation order of that operator is later. In the aforementioned gradient transfer strategy, the data transfer time of the k operators whose gradient generation order is later (i.e., whose execution order is earlier) is minimized.

[0133] Therefore, when the gradient transfer strategy is applied to the model training process by the management device, the data transfer time of the last gradient data transmitted in each computing device is the shortest, that is, the gradient tail time is the smallest, which helps to reduce the training time of the model and improve the training efficiency of the model.

[0134] Regarding possible implementation methods for determining the gradient transmission strategy by the management device, please refer to the following: Figure 7 , Figure 8A and Figure 8B The description of that will not be repeated here.

[0135] S330, the management device applies gradient transfer strategies to the training of the model.

[0136] In one possible implementation, after the management device applies the gradient transfer strategy to the model training, the model training method provided in this embodiment further includes: the management device acquiring the trained model; and, if the trained model meets the preset model convergence conditions, the management device outputs the trained model.

[0137] In the first possible example, the way the management device outputs the model includes: the management device deploying the model to the user (or terminal).

[0138] In a second possible example, the management device outputs the model in the following ways: the management device stores relevant information about the model (such as model parameters and deployment methods) in a designated storage device. This storage device provides a user interface, through which the user (or terminal) can call the trained model stored in the storage device to realize the corresponding functions of the model.

[0139] Compared to a management device transmitting gradients generated by computing devices at a fixed capacity, in this embodiment, the management device can obtain a gradient transmission strategy adapted to the data transmission performance of multiple computing devices based on the model's execution information. When this gradient transmission strategy is applied to the model's training process, the gradient data stream transmitted by each computing device each time can match the data transmission performance of the computing device. Therefore, the data transmission performance of the computing devices will not suffer from increased communication overhead due to the small amount of gradient data transmitted each time.

[0140] Furthermore, when the data transmission performance of computing devices is superior, the gradient transfer strategy can increase the amount of gradient data transmitted each time, thereby improving the distributed training system's ability to utilize data transmission performance. Moreover, because the gradient transfer strategy effectively utilizes the data transmission performance of computing devices, the last set of data to be transmitted by each computing device can also be adapted to its data transmission performance. This helps reduce gradient tail latency in each communication between computing and management devices, thereby reducing model training time and improving model training efficiency.

[0141] Regarding the implementation of S320 above, this embodiment provides a possible example, such as... Figure 7 As shown, Figure 7 A flowchart illustrating a model training method provided in this application embodiment. Figure 3 The process of determining the gradient transmission strategy for the management device may include the following steps S710 and S720.

[0142] S710, the management device predicts one or more candidate gradient transmission strategies that meet the first optimization condition based on the execution information and the data transmission performance of multiple computing devices.

[0143] The first optimization criterion is used to indicate that the transmission time of the fused data of the k gradients generated in the computing device in the order of k is the shortest, where k is a positive integer. In other words, the first optimization criterion is used to minimize the gradient tailing time of each computing device.

[0144] For example, the management device constructs an objective function based on execution information and the data transmission performance of multiple computing devices, and solves the objective function according to the first optimization condition to obtain one or more candidate gradient transmission strategies.

[0145] In one alternative implementation, the above objective functions include: the first function, the second function, the third function, the fourth function, and the fifth function.

[0146] The first function is used to minimize the sum of the gradient generation time and the propagation time for each gradient.

[0147] The second function is used to select the maximum value between the transmission end time of the first gradient and the communication generation time of the second gradient. The generation order of the first gradient and the generation order of the second gradient are adjacent, and the generation order of the first gradient precedes the generation order of the second gradient.

[0148] The third function is used to determine the transmission duration of each gradient based on the amount of data for each gradient and the data transmission performance of the computing device where each gradient is located.

[0149] The fourth function is used to determine the sum of the communication generation time and transmission duration of the first gradient as the transmission end time of the first gradient.

[0150] The fifth function is used to determine the total gradient data of the model, which is the sum of the data from multiple communications.

[0151] The above functions can be implemented using formulas:

[0152]

[0153] The meanings of the symbols in Formulas 1 to 5 above are shown in Table 3 below.

[0154] Table 3

[0155] Meaning of symbols Meaning of symbols M Total data volume of all gradients in a model n Number of communication operators executed by a model for one round of training ET Time at which each gradient is produced by a computing task (gradient production time) ct Data transmission duration of each gradient et Maximum of transmission end time of each gradient and production time of next gradient pre_et Transmission end time of previous gradient TS Data volume of gradients for each data communication

[0156] Here, n can also be called the number of data transmissions required for the model to perform one round of training, that is, one communication operator corresponds to one data communication.

[0157] Formula 1 is used to implement the function of the first function, Formula 2 is used to implement the function of the second function, Formula 3 is used to implement the function of the third function, Formula 4 is used to implement the function of the fourth function, and Formula 5 is used to implement the function of the fifth function.

[0158] It is worth noting that the objective functions and formulas described above are merely examples provided in this embodiment for minimizing the gradient tail duration of each computing device in the gradient transfer strategy, and should not be construed as limiting this application. When those skilled in the art use other functions or formulas to implement the model training method provided in this application, they also fall within the scope of protection of this application.

[0159] In this embodiment, the aforementioned gradient transfer strategy minimizes the data transfer time of the k operators whose gradient generation order is later (i.e., whose execution order is earlier). Therefore, when the management device applies the gradient transfer strategy to the model training process, the data transfer time of the last gradient data transmitted in each computing device is minimized, i.e., the gradient tailing time is minimized, which helps to reduce the training time of the model and improve the training efficiency of the model.

[0160] S720, The management device determines a gradient transfer strategy that meets the second optimization condition from one or more candidate gradient transfer strategies.

[0161] The second optimization condition is used to indicate that the gradient of the distributed training system is transmitted in the fewest possible ways.

[0162] For example, the second optimization condition can be achieved through the following formula 6, and the meaning of each symbol in formula 6 can be found in the relevant content of Table 3.

[0163] θ * =argmin(n),θ([i1,i1,…i n ]) Formula 6

[0164] Gradient tailing is an unavoidable part of data communication between different computing devices. In this embodiment, the optimization conditions used by the management device can reduce the number of data transmissions indicated by the training strategy. When the gradient tailing duration of each communication remains constant or shows little difference, fewer data transmissions result in a smaller total gradient tailing duration required for model training. This reduces the gradient transmission time between different computing devices, thereby reducing model training time and improving training efficiency.

[0165] For Figure 7 The optimization method provided in this embodiment also includes a specific example, such as... Figure 8A As shown, Figure 8A This is a schematic diagram of the optimization of the gradient transmission strategy provided in the embodiments of this application. For gradient 1 and gradient 2, the start time of the next gradient predicted by the management device is ET3(et2), and the end time of the transmission after merging gradient 1 and gradient 2 is ET(fusion1+2)<ET3(et2). Therefore, the management device confirms that gradient 1 and gradient 2 can be merged during the gradient transmission process.

[0166] For gradients 1 and 2, the management device predicts that the transmission end time of the gradients is et4, while the transmission end time of the merged gradients 3 and 4 is ET(fusion3+4)<et4. Therefore, the management device confirms that gradients 3 and 4 cannot be merged during the gradient transmission process.

[0167] As an optional implementation, the gradient transfer strategy provided in this embodiment may include two parts.

[0168] Part 1: During model training, one or more sets of gradients to be fused on any computing device. One set of gradients includes at least two sequentially adjacent gradients, which correspond to at least two adjacent operators among a plurality of operators.

[0169] Part Two: A set of gradients is transmitted during a single communication, and the transmission end time of this set of gradients is less than or equal to the transmission end time of the second gradient. This second gradient is the last gradient generated before the fusion of at least two gradients using the passage operator.

[0170] For any one of multiple computing devices, if m operators are deployed in that device, the communication fusion method for these m operators can be varied. For example, the management device determines a gradient transmission strategy based on the temporal distribution information of the gradients and data transmission performance. This strategy ensures that the transmission end time of the fused gradients is before the end time of the gradients before fusion, which helps reduce the gradient transmission time between different devices, thereby reducing model training time and improving model training efficiency.

[0171] For example, this embodiment also provides another specific example, such as Figure 8B As shown, Figure 8B This is a second schematic diagram illustrating the optimization of the gradient transfer strategy provided in this application embodiment. The computational flow includes operators 1 to m, and the training and optimization process.

[0172] These m operators generate a total of 62 gradients. After analyzing the model, the management device obtains the gradient transfer strategy, which includes the following four parts.

[0173] ①: Fusing gradients 1 to 13 (13 gradients fused) results in a data size of 28MB.

[0174] ②: Fuse gradients 14 to 52 (39 gradients fused), the corresponding data size is 15MB.

[0175] ③: Fuse gradients 53 to 61 (9 gradients fused), the corresponding data size is 0.4MB.

[0176] ④: Transmit gradient 62 separately, which corresponds to a data size of 0.2MB.

[0177] Given that the gradient transfer strategy includes the above-mentioned elements, this describes the process by which the management device trains the model according to the gradient transfer strategy. Figure 3 As shown in S330, this embodiment provides a possible specific example, such asFigure 9 As shown, Figure 9 A flowchart illustrating a model training method provided in this application embodiment. Figure 4 The above S330 may include steps S910 to S940.

[0178] S910, the management device synchronizes the gradient transfer strategy to multiple computing devices.

[0179] In the first possible scenario, the management device sends the gradient transfer strategy to multiple computing devices, and each computing device deploys the gradient transfer strategy separately.

[0180] In the second possible scenario, the management device writes the gradient transfer strategy into the configuration unit of the distributed training system. This configuration unit can be used to manage all configuration information during model training. Before the computing device runs the model, it reads the gradient transfer strategy from the configuration unit and executes the corresponding communication operator fusion (AllReduce, or gradient data fusion).

[0181] The two possible scenarios above are merely possible implementations provided in this example and should not be construed as limitations on the synchronization method of the gradient transmission strategy in this application.

[0182] S920: The computing device runs the model according to the synchronous gradient transmission strategy and sends the acquired gradients to the management device.

[0183] The gradients obtained in S920 include the gradients (fused data or fused gradients) after the computing device fuses the communication operators according to the gradient transmission strategy.

[0184] Accordingly, the management device receives all gradients sent by multiple computing devices after running the model, including gradients fused by communication operators according to the gradient transmission strategy.

[0185] S930, the management device processes all gradients and obtains the model's parameter update information.

[0186] For example, the process in S930 includes at least one of weighted or averaged.

[0187] For example, for a single operator in the model, the management device averages all gradients associated with that single operator to obtain the gradient mean (parameter update information), and then uses the gradient mean to update the parameters of that single operator.

[0188] For example, different computing devices have different weights. For a single operator in the model, the management device weights all gradients associated with that single operator with the weights of the computing device that generated the gradient. After obtaining the gradient weight value, the gradient weight value (parameter update information) is used to update the parameters of that single operator.

[0189] S940. The management device updates the parameters of the model based on the parameter update information.

[0190] For example, for a single operator in the model, the management device adds the parameter update information to the original model parameters of that single operator to obtain the updated model parameters for that single operator. After the management device updates the model parameters of all operators, it obtains the trained model.

[0191] It should be noted that the above embodiments are merely examples of the model training method provided in this application. The model training method provided in this application can be applied to different types of distributed training systems such as the MindSpore training framework or the PyTorch training framework. The specific type of distributed training system in this application is not limited.

[0192] It is understood that, in order to achieve the functions in the above embodiments, the management device and computing device include hardware structures and / or software modules corresponding to the execution of each function. Those skilled in the art should readily recognize that, based on the units and method steps of the various examples described in conjunction with the embodiments disclosed in this application, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in hardware or by computer software driving hardware depends on the specific application scenario and design constraints of the technical solution.

[0193] The above text combines Figures 1A to 9 The method provided according to this embodiment is described in detail below, and will be combined with Figure 10 This describes the model training apparatus provided according to this embodiment.

[0194] Figure 10 This is a schematic diagram of a model training device provided in an embodiment of this application. This model training device can be used to implement the device management function in the above method embodiments, and therefore can also achieve the beneficial effects of the above method embodiments. In this embodiment, the device can be as follows: Figure 1A The server 110 or data center shown could also be Figure 1A The distributed training system, management device 130, or any server shown may include computing devices that can be any module (such as a chip) applied to the aforementioned devices or systems.

[0195] like Figure 10As shown, the model training device 1000 includes: an acquisition module 1010, a gradient transfer optimization module 1020, and a training module 1030. The model training device 1000 is used to implement the device management function described in the above method embodiment.

[0196] For example, the acquisition module 1010 is used to acquire the execution information of the model and the data transmission performance of multiple computing devices; the execution information includes: the execution order of multiple operators, the execution duration, and the data volume of multiple gradients, with one gradient corresponding to one operator. The gradient transfer optimization module 1020 is used to determine the gradient transfer strategy of the model based on the execution information and the data transmission performance of multiple computing devices. The gradient transfer strategy is used to indicate the method of gradient fusion transfer among the computing devices during model training. The training module 1030 is used to apply the gradient transfer strategy to the training of the model.

[0197] In one optional implementation, the gradient transfer optimization module 1020 is specifically configured to: predict one or more candidate gradient transfer strategies that meet a first optimization condition based on execution information and the data transfer performance of multiple computing devices; and determine a gradient transfer strategy that meets a second optimization condition from the one or more candidate gradient transfer strategies. The first optimization condition indicates that the transfer time of the fused data generating the k gradients in the last order in the computing devices is minimized, where k is a positive integer. The second optimization condition indicates that the number of gradient transfers in the distributed training system is minimized.

[0198] In one alternative implementation, the gradient transfer optimization module 1020 is specifically used to: construct an objective function based on execution information and the data transfer performance of multiple computing devices; and solve the objective function according to the first optimization condition to obtain one or more candidate gradient transfer strategies.

[0199] In one optional implementation, the objective function includes: a first function, a second function, a third function, a fourth function, and a fifth function. The first function minimizes the sum of the generation time and transmission time of each gradient. The second function selects the maximum value between the transmission end time of the first gradient and the generation time of the second gradient; the generation order of the first gradient and the generation order of the second gradient are adjacent, and the generation order of the first gradient precedes the generation order of the second gradient. The third function determines the transmission time of each gradient based on the data volume of each gradient and the data transmission performance of the computing device where each gradient resides. The fourth function determines the transmission end time of the first gradient as the sum of its generation time and communication time. The fifth function determines the total data volume of all gradients in the model as the sum of the data volumes from multiple communications.

[0200] In one optional implementation, the acquisition module 1010 is specifically used to: perform communication processing on multiple computing devices to acquire communication profiles of the multiple computing devices; and, based on the communication profiles of the multiple computing devices, acquire the data transmission performance of the multiple computing devices. The communication processing includes at least one of communication modeling or communication simulation, and the communication profiles are used to indicate the communication latency for each computing device transmitting different amounts of data.

[0201] In one alternative implementation, the acquisition module 1010 is specifically used to: instruct multiple computing devices to perform at least one round of training on the model, and acquire the IR diagram and performance data of the model. The performance data is used to indicate the execution order and execution duration of multiple operators. The acquisition module 1010 is also specifically used to: acquire the data volume and generation order of each gradient in the model based on the IR diagram; and acquire execution information based on the performance data and the data volume and generation order of each gradient in the model.

[0202] In one alternative implementation, the gradient transfer strategy includes: during model training, one or more sets of gradients to be fused in any computing device, wherein a set of gradients includes at least two gradients that are sequentially adjacent, and the at least two gradients correspond to at least two adjacent operators among a plurality of operators; and, a set of gradients is transferred in a single communication, and the end time of the transfer of a set of gradients is less than or equal to the end time of the transfer of a second gradient, the second gradient being the last gradient generated before the fusion of the at least two gradients in the passing operator.

[0203] In one optional implementation, the training module 1030 is specifically used to: synchronize the gradient transfer strategy to multiple computing devices; and receive all gradients sent by the multiple computing devices after running the model, including gradients fused by communication operators according to the gradient transfer strategy. The training module 1030 is also specifically used to: process all gradients to obtain the model's parameter update information; and update the model's parameters according to the parameter update information. The processing includes at least one of weighted averaging or averaging.

[0204] It should be understood that the model training apparatus 1000 of this application embodiment can be implemented by a DPU. The model training apparatus 1000 according to the embodiment of this application can correspond to the execution of the methods described in the embodiment of this application, and the above and other operations and / or functions of each unit in the model training apparatus 1000 are respectively for implementing the corresponding processes of each method, which will not be described again here for the sake of brevity.

[0205] The method steps in this embodiment can be implemented in hardware or by a processor executing software instructions. The software instructions can consist of corresponding software modules, which can be stored in random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, portable hard disks, CD-ROMs, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor, enabling the processor to read information from and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and storage medium can reside in an ASIC. Alternatively, the ASIC can reside in a computing device. Of course, the processor and storage medium can also exist as discrete components in a network device or terminal device.

[0206] This application also provides a chip system including a processor for implementing the functions of the processing device described above. In one possible design, the chip system further includes a memory for storing program instructions and / or data. This chip system may be composed of chips or may include chips and other discrete devices.

[0207] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer programs or instructions. When the computer program or instructions are loaded and executed on a computer, the processes or functions described in the embodiments of this application are performed entirely or partially. The computer can be a general-purpose computer, a special-purpose computer, a computer network, a network device, a user equipment, or other programmable device. The computer program or instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another. For example, the computer program or instructions can be transferred from one website, computer, server, or data center to another website, computer, server, or data center via wired or wireless means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium, such as a floppy disk, hard disk, or magnetic tape; it can also be an optical medium, such as a digital video disc (DVD); or it can be a semiconductor medium, such as a solid-state drive (SSD).

[0208] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A model training method, characterized in that, The method is executed by a management device located in a distributed training system for training models, the distributed training system further comprising multiple computing devices, the method comprising: The execution information of the model and the data transmission performance of the multiple computing devices are obtained. The execution information includes the execution order, execution duration, and data volume of multiple gradients of the model, with one gradient corresponding to one operator. The data transmission performance includes data transmission bandwidth, the inherent latency of the computing device in executing the communication task, or the latency associated with the type of communication task. Based on the execution information and the data transmission performance of the plurality of computing devices, a gradient transmission strategy for the model is determined. The gradient transmission strategy is used to indicate the manner in which each computing device performs gradient fusion transmission during the training of the model. The gradient transmission strategy includes: during the training of the model, one or more sets of gradients to be fused in any computing device, the set of gradients including at least two gradients that are sequentially adjacent, the at least two gradients corresponding to at least two adjacent operators among the plurality of operators; and the set of gradients is transmitted in one communication process, and the transmission end time of the set of gradients is less than or equal to the transmission end time of a second gradient, the second gradient being the last gradient generated before the at least two gradients are fused by the communication operator. The gradient transfer strategy is applied to the training of the model.

2. The method according to claim 1, characterized in that, Determining the gradient transfer strategy of the model based on the execution information and the data transmission performance of the multiple computing devices includes: Based on the execution information and the data transmission performance of the plurality of computing devices, one or more candidate gradient transmission strategies that meet the first optimization condition are predicted; the first optimization condition is used to indicate that the transmission time of the fused data of the k gradients generated in the computing device is the shortest, where k is a positive integer. Gradient transfer strategy that meets the second optimization condition is determined from the one or more candidate gradient transfer strategies; the second optimization condition is used to indicate that the number of gradient transfers in the distributed training system is minimized.

3. The method according to claim 2, characterized in that, Based on the execution information and the data transmission performance of the plurality of computing devices, predict one or more candidate gradient transmission strategies that meet the first optimization condition, including: A target function is constructed based on the execution information and the data transmission performance of the multiple computing devices; Solve the objective function according to the first optimization condition to obtain the one or more candidate gradient transfer strategies.

4. The method according to claim 3, characterized in that, The objective function includes: a first function, a second function, a third function, a fourth function, and a fifth function; The first function is used to: minimize the sum of the generation time and transmission time of each gradient; The second function is used to: select the maximum value between the transmission end time of the first gradient and the generation time of the second gradient; the generation order of the first gradient and the generation order of the second gradient are adjacent, and the generation order of the first gradient precedes the generation order of the second gradient; The third function is used to: determine the transmission duration of each gradient based on the amount of data for each gradient and the data transmission performance of the computing device where each gradient is located; The fourth function is used to: determine the sum of the generation time and communication time of the first gradient as the transmission end time of the first gradient; The fifth function is used to determine the sum of the data volume from multiple communications as the total data volume of all gradients of the model.

5. The method according to any one of claims 1-4, characterized in that, Obtaining the data transmission performance of the plurality of computing devices includes: Communication processing is performed on the plurality of computing devices to obtain a communication profile of the plurality of computing devices; the communication processing includes at least one of communication modeling or communication simulation, and the communication profile is used to indicate the communication latency of each computing device transmitting data with different amounts of communication. Based on the communication profiles of the multiple computing devices, the data transmission performance of the multiple computing devices is obtained.

6. The method according to any one of claims 1-4, characterized in that, Obtain the execution information of the model, including: The plurality of computing devices are instructed to perform at least one round of training on the model to obtain the intermediate representation IR diagram and performance data of the model; the performance data is used to indicate the execution order and execution time of the plurality of operators; The amount of data and the generation order of each gradient in the model are obtained from the IR diagram. The execution information is obtained based on the performance data and the amount of data and generation order of each gradient in the model.

7. The method according to any one of claims 1-4, characterized in that, The step of applying the gradient transfer strategy to the training of the model includes: Synchronize the gradient transfer strategy to the multiple computing devices; Receive all gradients sent by the plurality of computing devices after running the model, wherein all gradients include gradients fused by communication operators according to the gradient transmission strategy; All gradients are processed to obtain the parameter update information of the model; the processing includes at least one of weighted averaging or averaging. The parameters of the model are updated based on the parameter update information.

8. A model training device, characterized in that, Located in a distributed training system for training models, the distributed training system also includes multiple computing devices; The device includes: An acquisition module is used to acquire the execution information of the model and the data transmission performance of the multiple computing devices; the execution information includes: the execution order, execution duration and data volume of multiple operators of the model, with one gradient corresponding to one operator; the data transmission performance includes data transmission bandwidth, the inherent latency of the computing device executing the communication task, or the latency associated with the type of communication task. A gradient transfer optimization module is used to determine the gradient transfer strategy of the model based on the execution information and the data transmission performance of the plurality of computing devices. The gradient transfer strategy is used to indicate the method of gradient fusion transfer among the computing devices during the training of the model. The gradient transfer strategy includes: during the training of the model, one or more sets of gradients to be fused in any computing device, the set of gradients including at least two gradients that are sequentially adjacent, the at least two gradients corresponding to at least two adjacent operators among the plurality of operators; and the set of gradients is transferred in one communication process, and the end time of the transfer of the set of gradients is less than or equal to the end time of the transfer of a second gradient, the second gradient being the last gradient generated before the at least two gradients are fused by the communication operator. The training module is used to apply the gradient transfer strategy to the training of the model.

9. The apparatus according to claim 8, characterized in that, The gradient transfer optimization module is specifically configured to: predict one or more candidate gradient transfer strategies that meet the first optimization condition based on the execution information and the data transfer performance of the plurality of computing devices; and determine a gradient transfer strategy that meets the second optimization condition from the one or more candidate gradient transfer strategies. The first optimization condition is used to indicate that the transmission time of the k gradients generated in the computing device in the order of k is the shortest, where k is a positive integer; the second optimization condition is used to indicate that the number of gradient transmissions in the distributed training system is the fewest.

10. The apparatus according to claim 9, characterized in that, The gradient transfer optimization module is specifically used to: construct an objective function based on the execution information and the data transmission performance of the multiple computing devices; and solve the objective function according to the first optimization condition to obtain one or more candidate gradient transfer strategies.

11. The apparatus according to claim 10, characterized in that, The objective function includes: a first function, a second function, a third function, a fourth function, and a fifth function; The first function is used to: minimize the sum of the generation time and transmission time of each gradient; The second function is used to: select the maximum value between the transmission end time of the first gradient and the generation time of the second gradient; the generation order of the first gradient and the generation order of the second gradient are adjacent, and the generation order of the first gradient precedes the generation order of the second gradient; The third function is used to: determine the transmission duration of each gradient based on the amount of data for each gradient and the data transmission performance of the computing device where each gradient is located; The fourth function is used to: determine the sum of the generation time and communication time of the first gradient as the transmission end time of the first gradient; The fifth function is used to determine the sum of the data volume from multiple communications as the total data volume of all gradients of the model.

12. The apparatus according to any one of claims 8-11, characterized in that, The acquisition module is specifically used for: performing communication processing on the plurality of computing devices to acquire communication profiles of the plurality of computing devices; and acquiring data transmission performance of the plurality of computing devices based on the communication profiles of the plurality of computing devices; the communication processing includes at least one of communication modeling or communication simulation, and the communication profile is used to indicate the communication latency of each computing device transmitting different amounts of data.

13. The apparatus according to any one of claims 8-11, characterized in that, The acquisition module is specifically used for: The plurality of computing devices are instructed to perform at least one round of training on the model to obtain the intermediate representation IR diagram and performance data of the model; the performance data is used to indicate the execution order and execution time of the plurality of operators; The acquisition module is further specifically used for: acquiring the data volume and generation order of each gradient in the model based on the IR diagram; and acquiring the execution information based on the performance data and the data volume and generation order of each gradient in the model.

14. The apparatus according to claim 13, characterized in that, The training module is specifically used to: synchronize the gradient transfer strategy to the plurality of computing devices; and receive all gradients sent by the plurality of computing devices after running the model, wherein all gradients include gradients fused by communication operators according to the gradient transfer strategy. The training module is further specifically used for: processing all gradients to obtain parameter update information of the model; and updating the parameters of the model according to the parameter update information; the processing includes at least one of weighted or averaged methods.

15. An electronic device, characterized in that, include: Memory and processor; The memory is used to store a set of computer instructions; The processor is used to invoke the set of computer instructions to implement the method of any one of claims 1 to 7.

16. A distributed training system, characterized in that, include: The system includes a management device and multiple computing devices, each of which is equipped with a model to be trained, the model comprising multiple operators; The management device is used to acquire the execution information of the model and the data transmission performance of the multiple computing devices; the execution information includes: the execution order and execution duration of the multiple operators and the data volume of multiple gradients, with one gradient corresponding to one operator; the data transmission performance includes data transmission bandwidth, the inherent latency of the computing devices executing communication tasks, or the latency associated with the type of communication task; based on the execution information and the data transmission performance of the multiple computing devices, a gradient transmission strategy for the model is determined, the gradient transmission strategy being used to indicate the method of gradient fusion transmission among the computing devices during the training of the model; the gradient transmission strategy is synchronized to the multiple computing devices; the gradient transmission strategy includes: during the training of the model, one or more sets of gradients to be fused in any computing device, the set of gradients including at least two gradients that are sequentially adjacent, the at least two gradients corresponding to at least two adjacent operators among the multiple operators; and the set of gradients is transmitted in one communication process, and the transmission end time of the set of gradients is less than or equal to the transmission end time of the second gradient, the second gradient being the last gradient generated before the at least two gradients are fused by the communication operator; The plurality of computing devices are used to train the model according to the gradient transfer strategy.

17. A computer-readable storage medium, characterized in that, include: Computer software instructions; When the computer software instructions are invoked by a computer, the method of any one of claims 1 to 7 is implemented.

18. A computer program product, characterized in that, When the computer program product is run on a computer, the computer performs the method according to any one of claims 1 to 7.

Images