Model training method, device, equipment, system and storage medium

Abstract

The invention discloses a model training method, device, equipment and system and a storage medium, and relates to the technical field of machine learning. The management device obtains the gradient transmission strategy adapted to the data transmission performance of the plurality of computing devices according to the execution information of the model, so that when the computing devices train the model based on the gradient transmission strategy, the communication overhead of the computing devices cannot be increased due to small gradient data volume transmitted each time. Moreover, under the condition that the data transmission performance of the computing equipment is better, the gradient transmission strategy can increase the gradient data volume transmitted each time, so that the utilization capability of the system on the data transmission performance is improved. According to the gradient transmission strategy, the data transmission performance of the computing equipment is effectively utilized, so that the last group of data to be transmitted by each computing equipment can also adapt to the data transmission performance of the computing equipment, the gradient trailing time delay of each communication of the computing equipment and the management equipment is reduced, the training time of a model is reduced, and the training efficiency is improved. And the model training efficiency is improved.

Description

[0001] This application is a divisional application. The application number of the original application is 202310208984.1, and the original application date is February 24, 2023. The entire contents of the original application are incorporated into this application by reference. Technical Field

[0002] The present application relates to the field of machine learning technology, and in particular to a model training method, apparatus, device, system and storage medium. Background Art

[0003] Neural network models (hereinafter referred to as "models") are widely used in computer vision (CV), speech recognition, natural language processing (NLP) and other fields. In order to improve the training efficiency of the model, the model is often trained in a distributed data parallel training mode. In this training mode, the distributed training system includes a management device and multiple computing devices, each of which maintains the same model parameters and the same computing tasks, that is, each computing device deploys the same model to be trained, but different computing devices process different training data. After each computing device completes its own data calculation process, it obtains the gradient determined by the model during the back propagation process, and sends the gradient to a device used to execute the gradient reduction of the model, such as a management device or other computing device. Taking the execution of the gradient reduction by the management device as an example, the management device processes all the gradients to obtain update information, and updates the model according to the update information to obtain the target model.

[0004] During the training process of the model, the communication of gradients takes up a lot of time. In order to reduce the transmission time of the gradients, the management device adopts a fixed-capacity gradient fusion method: the computing device packages the gradients generated by multiple tasks in the model and transmits them, and the amount of packaged gradients is less than or equal to the preset fixed capacity. However, if the fixed capacity is too small, the communication overhead between the management device and the computing device will increase and the bandwidth utilization will be low; if the fixed capacity is too large, the amount of gradients finally transmitted in the computing device will be large, resulting in a large communication tailing time in the distributed training system. As a result, the training time of the model increases and the efficiency of model training is affected. Summary of the invention

[0005] The present application provides a model training method, apparatus, device, system and storage medium, which solve the problems of low bandwidth utilization and long communication tail time caused by the computing device fusing and transmitting gradients according to a fixed capacity.

[0006] In a first aspect, the present application provides a model training method. The model training method is applied to a distributed training system, which includes a management device and multiple computing devices, each computing device deploys a model to be trained, and the model includes multiple operators. The model training method includes: the management device obtains the execution information of the model and the data transmission performance of multiple computing devices. The execution information includes: the execution order of multiple operators, the execution time and the data volume of multiple gradients, one gradient corresponds to one operator. And, the management device determines the gradient transmission strategy of the model based on the execution information and the data transmission performance of multiple computing devices, and applies the gradient transmission strategy to the training of the model. The gradient transmission strategy is used to indicate the way in which each computing device performs gradient fusion transmission when training the model.

[0007] In this embodiment, the management device can obtain a gradient transmission strategy adapted to the data transmission performance of multiple computing devices based on the execution information of the model. When the gradient transmission strategy is applied to the training process of the model, the amount of gradient data transmitted by each computing device each time can match the data transmission performance of the computing device. Therefore, the data transmission performance of the computing device will not cause the problem of increased communication overhead due to the small amount of gradient data transmitted each time. In addition, when the data transmission performance of the computing device is better, the gradient transmission strategy can increase the amount of gradient data transmitted each time to improve the distributed training system's ability to utilize the data transmission performance. Moreover, since the gradient transmission strategy effectively utilizes the data transmission performance of the computing device, the last set of data to be transmitted by each computing device can also adapt to the data transmission performance of the computing device, which is conducive to reducing the gradient tailing delay of each communication between the computing device and the management device, thereby reducing the training time of the model and improving the training efficiency of the model.

[0008] Exemplarily, the gradient transmission strategy enables the gradient in any computing device to be fused by the communication operator at least once, and the transmission time of the fused data of the k gradients generated in the latter order in any computing device is the shortest, where k is a positive integer.

[0009] In an optional implementation, the management device determines the gradient transmission strategy of the model based on the execution information and the data transmission performance of multiple computing devices, which may include: the management device predicts one or more candidate gradient transmission strategies that meet the first optimization condition based on the execution information and the data transmission performance of multiple computing devices. And, the management device determines the gradient transmission strategy that meets the second optimization condition from one or more candidate gradient transmission strategies. The first optimization condition is used to indicate that the transmission time of the fusion data of the k gradients generated in the computing device in the latter order is the shortest, and k is a positive integer; the second optimization condition is used to indicate that the number of transmissions of the gradient of the distributed training system is the least.

[0010] When the management device applies the gradient transmission strategy to the model training process, the data transmission time of the gradient data last transmitted in each computing device is the shortest, that is, the gradient tailing time is the shortest, which is beneficial to reducing the training time of the model and improving the training efficiency of the model.

[0011] In an optional implementation, the management device predicts one or more candidate gradient transmission strategies that meet the first optimization condition based on the execution information and the data transmission performance of multiple computing devices, including: the management device constructs an objective function based on the execution information and the data transmission performance of multiple computing devices, and solves the objective function according to the first optimization condition to obtain one or more candidate gradient transmission strategies.

[0012] Exemplarily, the objective function includes: a first function, a second function, a third function, a fourth function and a fifth function. The first function is used to: minimize the sum of the generation time and the transmission time of each gradient. The second function is used to: select the maximum value of the transmission end time of the first gradient and the generation time of the second gradient; the generation order of the first gradient and the generation order of the second gradient are adjacent, and the generation order of the first gradient is before the generation order of the second gradient. The third function is used to: determine the transmission time of each gradient according to the data volume of each gradient and the data transmission performance of the computing device where each gradient is located. The fourth function is used to: determine that the sum of the generation time and communication time of the first gradient is the transmission end time of the first gradient. The fifth function is used to: determine that the sum of the data volume of multiple communications is the data volume of all gradients of the model.

[0013] In this embodiment, the aforementioned gradient transmission strategy makes the data transmission time of the k operators whose gradient generation order is later (that is, the execution order of the operators is earlier) the shortest. Therefore, when the management device applies the gradient transmission strategy to the model training process, the data transmission time of the gradient data transmitted last in each computing device is the shortest, that is, the gradient tailing time is the shortest, which is conducive to reducing the training time of the model and improving the training efficiency of the model.

[0014] It is worth noting that the data communication process between different computing devices inevitably has a gradient tailing time. The optimization condition adopted by the management device in this embodiment can be used to reduce the number of data transmissions indicated by the training strategy. When the gradient tailing time of each communication remains unchanged or the difference is not large, the fewer the number of data transmissions, the smaller the total gradient tailing time required for model training, which reduces the gradient transmission time between different computing devices, thereby reducing the training time of the model and improving the training efficiency of the model.

[0015] In an optional implementation, the management device obtains the data transmission performance of multiple computing devices, including: the management device performs communication processing on the multiple computing devices, obtains the communication profiles of the multiple computing devices, and obtains the data transmission performance of the multiple computing devices according to the communication profiles of the multiple computing devices. The communication processing includes at least one of communication modeling or communication simulation, and the communication profile is used to indicate the communication delay of each computing device transmitting data of different communication volumes.

[0016] In this embodiment, the management device can use the acquired data transmission performance to optimize the gradient transmission process of the model to obtain a gradient transmission strategy that maximizes the utilization of the data transmission performance and shortens the gradient tailing time as much as possible, thereby improving the training efficiency of the model.

[0017] In an optional implementation, the management device obtains the execution information of the model, including: the management device instructs multiple computing devices to perform at least one round of training on the model, and obtains the intermediate representation (IR) graph and performance data of the model; the performance data is used to indicate the execution order and execution duration of multiple operators. And, the management device obtains the data volume and generation order of each gradient in the model according to the IR graph, and obtains the aforementioned execution information according to the performance data and the data volume and generation order of each gradient in the model.

[0018] Compared with training the model on a separate computing device, the management device distributes multiple operators of the model to be trained to different computing devices, which is conducive to improving the training efficiency of the model.

[0019] Moreover, the management device obtains the information obtained by the model after at least one round of training (such as the gradient and execution order of operators), and obtains the execution information required for the optimization strategy based on the information, which is conducive to the management device to perform training optimization according to the actual operation of the model on multiple computing devices, thereby improving the accuracy and training effect of model training.

[0020] In an optional implementation, the above gradient transmission strategy includes: in the training process of the model, one or more groups of gradients to be fused in any computing device, the group of gradients includes at least two gradients that are generated in an order of being adjacent, and the at least two gradients correspond to at least two adjacent operators among multiple operators. And, the group of gradients is transmitted in one communication process, and the transmission end time of the group of gradients is less than or equal to the transmission end time of the second gradient, and the second gradient is the last gradient generated before the at least two gradients are fused by the passing operator.

[0021] Exemplarily, the management device applies the gradient transmission strategy to the training of the model, including: after the management device synchronizes the gradient transmission strategy to multiple computing devices, it receives all gradients sent by the multiple computing devices after running the model, and all gradients include gradients after communication operator fusion according to the gradient transmission strategy. And, the management device processes all gradients, obtains parameter update information of the model, and updates the parameters of the model according to the parameter update information. The gradient processing process includes at least one of weighting or averaging.

[0022] In this embodiment, the management device uses the gradients fed back by multiple computing devices to update the parameters of the model, so that each computing device can participate in the model training process, avoiding the problem of some computing devices not participating in the model training, resulting in reduced model accuracy, which is beneficial to improving the accuracy of model training.

[0023] In an optional implementation, after the management device applies the gradient transmission strategy to the training of the model, the model training method provided in this embodiment also includes: the management device obtains the trained model, and when the trained model meets the preset model convergence conditions, the management device outputs the trained model.

[0024] In one example, the output method of the model may be: the management device deploys the model to the user end (or terminal).

[0025] In another example, the output method of the model can be: the management device stores the relevant information of the model (such as model parameters and deployment methods, etc.) to a designated storage device, and the storage device provides a user interface. The user end (or terminal) can call the trained model stored in the storage device through the user interface to implement the corresponding functions of the model.

[0026] In the second aspect, the present application provides a model training device. The device is applied to a management device of a distributed training system, and the distributed training system also includes multiple computing devices, each computing device deploys a model to be trained, and the model includes multiple operators. Exemplarily, the model training device includes: an acquisition module, a gradient transmission optimization module and a training module. The acquisition module is used to obtain the execution information of the model and the data transmission performance of multiple computing devices; the execution information includes: the execution order of multiple operators, the execution time and the data volume of multiple gradients, and one gradient corresponds to one operator. The gradient transmission optimization module is used to determine the gradient transmission strategy of the model based on the execution information and the data transmission performance of multiple computing devices. The gradient transmission strategy is used to indicate the way in which each computing device performs gradient fusion transmission when training the model. The training module is used to apply the gradient transmission strategy to the training of the model.

[0027] In an optional implementation, the gradient transmission optimization module is specifically used to: predict one or more candidate gradient transmission strategies that meet the first optimization condition based on the execution information and the data transmission performance of multiple computing devices. And, determine the gradient transmission strategy that meets the second optimization condition from one or more candidate gradient transmission strategies. Among them, the first optimization condition is used to indicate that the transmission time of the fusion data of the k gradients generated in the computing device in the latter order is the shortest, and k is a positive integer. The second optimization condition is used to indicate that the number of transmissions of the gradient of the distributed training system is the least.

[0028] In an optional implementation, the gradient transmission optimization module is specifically used to: construct an objective function according to the execution information and the data transmission performance of multiple computing devices, and solve the objective function according to the first optimization condition to obtain one or more candidate gradient transmission strategies.

[0029] In an optional implementation, the objective function includes: a first function, a second function, a third function, a fourth function and a fifth function. The first function is used to minimize the sum of the generation time and the transmission time of each gradient. The second function is used to select the maximum value of the transmission end time of the first gradient and the generation time of the second gradient; the generation order of the first gradient and the generation order of the second gradient are adjacent, and the generation order of the first gradient is before the generation order of the second gradient. The third function is used to determine the transmission time of each gradient according to the data volume of each gradient and the data transmission performance of the computing device where each gradient is located. The fourth function is used to determine that the sum of the generation time and the communication time of the first gradient is the transmission end time of the first gradient. The fifth function is used to determine that the sum of the data volume of multiple communications is the data volume of all gradients of the model.

[0030] In an optional implementation, the acquisition module is specifically used to: perform communication processing on multiple computing devices to obtain communication profiles of the multiple computing devices. And, based on the communication profiles of the multiple computing devices, obtain data transmission performance of the multiple computing devices. The communication processing includes at least one of communication modeling or communication simulation, and the communication profile is used to indicate the communication delay of each computing device transmitting data of different communication volumes.

[0031] In an optional implementation, the acquisition module is specifically used to: instruct multiple computing devices to perform at least one round of training on the model, and obtain the IR graph and performance data of the model. The performance data is used to indicate the execution order and execution duration of multiple operators. The acquisition module is also specifically used to: obtain the data volume and generation order of each gradient in the model according to the IR graph. And, according to the performance data and the data volume and generation order of each gradient in the model, obtain execution information.

[0032] In an optional implementation, the gradient transmission strategy includes: during the training process of the model, one or more groups of gradients to be fused in any computing device, a group of gradients includes at least two gradients that are adjacent in generation order, and the at least two gradients correspond to at least two adjacent operators among multiple operators. Also, a group of gradients is transmitted in one communication process, and the transmission end time of a group of gradients is less than or equal to the transmission end time of a second gradient, and the second gradient is the last gradient generated before the at least two gradients are fused by a passing operator.

[0033] In an optional implementation, the training module is specifically used to: synchronize the gradient transmission strategy to multiple computing devices. And receive all gradients sent by multiple computing devices after running the model, all gradients include gradients after communication operator fusion according to the gradient transmission strategy. The training module is also specifically used to: process all gradients to obtain parameter update information of the model. And, update the parameters of the model according to the parameter update information. The processing process includes at least one of weighting or averaging.

[0034] In a third aspect, the present application provides an electronic device. The electronic device includes: a memory and a processor. The memory is used to store a set of computer instructions; the processor is used to call a set of computer instructions to implement the method of the first aspect or any implementation method of the first aspect.

[0035] Exemplarily, the electronic device may refer to the management device in the first aspect, or other devices used to implement the functions of the management device in the first aspect.

[0036] In a fourth aspect, the present application provides a distributed training system. The distributed training system includes: a management device and multiple computing devices, each computing device deploys a model to be trained, and the model includes multiple operators. The management device obtains the execution information of the model and the data transmission performance of multiple computing devices; the execution information includes: the execution order of multiple operators, the execution time and the data volume of multiple gradients, and one gradient corresponds to one operator. And, the management device determines the gradient transmission strategy of the model based on the execution information and the data transmission performance of multiple computing devices, and the gradient transmission strategy is used to indicate the way in which each computing device performs gradient fusion transmission when training the model. Afterwards, the management device synchronizes the gradient transmission strategy to multiple computing devices. Multiple computing devices train the model according to the gradient transmission strategy and obtain all gradients of the model, and all gradients include gradients that are gradient-fused and transmitted according to the gradient transmission strategy. Thereby, the management device processes all gradients, obtains parameter update information of the model, and updates the parameters of the model according to the parameter update information. The gradient processing process includes at least one of weighting or averaging.

[0037] In a fifth aspect, the present application provides a computer-readable storage medium. The computer-readable storage medium includes: computer software instructions. When the computer software instructions are called by a computer, the method of the first aspect or any one of the implementation modes of the first aspect is implemented.

[0038] In a sixth aspect, the present application provides a computer program product. When the computer program product is run on a computer, the computer executes the method in the first aspect or any one of the implementations of the first aspect.

[0039] The beneficial effects of the second to sixth aspects can refer to the description of any implementation in the first aspect, which will not be repeated here. Based on the implementations provided in the above aspects, this application can also be further combined to provide more implementations. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1A A schematic diagram of the structure of a model training system provided in an embodiment of the present application;

[0041] Figure 1B A schematic diagram of the structure of a distributed training system provided in an embodiment of the present application;

[0042] Figure 2A A software architecture diagram for model training provided in an embodiment of the present application;

[0043] Figure 2B A schematic diagram of data transmission performance of multi-card training provided in an embodiment of the present application;

[0044] Figure 3 A flowchart of a model training method provided in an embodiment of the present application is shown in FIG1 ;

[0045] Figure 4 A second flow chart of a model training method provided in an embodiment of the present application;

[0046] Figure 5 A schematic diagram of time distribution information of a ResNet18 model provided in an embodiment of the present application;

[0047] Figure 6 A schematic diagram of gradient distribution information of a ResNet18 model provided in an embodiment of the present application;

[0048] Figure 7 A schematic diagram of a model training method provided in an embodiment of the present application Figure 3 ;

[0049] Fig. 8A Schematic diagram 1 of the optimization of the gradient transmission strategy provided in the embodiment of the present application;

[0050] Figure 8B Optimization schematic diagram 2 of the gradient transmission strategy provided in the embodiment of the present application;

[0051] Fig. 9 A schematic diagram of a model training method provided in an embodiment of the present application Figure 4 ;

[0052] Fig.10 A schematic diagram of the structure of a model training device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0053] The present application provides a model training method. The model training method includes: the management device can obtain a gradient transmission strategy adapted to the data transmission performance of multiple computing devices according to the execution information of the model. When the gradient transmission strategy is applied to the training process of the model, the amount of gradient data transmitted by each computing device each time can match the data transmission performance of the computing device. Therefore, the data transmission performance of the computing device will not cause the problem of increased communication overhead due to the small amount of gradient data transmitted each time. And, when the data transmission performance of the computing device is better, the gradient transmission strategy can increase the amount of gradient data transmitted each time to improve the distributed training system's ability to utilize the data transmission performance. Moreover, because the gradient transmission strategy effectively utilizes the data transmission performance of the computing device, the last set of data to be transmitted by each computing device can also adapt to the data transmission performance of the computing device, which is conducive to reducing the gradient tailing delay of each communication between the computing device and the management device, thereby reducing the training time of the model and improving the training efficiency of the model.

[0054] In order to make the following embodiments clearer, an introduction to the related art is first given.

[0055] Deep learning: A type of machine learning technology based on deep neural network algorithms, whose main feature is the use of multiple nonlinear transformations to process and analyze data. Deep learning is mainly used in perception, decision-making and other scenarios in the field of artificial intelligence (AI), such as image and speech recognition, natural language translation, computer games, etc.

[0056] Distributed data parallel training: As the model and training data sets increase, it takes too long to train the model using a single computing device. Therefore, the training task is often divided into multiple processes (or multiple computing devices). Each process maintains the same model parameters and the same computing tasks, but different processes are used to process different training data (mini batch data). After each process completes its own data calculation, it communicates data between the computing devices and reduces the gradient. Then, the training device (such as a management device or a computing device) updates the parameters of the model according to the reduced gradient; if the parameters of the model meet the preset convergence conditions after multiple gradient reductions, the training device outputs the trained model.

[0057] Operator: The operator included in the model refers to the computing unit in the model, such as the convolution layer, pooling layer or fully connected layer in the deep learning model. Different processing layers have different processing logics. Therefore, computing units with different processing logics are also called different types of operators. Usually, a model includes multiple different types of operators, and the support of computing devices for different types of operators is different. Exemplarily, the difference in support indicates the difference in the computing accuracy of the computing device running the operator. If the computing device supports the operator poorly, the computing accuracy of the operator in the computing device is low; if the computing device supports the operator well, the computing accuracy of the operator in the computing device is high.

[0058] Communication operator fusion: Since there is a fixed network delay overhead each time different devices communicate, in order to reduce the impact of this delay overhead, the computing device will choose to reduce the number of communications. As a result, the computing device accumulates data that originally requires multiple communication operators to communicate to a certain amount of data and then uses one communication operator to communicate, thereby improving the bandwidth utilization of communication. In some optional situations, communication operator fusion is also called model fusion communication.

[0059] Gradient fusion: Store at least two gradients of the model in a buffer in order, and integrate the buffer to obtain the integrated data.

[0060] Gradient fusion transmission: In a distributed training system, the computing device transmits the above integrated data (including at least two sequentially adjacent gradients) to other devices, and the integrated data is transmitted in the same data communication process.

[0061] It is worth noting that during the training process of the model, one gradient corresponds to one operator. In the case of gradient fusion transmission, multiple gradients of the model are transmitted in one communication process. Therefore, the multiple operators corresponding to these multiple gradients only need to perform a single output process, that is, the multiple operators can use one communication operator for communication. In other words, the description angles of gradient fusion transmission and communication operator fusion are different. Gradient fusion transmission describes the transmission process of the fused gradient (integrated data) from the device, and communication operator fusion describes the generation process of fused gradients from a single device.

[0062] Figure 1A A schematic diagram of the structure of a model training system provided in an embodiment of the present application. Figure 1A As shown, the model training system includes a data center and multiple terminals (such as Figure 1A The data center can communicate with the terminals 111 to 113 shown in the figure through a network, which can be the Internet or other networks. The network can include one or more network devices, such as a router or a switch.

[0063] A data center consists of one or more servers, such as Figure 1A The server 110 shown is, for example, an application server supporting application services, which can provide video services, image services, other AI processing services based on video or images, etc. In an optional scenario, the server 110 refers to a cluster in which multiple computing devices are deployed, and the cluster may have a rack, and the rack may establish communication for the multiple servers through a wired connection, such as a universal serial bus (USB) or a peripheral component interconnect express (PCIe) high-speed bus.

[0064] The server 110 may also obtain data from the terminal, perform AI processing on the data, and send the result of the AI ​​processing to the corresponding terminal. The AI ​​processing may refer to object recognition and target detection of the data using an AI model, or may refer to obtaining an AI model that meets the requirements based on the samples collected by the terminal.

[0065] In addition, the data center can also include other physical devices with AI processing capabilities, such as mobile phones, tablets, or other devices.

[0066] Terminal can also be called terminal equipment, user equipment (UE), mobile station (MS), mobile terminal (MT), etc. A terminal can be a mobile phone (such as Figure 1A The terminal 111 shown), a face-scanning payment device with a mobile payment function (such as Figure 1A The terminal 112 shown in the figure), a camera device (such as a camera 112) having data (such as image or video) acquisition and processing functions Figure 1A The terminal 113 shown in the figure may also be a tablet computer (Pad), a computer with wireless transceiver function, a virtual reality (VR) terminal device, an augmented reality (AR) terminal device, a wireless terminal in industrial control, a wireless terminal in self-driving, a wireless terminal in smart grid, a wireless terminal in transportation safety, a wireless terminal in smart city, a wireless terminal in smart home, etc. The embodiments of the present application do not limit the specific technology and specific device form adopted by the terminal device.

[0067] It is worth noting that the user can obtain the AI ​​model stored in the server 110 through the terminal. For example, the AI ​​model can be a model for performing operations such as target detection, object recognition or classification on data.

[0068] For example, user 1 uses the AI ​​model deployed in terminal 111 to implement functions such as mobile phone face recognition and fingerprint recognition.

[0069] For another example, user 2 utilizes the AI ​​model deployed in terminal 112 to implement functions such as face-swiping payment and object classification (such as product classification).

[0070] For example, the AI ​​model deployed in terminal 113 can be used to implement functions such as object detection.

[0071] Figure 1A This is only a schematic diagram and should not be construed as limiting the present application. The embodiments of the present application do not limit the application scenarios of the terminal and the server.

[0072] The following uses the distributed training system as an example to illustrate the hardware architecture of the distributed training system. Figure 1B A schematic diagram of a distributed training system provided in an embodiment of the present application. The distributed training system 100 can be used to implement Figure 1A In order to realize the function of the server 110, the distributed training system 100 includes a management device 130 and multiple servers, such as Figure 1BDifferent servers can communicate through the switch 114. In some optional situations, the switch 114 can also be other network devices that support data communication, which is not limited in this application.

[0073] The management device 130 is used to assign different tasks to multiple computing devices. The multiple computing devices may be located on the same server, such as computing device 1 and computing device 2 are located on server 111 .

[0074] Optionally, the above-mentioned multiple computing devices may also be located in different servers. For example, the multiple computing devices include computing device 112B and computing device 113B, computing device 112B is located in server 112 , and computing device 113B is located in server 113 .

[0075] Taking model training as an example, the management device 130 may distribute multiple operators in the model to different multiple computing devices, and each computing device is used to operate on some of the multiple operators. In some optional implementations, the training data required for training the model can be stored on a dedicated storage device in the distributed training system 100, such as the storage device is a memory included in the management device 130, or the storage device is a shared storage space accessible to multiple computing devices, etc., which is not limited in this application.

[0076] In hardware implementation, the management device 130 may include: a memory 1310 and at least one processor 1320, the processor 1320 may implement the model training method provided in the above embodiment, and the memory 1310 is used to store software instructions corresponding to the above model training method. As an optional implementation, in hardware implementation, the management device 130 may refer to a chip or chip system encapsulated with one or more processors 1320.

[0077] In an optional scenario, the management device 130 may further include a communication interface 1330, which may be used to send and receive data. For example, the communication interface 1330 is used to receive or send model parameters, intermediate data of the model, network load information of the distributed training system, execution information of the model, etc.; the communication interface 1330 may be implemented by an interface circuit included in the management device 130.

[0078] In the embodiment of the present application, the communication interface 1330, the processor 1320 and the memory 1310 may be connected via a bus 1340, and the bus 1340 may be divided into an address bus, a data bus, a control bus, etc. The bus 1340 may be a PCIe bus, or an extended industry standard architecture (EISA) bus, a unified bus (Ubus or UB), a compute express link (CXL), a cache coherent interconnect for accelerators (CCIX), etc.

[0079] The processor 1320 is the computing core and control core of the management device 130. The processor 1320 can be a very large scale integrated circuit. An operating system and other software programs are installed in the processor 1320, so that the processor 1320 can access the memory and various PCIe devices. The processor 1320 includes one or more processor cores. The processor core in the processor 1320 is, for example, a central processing unit (CPU) or other application specific integrated circuit (ASIC). The processor 1320 can also be other general-purpose processors, digital signal processors (DSP), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. In actual applications, the management device 130 may also include multiple processors.

[0080] The memory 1310 may correspond to a storage medium for storing information such as execution information. The memory 1310 may be a volatile memory or a nonvolatile memory, or may include both volatile and nonvolatile memories. Among them, the nonvolatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct rambus RAM (DR RAM).

[0081] Memory 1310 can be used to store execution information of the model to be trained by the distributed training system, such as the execution information used to indicate the execution timing of each operator in the model, the amount of data of the gradient corresponding to the operator, and the computing device to which the operator belongs.

[0082] Taking server 111 as an example, Figure 1B The hardware implementation of the server in is exemplified, and the server 111 includes a network card 111A, a computing device 1 and a computing device 2. The network card and the computing device, and different computing devices can communicate with each other through a PCIe bus, an EISA bus, UB, CXL or CCIX, etc.

[0083] It is worth noting that the data transmission bandwidth between different computing devices can be the same or different. The data transmission bandwidth here refers to the maximum value of data transmitted per unit time, or the data transmission bandwidth can also be called the maximum amount of communication that can be transmitted per unit time. When the amount of communication remains unchanged, the larger the data transmission bandwidth, the shorter the time (or communication delay) required for data exchange or communication between computing devices executing tasks (or operators).

[0084] Combination Figure 1B The computing devices shown and Table 1 below illustrate the data transmission bandwidth between multiple computing devices. The unit of the data transmission bandwidth is: million bytes per second (MB / s).

[0085] Table 1

[0086] Unit: MB / s Computing device 1 Computing device 2 Computing device 112B Computing device 112C Computing device 1 - 1000 500 500 Computing device 2 - - 500 500 Computing device 112B - - - 1000 Computing device 112C - - - -

[0087] Depending on the server to which the computing device belongs, the communication of the distributed training system includes communication between devices within the server (local communication) and communication between multiple servers (cross-server communication). In addition to the data communication within the server to be transmitted to the network card, cross-server communication also requires the coordination between the network cards and switches corresponding to multiple servers, resulting in a greater communication overhead for cross-server communication than for local communication. As the cluster size and the number of servers continue to expand, cross-server communication will become a bottleneck for high-performance cluster computing tasks.

[0088] With reference to Table 1, when different computing devices are located on the same server, the communication mode between the computing devices is local communication: for example, the data transmission bandwidth between computing device 1 and computing device 2 is 1000MB / s, and the data transmission bandwidth between computing device 112B and computing device 112C is 1000MB / s.

[0089] When different computing devices are located on different servers, the communication mode between the computing devices is cross-server communication: for example, the data transmission bandwidth between computing device 1 and computing device 112B, computing device 112C is 500MB / s; the data transmission bandwidth between computing device 2 and computing device 112B, computing device 112C is also 500MB / s.

[0090] The above computing device may be a processor, such as a processor including one or more processor cores. The processor core in the processor is, for example, a CPU or an ASIC. The processor may also be other general-purpose processors, graphics processing units (GPUs), neural network processing units (NPUs), DSPs, FPGAs or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. In practical applications, the computing device may also include multiple processors.

[0091] The above hardware implementation of the server is explained using server 111 as an example. Server 112 and server 113 may also include the hardware required to perform model training. For example, server 112 includes network card 112A, computing device 112B and computing device 112C, and server 113 includes network card 113A, computing device 113B and computing device 113C.

[0092] Figure 1B Only the operators deployed in server 111 (operator 1 to operator m) are shown, but servers 112 and 113 may also be deployed with operators in the model, thereby enabling the distributed training system 100 to train the model.

[0093] It is worth noting that Figure 1B The management device 130 and the server are different physical devices, but in some optional implementations, the functions of the management device 130 and the server can also be integrated into one physical device. For example, a designated computing device in the server can be used to implement the functions of the management device 130 in the distributed training system 100, and this application is not limited to this.

[0094] In the above embodiments, the process of training the model by the distributed training system 100 is implemented on multiple servers, but in some optional situations, the process of training the model by the distributed training system 100 can be implemented on a single server, such as server 111. A computing device is allocated in the server 111 to implement the functions of the management device 130, and other computing devices are used to execute tasks of different operators in the model.

[0095] Below Figure 1A and Figure 1B Based on this, this embodiment provides a model training architecture, such as Figure 2A As shown, Figure 2A A software architecture diagram of a model training provided in an embodiment of the present application. The software architecture 200 includes: a communication performance predictor 210, a model gradient information parser 220, a model operation information parser 230 and a gradient fusion transmission strategy optimizer 240.

[0096] The communication performance predictor 210 is used to predict the training environment (such as Figure 1B The distributed training system 100) performs communication modeling or communication simulation to obtain the data transmission performance of the computing device in the training environment. Exemplarily, the communication performance predictor 210 constructs a performance predictor for the communication task, inputs the communication volume (amount of data transmitted) of each communication, and outputs the communication delay.

[0097] For example, the communication performance predictor 210 may obtain the data transmission performance of multiple computing devices in the following manner, including: the communication performance predictor 210 performs communication processing on multiple computing devices, obtains communication profiles of the multiple computing devices, and obtains the data transmission performance of the multiple computing devices based on the communication profiles. The communication profiles here are used to indicate the communication delay of each computing device transmitting data of different communication volumes. It is worth noting that the communication processing here includes at least one of communication modeling or communication simulation.

[0098] like Figure 2B As shown, Figure 2B A schematic diagram of data transmission performance of multi-card training provided in an embodiment of the present application. Multi-card refers to multiple computing devices, one card is one computing device, and in physical form, the card may refer to an accelerator card for training models or other hardware cards with model training functions. Taking 8 cards as an example, the AllReduce algorithm is used between different cards to perform communication operator fusion to transmit gradients or other data between different cards. Figure 2B (1) shows the communication delay of these 8 cards with different communication volumes. As the communication volume increases, the communication delay of these 8 cards increases. Figure 2B (2) shows the data transmission bandwidth provided by these 8 cards for data transmission under different communication volumes. As the communication volume increases, the data transmission bandwidth provided by these 8 cards for data transmission increases first and then decreases. That is, when the communication volume gradually increases, the bandwidth utilization rate first increases and then decreases.

[0099] In this embodiment, the communication performance predictor 210 can use the acquired data transmission performance to optimize the gradient transmission process of the model to obtain a gradient transmission strategy that maximizes the utilization of the data transmission performance and shortens the gradient tailing time as much as possible, thereby improving the training efficiency of the model.

[0100] In an optional manner, the data transmission performance may be represented by data transmission bandwidth and other indicators, such as the other indicators referring to the inherent delay of the computing device in performing the communication task or the delay associated with the communication task type, etc. For the description of the data transmission bandwidth, please refer to the relevant content of Table 1 above, which will not be elaborated here.

[0101] The model gradient information parser 220 is used to obtain information such as gradient size distribution information (such as gradient generation time and data volume of each gradient) in the trained model, dependency relationship (or call relationship, etc.) of operators that generate gradients, operators that generate gradients, etc. One gradient corresponds to one operator. It is worth noting that in some cases, some operators in the model have no model parameters, and these operators are only used to transfer data, etc., then these operators have no corresponding gradients.

[0102] The model operation information parser 230 is used to obtain the timing information of the computing tasks of different operators during the model training, such as the execution order of the operators, the execution duration and start and end time of each operator (such as the transmission start time and the transmission end time) and other information.

[0103] The gradient fusion transmission strategy optimizer 240 is used to construct an objective function for communication operator fusion and solve the objective function according to a preset optimization condition, thereby realizing the optimization of the gradient transmission strategy during the model training process. For possible implementation methods of the objective function, please refer to the relevant contents of the following formulas 1 to 6, which will not be repeated here.

[0104] Each software included in the above software architecture 200 can be provided with corresponding operating resources (such as computing resources, storage resources or network resources, etc.) by the computing device included in the server. It is worth noting that Figure 2A The possible implementation methods for implementing model training provided in this embodiment are only for this purpose and should not be construed as limiting the present application. Depending on the hardware resources included in the server, the above software can be split into more or fewer modules (or units), etc., and the present application does not limit this.

[0105] To solve the problems raised by the background technology, the following Figure 1A , Figure 1B and Figure 2A Based on the architecture shown, the present application embodiment provides a model training method. Figure 3 As shown, Figure 3 A flow chart of a model training method provided in an embodiment of the present application is shown in FIG1. ​​The model training method can be applied to Figure 1A The server 110 or data center shown, or Figure 1B The distributed training system 100 or any server is shown.

[0106] The model training method is executed by a management device, such as Figure 1B The management device 130 in the embodiment of the present application may be a computing device in any server for implementing the functions of the management device 130. The model training method provided in the embodiment of the present application includes the following steps S310 to S330.

[0107] S310. The management device obtains the execution information of the model and the data transmission performance of multiple computing devices in response to a training request for the model.

[0108] In a possible example, the training request is an instruction from a user received by the management device.

[0109] In another possible example, the training request is an instruction for the model generated by the management device based on the training status of multiple models of the distributed training system after the training time of the model in S310 is reached.

[0110] Optionally, the management device may also obtain execution information and data transmission performance, etc. during the model training process, and can start the subsequent gradient transmission optimization process and model training process without responding to the training request.

[0111] In S310, the execution information includes: the execution order of multiple operators, the execution duration and the data volume of multiple gradients, where one gradient corresponds to one operator.

[0112] by Figure 1B Take the model in as an example, the model includes operators 1 to m, and the sequence number of each operator is the execution order of the operator. Table 2 below shows the execution time of different operators and the data volume information of the gradient corresponding to the operator.

[0113] Table 2

[0114]

[0115] Among them, operator 3 is an operator used to transfer the intermediate results in the model (that is, operator 3 does not have model parameters), so in the back propagation processing of the model, operator 3 does not have a gradient.

[0116] It is worth noting that the contents shown in Table 2 above are merely examples provided in this embodiment and should not be construed as limiting the present application.

[0117] Regarding the method of obtaining execution information, this embodiment provides a possible implementation method, such as Figure 4 As shown, Figure 4 Schematic diagram 2 of a flow chart of a model training method provided in an embodiment of the present application. The above S310 may include steps S311 to S313.

[0118] S311. The management device instructs multiple computing devices to perform at least one round of training on the model to obtain an intermediate representation (IR) graph and performance data of the model.

[0119] A round of training means that all computing devices fully utilize the training data that each computing device can call to run the model and obtain the output of the model. When the output of the model is compared with the data label of the training data, the loss value of the model can be determined, and the size of the loss value can be used to determine the training direction of the model (i.e., gradient).

[0120] The IR graph of the model is used to indicate: establish the control flow in the model and the dependencies between operators and training data, and provide an interface for graph-level optimization. The IR graph can also contain rich semantic information for compilation and provide extensible editing semantics for custom operators.

[0121] The performance data is used to indicate the execution order and execution duration of multiple operators. For the contents of the execution order and execution duration, please refer to the relevant contents of Table 2, which will not be repeated here.

[0122] In a possible example, the performance data may be the runtime information of the model obtained by the management device by parsing the profiling trace file during model training, thereby obtaining the execution order, start and end time, etc. of the computing tasks corresponding to each gradient. Figure 5 As shown, Figure 5 A schematic diagram of the time distribution information of a ResNet18 model provided in an embodiment of the present application. Figure 5 The horizontal axis in is the execution order of operators, and the vertical axis is the time interval distribution of two adjacent gradient generation moments.

[0123] S312: The management device obtains the data amount and generation order of each gradient in the model according to the IR graph.

[0124] Exemplarily, for multiple operators in the model, the management device constructs a model gradient information parser (such as Figure 2B The model gradient information parser in is used to obtain the gradient distribution information of the model. Figure 6 As shown, Figure 6 A schematic diagram of the gradient distribution information of a ResNet18 model provided in an embodiment of the present application. For different operators in the ResNet18 model, the amount of data corresponding to the gradient of the operator is different, and some operators do not have model parameters, so these operators do not have gradients.

[0125] S313: The management device obtains execution information according to the performance data and the data volume and generation order of each gradient in the model.

[0126] Compared with training the model on a separate computing device, the management device distributes multiple operators of the model to be trained to different computing devices, which is conducive to improving the training efficiency of the model. In addition, the management device obtains the information obtained by the model after at least one round of training (such as the gradient and execution order of the operator), and obtains the execution information required for the optimization strategy based on the information, which is conducive to the management device to perform training optimization according to the actual operation of the model on multiple computing devices, thereby improving the accuracy and training effect of model training.

[0127] For more information about how to obtain data transmission performance and its specific contents, please refer to Figure 2A The description related to Table 1 is not repeated here.

[0128] Please continue to refer to Figure 3 , the model training method provided in this embodiment also includes the following steps S320 and S330.

[0129] S320. The management device determines a gradient transmission strategy of the model according to the execution information and the data transmission performance of the plurality of computing devices.

[0130] The gradient transmission strategy is used to indicate the way in which each computing device performs gradient fusion transmission when training the model.

[0131] In an optional implementation, the gradient transmission strategy causes the gradient in any computing device to be fused by the communication operator at least once, and the transmission time of the fused data of the k gradients generated in the latter order in any computing device is the shortest, where k is a positive integer.

[0132] If the execution order of the operator in the model is first, then the gradient generation order of the operator is later. In the aforementioned gradient transmission strategy, the data transmission time of the k operators whose gradient generation order is later (that is, the execution order of the operator is first) is the shortest.

[0133] Therefore, when the management device applies the gradient transmission strategy to the model training process, the data transmission time of the gradient data last transmitted in each computing device is the shortest, that is, the gradient tailing time is the shortest, which is beneficial to reducing the model training time and improving the model training efficiency.

[0134] For possible implementation methods of the management device to determine the gradient transmission strategy, please refer to the following Figure 7 , Fig. 8A and Figure 8B The description is not repeated here.

[0135] S330: The management device applies the gradient transmission strategy to the training of the model.

[0136] In one possible implementation, after the management device applies the gradient transmission strategy to the training of the model, the model training method provided in this embodiment also includes: the management device obtains the trained model; and if the trained model meets the preset model convergence conditions, the management device outputs the trained model.

[0137] In a first possible example, the way in which the management device outputs the model includes: the management device deploys the model to a user end (or terminal).

[0138] In a second possible example, the way in which the management device outputs the model includes: the management device stores relevant information of the model (such as model parameters and deployment methods, etc.) to a designated storage device, and the storage device provides a user interface. The user end (or terminal) can call the trained model stored in the storage device through the user interface to implement the corresponding functions of the model.

[0139] Compared with the management device transmitting the gradient generated by the computing device according to a fixed capacity, in this embodiment, the management device can obtain a gradient transmission strategy adapted to the data transmission performance of multiple computing devices according to the execution information of the model. When the gradient transmission strategy is applied to the training process of the model, the data flow of the gradient transmitted by each computing device each time can match the data transmission performance of the computing device. Therefore, the data transmission performance of the computing device will not cause the problem of increased communication overhead due to the small amount of gradient data transmitted each time.

[0140] Furthermore, when the data transmission performance of the computing device is better, the gradient transmission strategy can increase the amount of gradient data transmitted each time to improve the distributed training system's ability to utilize the data transmission performance. Moreover, since the gradient transmission strategy effectively utilizes the data transmission performance of the computing device, the last set of data to be transmitted by each computing device can also be adapted to the data transmission performance of the computing device, which is conducive to reducing the gradient tailing delay of each communication between the computing device and the management device, thereby reducing the training time of the model and improving the training efficiency of the model.

[0141] With respect to the implementation of the above S320, this embodiment provides a possible example, such as Figure 7 As shown, Figure 7 A schematic diagram of a model training method provided in an embodiment of the present application Figure 3 The process of determining the gradient transmission strategy for the management device may include the following steps S710 and S720.

[0142] S710: The management device predicts one or more candidate gradient transmission strategies that meet the first optimization condition based on the execution information and the data transmission performance of the plurality of computing devices.

[0143] The first optimization condition is used to indicate that the transmission time of the fusion data of the k gradients generated in the latter order in the computing device is the shortest, where k is a positive integer. In other words, the first optimization condition is used to minimize the gradient tailing time of each computing device.

[0144] Exemplarily, the management device constructs an objective function according to the execution information and the data transmission performance of the plurality of computing devices, and solves the objective function according to the first optimization condition to obtain one or more candidate gradient transmission strategies.

[0145] In an optional implementation, the above objective function includes: a first function, a second function, a third function, a fourth function and a fifth function.

[0146] The first function is to minimize the sum of the gradient generation time and the transmission time of each gradient.

[0147] The second function is used to select the maximum value of the transmission end time of the first gradient and the communication generation time of the second gradient. The generation order of the first gradient and the generation order of the second gradient are adjacent, and the generation order of the first gradient is before the generation order of the second gradient.

[0148] The third function is used to determine the transmission time of each gradient according to the data volume of each gradient and the data transmission performance of the computing device where each gradient is located.

[0149] The fourth function is used to determine the sum of the communication generation time and the transmission duration of the first gradient as the transmission end time of the first gradient.

[0150] The fifth function is used to determine that the sum of the data amounts of multiple communications is the gradient data amount of all gradients of the model.

[0151] The above functions can be realized by formula:

[0152]

[0153] In the above formulas 1 to 5, the meaning of each symbol is shown in Table 3 below.

[0154] Table 3

[0155] symbol Meaning of symbols M The total amount of data for all gradients in the model n The number of communication operators executed by the model for one round of training ET The time when each gradient is generated by the calculation task (the time when the gradient is generated) ct Data transmission time for each gradient et The maximum value between the end time of each gradient transmission and the generation time of the next gradient pre_et The transmission end time of the previous gradient TS The amount of data in the gradient for each data communication

[0156] Here, n can also be called the number of data transmissions required for one round of model training, that is, one communication operator corresponds to one data communication.

[0157] Formula 1 is used to implement the function of the first function, Formula 2 is used to implement the function of the second function, Formula 3 is used to implement the function of the third function, Formula 4 is used to implement the function of the fourth function, and Formula 5 is used to implement the function of the fifth function.

[0158] It is worth noting that the above objective function and each formula are only examples provided in this embodiment for minimizing the gradient tailing time of each computing device in the gradient transmission strategy, and should not be understood as limiting the present application. When those skilled in the art use other functions or formulas to implement the model training method provided in this application, they also fall within the scope of protection of this application.

[0159] In this embodiment, the aforementioned gradient transmission strategy makes the data transmission time of the k operators whose gradient generation order is later (that is, the execution order of the operators is earlier) the shortest. Therefore, when the management device applies the gradient transmission strategy to the model training process, the data transmission time of the gradient data transmitted last in each computing device is the shortest, that is, the gradient tailing time is the shortest, which is conducive to reducing the training time of the model and improving the training efficiency of the model.

[0160] S720: The management device determines a gradient transmission strategy that meets the second optimization condition from one or more candidate gradient transmission strategies.

[0161] The second optimization condition is used to indicate that the number of transmissions of the gradient of the distributed training system is minimal.

[0162] For example, the second optimization condition can be implemented by the following formula 6. The meaning of each symbol in formula 6 can refer to the relevant content in Table 3.

[0163] θ * =argmin(n),θ([i 1 ,i 1 ,…i n ]) Formula 6

[0164] The data communication process between different computing devices inevitably has a gradient tailing time. The optimization condition adopted by the management device in this embodiment can be used to reduce the number of data transmissions indicated by the training strategy. When the gradient tailing time of each communication remains unchanged or the difference is not large, the fewer the number of data transmissions, the smaller the total gradient tailing time required for model training, which reduces the gradient transmission time between different computing devices, thereby reducing the training time of the model and improving the training efficiency of the model.

[0165] For Figure 7 The present embodiment also provides a specific example of the optimization method provided. Fig. 8A As shown, Fig. 8A Schematic diagram 1 of the optimization of the gradient transmission strategy provided in the embodiment of the present application. For gradient 1 and gradient 2, the management device predicts that the start time of the next gradient is ET3(et2), and the end time of the transmission after the fusion of gradient 1 and gradient 2 is ET(fusion1+2)<ET3(et2), then the management device confirms that gradient 1 and gradient 2 can be fused during the gradient transmission process.

[0166] For gradient 1 and gradient 2, the management device predicts that the gradient transmission end time is et4, and the transmission end time after the fusion of gradient 3 and gradient 4 is ET(fusion3+4)<et4, then the management device confirms that gradient 3 and gradient 4 cannot be fused during the gradient transmission process.

[0167] As an optional implementation manner, the gradient transmission strategy provided in this embodiment may include two parts.

[0168] Part 1: In the training process of the model, one or more groups of gradients to be fused in any computing device. One group of gradients includes at least two gradients generated in sequence, and the at least two gradients correspond to at least two adjacent operators in a plurality of operators.

[0169] The second part: A set of gradients is transmitted in a communication process, and the transmission end time of the set of gradients is less than or equal to the transmission end time of the second gradient. The second gradient is the last gradient generated before the at least two gradients are fused by the pass operator.

[0170] For any computing device among the multiple computing devices, if the computing device has m operators deployed in it, the communication fusion mode of the m operators can be various. Exemplarily, the management device determines the gradient transmission strategy according to the time distribution information of the gradient and the data transmission performance. The gradient transmission strategy makes the transmission end time of the gradient after fusion before the end time of the gradient before fusion, which is conducive to reducing the gradient transmission time between different devices, thereby reducing the training time of the model and improving the training efficiency of the model.

[0171] For example, this embodiment also provides another specific example, such as Figure 8B As shown, Figure 8B Schematic diagram 2 of the optimization of the gradient transmission strategy provided in the embodiment of the present application. The calculation flow includes operators 1 to m, and the training optimization process.

[0172] These m operators generate a total of 62 gradients. After the management device analyzes the model, the gradient transmission strategy obtained includes the following four parts.

[0173] ①: Fuse gradients 1 to 13 (13 gradients are fused), and the corresponding data volume is 28MB.

[0174] ②: Fuse gradients 14 to 52 (39 gradients), and the corresponding data volume is 15MB.

[0175] ③: Fuse gradients 53 to 61 (9 gradients are fused), and the corresponding data volume is 0.4MB.

[0176] ④: Transmit gradient 62 separately, and the corresponding data volume is 0.2MB.

[0177] In the case where the gradient transmission strategy includes the above content, the process of training the model according to the gradient transmission strategy by the management device ( Figure 3 S330 shown), this embodiment provides a possible specific example, such as Fig. 9 As shown, Fig. 9 A schematic diagram of a model training method provided in an embodiment of the present application Figure 4 The above S330 may include steps S910 to S940.

[0178] S910: The management device synchronizes the gradient transmission strategy to multiple computing devices.

[0179] In a first possible scenario, the management device sends the gradient transmission strategy to multiple computing devices, and each computing device deploys the gradient transmission strategy respectively.

[0180] In the second possible scenario, the management device writes the gradient transmission strategy into the configuration unit of the distributed training system, which can be used to manage all configuration information during the model training process. Before the computing device runs the model, it reads the above gradient transmission strategy from the configuration unit and executes the corresponding communication operator fusion (AllReduce, or gradient data fusion).

[0181] The above two possible situations are only possible implementations provided in this example and should not be understood as limitations of the synchronization method of the gradient transmission strategy in this application.

[0182] S920: The computing device runs the model according to the synchronized gradient transmission strategy and sends the acquired gradient to the management device.

[0183] The gradient obtained in S920 includes the gradient (fused data or fused gradient) after the computing device fuses the communication operators according to the gradient transmission strategy.

[0184] Correspondingly, the management device receives all gradients sent by multiple computing devices after running the model, and all gradients include gradients after communication operators are fused according to the gradient transmission strategy.

[0185] S930: The management device processes all gradients to obtain parameter update information of the model.

[0186] Exemplarily, the processing in S930 includes at least one of weighting or averaging.

[0187] For example, for a single operator in the model, the management device averages the gradients associated with the single operator in all gradients, obtains the gradient mean (parameter update information), and then uses the gradient mean to update the parameters of the single operator.

[0188] For another example, different computing devices have different weights. For a single operator in the model, the management device weights the gradients associated with the single operator in all gradients with the weight of the computing device that generates the gradient, and after obtaining the gradient weighted value, uses the gradient weighted value (parameter update information) to update the parameters of the single operator.

[0189] S940: The management device updates the parameters of the model according to the parameter update information.

[0190] Exemplarily, for a single operator in the model, the management device adds the parameter update information to the original model parameters of the single operator to obtain the updated model parameters of the single operator. After the management device updates the model parameters of all operators, the trained model is obtained.

[0191] It should be noted that the above embodiments are only examples of the model training method provided in this application. The model training method provided in this application can be applied to different types of distributed training systems such as the MindSpore training framework or the Pytorch training framework. The specific type of the distributed training system in this application is not limited.

[0192] It is understandable that in order to implement the functions in the above embodiments, the management device and the computing device include hardware structures and / or software modules corresponding to the execution of each function. Those skilled in the art should easily realize that, in combination with the units and method steps of each example described in the embodiments disclosed in this application, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application scenario and design constraints of the technical solution.

[0193] Combined with the above Figures 1A to 9 , describes in detail the method provided according to this embodiment, and will be combined with Fig.10 , describing the model training device provided according to this embodiment.

[0194] Fig.10 The following is a schematic diagram of the structure of a model training device provided in an embodiment of the present application. The model training device can be used to implement the functions of the management device in the above method embodiment, and thus can also achieve the beneficial effects of the above method embodiment. In this embodiment, the device can be as follows Figure 1A The server 110 or data center shown may also be Figure 1A The computing device included in the distributed training system, the management device 130 or any server shown may also be any module (such as a chip) applied to the aforementioned device or system.

[0195] like Fig.10As shown, the model training device 1000 includes: an acquisition module 1010, a gradient transmission optimization module 1020 and a training module 1030. The model training device 1000 is used to implement the function of the management device in the above method embodiment.

[0196] Exemplarily, the acquisition module 1010 is used to obtain the execution information of the model and the data transmission performance of multiple computing devices; the execution information includes: the execution order of multiple operators, the execution time and the data volume of multiple gradients, and one gradient corresponds to one operator. The gradient transmission optimization module 1020 is used to determine the gradient transmission strategy of the model based on the execution information and the data transmission performance of multiple computing devices. The gradient transmission strategy is used to indicate the way in which each computing device performs gradient fusion transmission when training the model. The training module 1030 is used to apply the gradient transmission strategy to the training of the model.

[0197] In an optional implementation, the gradient transmission optimization module 1020 is specifically used to: predict one or more candidate gradient transmission strategies that meet the first optimization condition based on the execution information and the data transmission performance of multiple computing devices. And, determine the gradient transmission strategy that meets the second optimization condition from one or more candidate gradient transmission strategies. Among them, the first optimization condition is used to indicate that the transmission time of the fusion data of the k gradients generated in the computing device in the latter order is the shortest, and k is a positive integer. The second optimization condition is used to indicate that the number of transmissions of the gradient of the distributed training system is the least.

[0198] In an optional implementation, the gradient transmission optimization module 1020 is specifically used to: construct an objective function according to the execution information and the data transmission performance of multiple computing devices, and solve the objective function according to the first optimization condition to obtain one or more candidate gradient transmission strategies.

[0199] In an optional implementation, the objective function includes: a first function, a second function, a third function, a fourth function and a fifth function. The first function is used to minimize the sum of the generation time and the transmission time of each gradient. The second function is used to select the maximum value of the transmission end time of the first gradient and the generation time of the second gradient; the generation order of the first gradient and the generation order of the second gradient are adjacent, and the generation order of the first gradient is before the generation order of the second gradient. The third function is used to determine the transmission time of each gradient according to the data volume of each gradient and the data transmission performance of the computing device where each gradient is located. The fourth function is used to determine that the sum of the generation time and the communication time of the first gradient is the transmission end time of the first gradient. The fifth function is used to determine that the sum of the data volume of multiple communications is the data volume of all gradients of the model.

[0200] In an optional implementation, the acquisition module 1010 is specifically used to: perform communication processing on multiple computing devices to obtain communication profiles of the multiple computing devices. And, based on the communication profiles of the multiple computing devices, obtain the data transmission performance of the multiple computing devices. The communication processing includes at least one of communication modeling or communication simulation, and the communication profile is used to indicate the communication delay of each computing device transmitting data of different communication volumes.

[0201] In an optional implementation, the acquisition module 1010 is specifically used to: instruct multiple computing devices to perform at least one round of training on the model, and obtain the IR graph and performance data of the model. The performance data is used to indicate the execution order and execution duration of multiple operators. The acquisition module 1010 is also specifically used to: obtain the data volume and generation order of each gradient in the model according to the IR graph. And, according to the performance data and the data volume and generation order of each gradient in the model, obtain execution information.

[0202] In an optional implementation, the gradient transmission strategy includes: during the training process of the model, one or more groups of gradients to be fused in any computing device, a group of gradients includes at least two gradients that are adjacent in generation order, and the at least two gradients correspond to at least two adjacent operators among multiple operators. Also, a group of gradients is transmitted in one communication process, and the transmission end time of a group of gradients is less than or equal to the transmission end time of a second gradient, and the second gradient is the last gradient generated before the at least two gradients are fused by a passing operator.

[0203] In an optional implementation, the training module 1030 is specifically used to: synchronize the gradient transmission strategy to multiple computing devices. And receive all gradients sent by multiple computing devices after running the model, all gradients include gradients after communication operator fusion according to the gradient transmission strategy. The training module 1030 is also specifically used to: process all gradients to obtain parameter update information of the model. And, update the parameters of the model according to the parameter update information. The processing process includes at least one of weighting or averaging.

[0204] It should be understood that the model training device 1000 of the embodiment of the present application can be implemented by a DPU. The model training device 1000 according to the embodiment of the present application can correspond to the method described in the embodiment of the present application, and the above and other operations and / or functions of each unit in the model training device 1000 are respectively for implementing the corresponding processes of each method, and for the sake of brevity, they are not repeated here.

[0205] The method steps in this embodiment can be implemented by hardware or by a processor executing software instructions. The software instructions can be composed of corresponding software modules, which can be stored in random access memory (RAM), flash memory, read-only memory (ROM), programmable ROM (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, mobile hard disks, CD-ROMs, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in an ASIC. In addition, the ASIC can be located in a computing device. Of course, the processor and the storage medium can also exist as discrete components in a network device or a terminal device.

[0206] The present application also provides a chip system, which includes a processor for implementing the functions of the processing device in the above method. In one possible design, the chip system also includes a memory for storing program instructions and / or data. The chip system can be composed of a chip, or can include a chip and other discrete devices.

[0207] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer programs or instructions. When the computer program or instruction is loaded and executed on a computer, the process or function described in the embodiment of the present application is executed in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, a network device, a user device or other programmable device. The computer program or instruction may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer program or instruction may be transmitted from one website site, computer, server or data center to another website site, computer, server or data center by wired or wireless means. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium may be a magnetic medium, for example, a floppy disk, a hard disk, a tape; it may also be an optical medium, for example, a digital video disc (DVD); it may also be a semiconductor medium, for example, a solid state drive (SSD).

[0208] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present application, and these modifications or replacements should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A model training method, characterized in that: The method is performed by a management device, the management device is located in a distributed training system for training a model, the distributed training system also includes multiple computing devices, and the method includes: Acquire execution information of the model and data transmission performance of the multiple computing devices; the execution information includes: execution order, execution duration and data volume of multiple gradients of the multiple operators of the model, where one gradient corresponds to one operator; Determine, according to the execution information and the data transmission performance of the plurality of computing devices, a gradient transmission strategy of the model, wherein the gradient transmission strategy is used to indicate a manner in which each computing device performs gradient fusion transmission when training the model; The gradient transfer strategy is applied to the training of the model.

2. The method according to claim 1, characterized in that Determining the gradient transmission strategy of the model according to the execution information and the data transmission performance of the plurality of computing devices includes: Predicting one or more candidate gradient transmission strategies that meet a first optimization condition according to the execution information and the data transmission performance of the plurality of computing devices; the first optimization condition is used to indicate that the transmission time of fused data of k gradients generated in a later order in the computing device is the shortest, where k is a positive integer; Determine a gradient transmission strategy that meets a second optimization condition from the one or more candidate gradient transmission strategies; the second optimization condition is used to indicate that the number of transmissions of the gradient of the distributed training system is the least.

3. The method according to claim 2, characterized in that Predicting one or more candidate gradient transmission strategies that meet the first optimization condition according to the execution information and the data transmission performance of the multiple computing devices, including: constructing an objective function based on the execution information and the data transfer performance of the plurality of computing devices; The objective function is solved according to the first optimization condition to obtain the one or more candidate gradient transmission strategies.

4. The method according to claim 3, characterized in that The objective function includes: a first function, a second function, a third function, a fourth function and a fifth function; The first function is used to: minimize the sum of the generation time and transmission time of each gradient; The second function is used to: select the maximum value of the transmission end time of the first gradient and the generation time of the second gradient; the generation order of the first gradient is adjacent to the generation order of the second gradient, and the generation order of the first gradient is before the generation order of the second gradient; The third function is used to: determine the transmission time of each gradient according to the data volume of each gradient and the data transmission performance of the computing device where each gradient is located; The fourth function is used to: determine the sum of the generation time and the communication time of the first gradient as the transmission end time of the first gradient; The fifth function is used to determine that the sum of the data amounts of multiple communications is the data amount of all gradients of the model.

5. The method according to any one of claims 1 to 4, characterized in that Acquiring data transmission performance of the plurality of computing devices, including: Performing communication processing on the multiple computing devices to obtain communication profiles of the multiple computing devices; the communication processing includes at least one of communication modeling or communication simulation, and the communication profile is used to indicate the communication delay of each computing device in transmitting data of different communication volumes; According to the communication profiles of the multiple computing devices, the data transmission performance of the multiple computing devices is obtained.

6. The method according to any one of claims 1 to 5, characterized in that Get the execution information of the model, including: Instructing the multiple computing devices to perform at least one round of training on the model to obtain an intermediate representation IR graph and performance data of the model; the performance data is used to indicate the execution order and execution duration of the multiple operators; Acquire the data amount and generation order of each gradient in the model according to the IR graph; The execution information is obtained according to the performance data and the data amount and generation order of each gradient in the model.

7. The method according to any one of claims 1 to 6, characterized in that The gradient transmission strategy includes: during the training process of the model, one or more groups of gradients to be fused in any computing device, the group of gradients includes at least two gradients that are generated in a sequential order, and the at least two gradients correspond to at least two adjacent operators among the multiple operators; and the group of gradients is transmitted in one communication process, and the transmission end time of the group of gradients is less than or equal to the transmission end time of the second gradient, and the second gradient is the last gradient generated before the at least two gradients are fused by a passing operator.

8. The method according to claim 7, characterized in that The applying the gradient transmission strategy to the training of the model includes: synchronizing the gradient transmission strategy to the plurality of computing devices; receiving all gradients sent by the plurality of computing devices after running the model, wherein all gradients include gradients after communication operator fusion according to the gradient transmission strategy; Processing all the gradients to obtain parameter update information of the model; the processing includes at least one of weighting or averaging; The parameters of the model are updated according to the parameter update information.

9. A model training device, characterized in that: A distributed training system for training a model, wherein the distributed training system further comprises a plurality of computing devices; The device comprises: An acquisition module, used to acquire execution information of the model and data transmission performance of the multiple computing devices; the execution information includes: execution order, execution duration and data volume of multiple gradients of the multiple operators of the model, where one gradient corresponds to one operator; A gradient transmission optimization module, used to determine the gradient transmission strategy of the model according to the execution information and the data transmission performance of the multiple computing devices, wherein the gradient transmission strategy is used to indicate the way in which each computing device performs gradient fusion transmission when training the model; A training module is used to apply the gradient transmission strategy to the training of the model.

10. The device according to claim 9, characterized in that The gradient transmission optimization module is specifically used to: predict one or more candidate gradient transmission strategies that meet the first optimization condition according to the execution information and the data transmission performance of the multiple computing devices; and determine a gradient transmission strategy that meets the second optimization condition from the one or more candidate gradient transmission strategies; Among them, the first optimization condition is used to indicate that the transmission time of the fusion data of the k gradients generated in the computing device in the latter order is the shortest, and k is a positive integer; the second optimization condition is used to indicate that the number of transmission times of the gradients of the distributed training system is the least.

11. The device according to claim 10, characterized in that The gradient transmission optimization module is specifically used to: construct an objective function according to the execution information and the data transmission performance of the multiple computing devices; and solve the objective function according to the first optimization condition to obtain the one or more candidate gradient transmission strategies.

12. The device according to claim 11, characterized in that The objective function includes: a first function, a second function, a third function, a fourth function and a fifth function; The first function is used to: minimize the sum of the generation time and transmission time of each gradient; The second function is used to: select the maximum value of the transmission end time of the first gradient and the generation time of the second gradient; the generation order of the first gradient is adjacent to the generation order of the second gradient, and the generation order of the first gradient is before the generation order of the second gradient; The third function is used to: determine the transmission time of each gradient according to the data volume of each gradient and the data transmission performance of the computing device where each gradient is located; The fourth function is used to: determine the sum of the generation time and the communication time of the first gradient as the transmission end time of the first gradient; The fifth function is used to determine that the sum of the data amounts of multiple communications is the data amount of all gradients of the model.

13. The device according to any one of claims 9 to 12, characterized in that The acquisition module is specifically used to: perform communication processing on the multiple computing devices to obtain communication portraits of the multiple computing devices; and, based on the communication portraits of the multiple computing devices, obtain the data transmission performance of the multiple computing devices; the communication processing includes at least one of communication modeling or communication simulation, and the communication portrait is used to indicate the communication delay of each computing device transmitting data of different communication volumes.

14. The device according to any one of claims 9 to 13, characterized in that The acquisition module is specifically used for: Instructing the multiple computing devices to perform at least one round of training on the model to obtain an intermediate representation IR graph and performance data of the model; the performance data is used to indicate the execution order and execution duration of the multiple operators; The acquisition module is also specifically used to: acquire the data amount and generation order of each gradient in the model according to the IR graph; and acquire the execution information according to the performance data and the data amount and generation order of each gradient in the model.

15. The device according to any one of claims 9 to 14, characterized in that The gradient transmission strategy includes: during the training process of the model, one or more groups of gradients to be fused in any computing device, the group of gradients includes at least two gradients that are generated in a sequential order, and the at least two gradients correspond to at least two adjacent operators among the multiple operators; and the group of gradients is transmitted in one communication process, and the transmission end time of the group of gradients is less than or equal to the transmission end time of the second gradient, and the second gradient is the last gradient generated before the at least two gradients are fused by a passing operator.

16. The device according to claim 15, characterized in that The training module is specifically used to: synchronize the gradient transmission strategy to the multiple computing devices; and receive all gradients sent by the multiple computing devices after running the model, wherein all gradients include gradients after communication operator fusion according to the gradient transmission strategy; The training module is also specifically used to: process all the gradients to obtain parameter update information of the model; and update the parameters of the model according to the parameter update information; the processing process includes at least one of weighting or averaging.

17. An electronic device, characterized in that: include: Memory and processor; The memory is used to store a set of computer instructions; The processor is configured to call the set of computer instructions to implement the method according to any one of claims 1 to 8.

18. A distributed training system, characterized in that: include: A management device and multiple computing devices, each computing device is deployed with a model to be trained, wherein the model includes multiple operators; The management device is used to obtain the execution information of the model and the data transmission performance of the multiple computing devices; the execution information includes: the execution order, execution time and data volume of multiple gradients of the multiple operators, one gradient corresponds to one operator; according to the execution information and the data transmission performance of the multiple computing devices, determine the gradient transmission strategy of the model, the gradient transmission strategy is used to indicate the way in which each computing device performs gradient fusion transmission when training the model; synchronize the gradient transmission strategy to the multiple computing devices; The multiple computing devices are used to train the model according to the gradient transmission strategy.

19. A computer-readable storage medium, characterized in that: include: Computer software instructions; When the computer software instructions are called by a computer, the method according to any one of claims 1 to 8 is implemented.

20. A computer program product, characterized in that When the computer program product is run on a computer, the computer executes the method according to any one of claims 1 to 8.

Images