Distributed training time prediction method and device, computing equipment and storage medium

Abstract

The invention provides a distributed training time prediction method, a distributed training time prediction device, computing equipment and a computer readable storage medium. The method comprises the following steps: constructing a communication network simulation model according to actual measurement information of an actual training scene for distributed training; receiving training task information, and analyzing a communication task from the training task information; inputting the communication task into the communication network simulation model, and calculating communication time; and predicting the time of distributed training according to the communication time. According to the method and the device, the communication network simulation model with a high simulation degree can be quickly constructed, so that an accurate prediction result of the distributed training time can be quickly obtained.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a distributed training time prediction method, a distributed training time prediction device, a computing device, and a computer-readable storage medium. Background Art

[0002] There are technologies in the art for predicting the time required for distributed training of artificial intelligence models. These technologies include technologies for predicting the time spent on communication required for distributed training. Existing communication time prediction technologies usually require the construction of complex and sophisticated communication models for specific training tasks in order to achieve the most accurate prediction possible. However, such models are expensive to build, and the amount of computation and time required for prediction are very large, which makes it difficult to meet the needs brought about by the current changes in communication algorithms and the increasing size of models. Therefore, there is an urgent need in the art for a distributed training time prediction technology with a simple communication model construction method and accurate prediction results. Summary of the invention

[0003] To this end, the present application is committed to providing a distributed training time prediction method, a distributed training time prediction device, a computing device and a computer-readable storage medium, which can quickly construct a communication network simulation model with a high degree of simulation and can quickly obtain accurate prediction results of distributed training time.

[0004] On the one hand, the present application provides a distributed training time prediction method, including: building a communication network simulation model based on measured information of actual training scenarios for distributed training; receiving training task information and parsing communication tasks from the training task information; inputting the communication tasks into the communication network simulation model and calculating the communication time; and predicting the time of distributed training based on the communication time.

[0005] According to this aspect, by constructing a communication network simulation model based on measured information, the difficulty of constructing the communication network simulation model is greatly simplified, and the details of the communication network simulation model can be directly parsed from the measured results, avoiding tedious calculations. In this way, the communication network simulation model can quickly calculate the communication time required for distributed training based on the communication task, and then the total time of distributed training can be obtained, which improves the prediction efficiency and reduces the prediction cost.

[0006] In a particular embodiment, a communication network simulation model is constructed based on measured information of an actual training scenario for distributed training, including: constructing a communication algorithm model for generating a traffic transmission task based on a communication task based on a record file generated during the measurement process; and constructing a network model for simulating the completion of a traffic transmission task based on measured network data.

[0007] According to this embodiment, the communication network simulation model includes a communication algorithm model and a network model, and both models are constructed based on measured information. In this way, an accurate model for simulating the communication algorithm can be constructed at a relatively low cost, thereby being able to adapt to the actual situation where the communication algorithm changes rapidly and avoiding the high cost of rebuilding the model every time the communication algorithm is updated. In addition, an accurate network model can be constructed in a simple and direct manner, avoiding many details such as protocol stacks and data packet segmentation, which can speed up the construction of the network model and perform rapid simulation and calculation.

[0008] In a particular embodiment, based on the record files generated during the actual measurement process, a communication algorithm model for generating a traffic transmission task based on the communication task is constructed, including: determining the correspondence between the communication task and the traffic transmission task based on the record files generated during the actual measurement process; and constructing a communication algorithm model based on the correspondence.

[0009] According to this embodiment, the measured record file usually contains information such as the sending target and the data volume, which can represent the specific traffic behavior generated by the communication algorithm, so that the corresponding relationship between the communication task and the traffic transmission task can be quickly established based on this information, so that by inputting the communication task, the corresponding traffic behavior can be quickly obtained. Such a communication algorithm model has low construction cost, high simulation degree, and accurate calculation results.

[0010] In a particular embodiment, the log file includes a trace file.

[0011] According to this embodiment, the trace file is a file that is usually automatically generated in a distributed training environment in the field. This file contains a large amount of operating status information and parameters, which is conducive to directly and quickly analyzing the information related to the communication algorithm, so that a corresponding relationship can be established between the communication task and the traffic transmission task.

[0012] In a particular embodiment, a network model for simulating the completion of a traffic transmission task is constructed based on measured network data, including: determining network parameters that affect traffic transmission time based on the measured network data; and constructing a network model based on the network parameters.

[0013] According to this embodiment, by determining the network parameters based on the measured data, it is helpful to accurately and quickly determine the various characteristics of the network model, so as to construct a network model with a high degree of simulation and a simple structure with high efficiency, which is helpful to improve the efficiency of subsequent communication time calculation.

[0014] In a particular embodiment, network parameters that affect traffic transmission time are determined based on measured network data, including: determining multiple network attributes that affect traffic transmission time based on the structure of the network model; and determining the values ​​of multiple network attributes as network parameters by fitting the measured network data.

[0015] According to this embodiment, the real network situation is simulated by fitting, which is beneficial to omit the simulation of network details and efficiently obtain the time spent on network transmission through a simple network topology structure.

[0016] In a particular embodiment, the communication task is input into the communication network simulation model to calculate the communication time, including: inputting the communication task into the communication algorithm model to generate a traffic transmission task; inputting the traffic transmission task into the network model to calculate the communication time.

[0017] According to this embodiment, the calculation of communication time is divided into two steps, namely, the generation of traffic transmission tasks and the time calculation of traffic transmission tasks, so that the communication network simulation model can simulate the actual behavior of the communication algorithm adopted by the communication task, and use the traffic transmission tasks generated by the communication algorithm as a medium to accurately calculate the time spent on the traffic transmission tasks, thereby efficiently and accurately calculating the communication time.

[0018] In a particular embodiment, the network model includes network parameters and link bandwidth information, and the traffic transmission task is input into the network model to calculate the communication time, including: determining the fixed delay based on the traffic transmission task and the network parameters; determining the transmission delay based on the traffic transmission task and the link bandwidth information; and calculating the communication time based on the fixed delay and the transmission delay.

[0019] According to this embodiment, by dividing the communication time into a fixed delay and a transmission delay, different calculation methods can be performed according to the properties of different delays, so that the overall communication time can be quickly calculated in a simple and efficient manner.

[0020] On the other hand, the present application provides a distributed training time prediction device, including: a construction module, used to construct a communication network simulation model based on measured information of an actual training scenario for distributed training; a receiving module, used to receive training task information and parse communication task information from the training task information; an input module, used to input the communication task information into the communication network simulation model and calculate the communication time; a prediction module, used to predict the time of distributed training based on the communication time.

[0021] On the other hand, the present application provides a computing device, which includes a processor and a memory, and the processor is used to execute a computer program stored in the memory to implement the above-mentioned distributed training time prediction method.

[0022] On the other hand, the present application provides a computer-readable storage medium, which stores a computer program, and the computer program is used to execute the above-mentioned distributed training time prediction method.

[0023] On the other hand, the present application provides a computer program product, including program code, which enables a computer to implement the above-mentioned distributed training time prediction method when the computer runs the computer program product.

[0024] Any of the distributed training time prediction devices, computing devices, computer-readable storage media or computer program products provided above are used to execute the distributed training time prediction method provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects of the corresponding schemes in the corresponding methods provided above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The specific implementation of the present application is described in detail below with reference to the accompanying drawings, wherein:

[0026] Figure 1 A schematic diagram showing the architecture of a distributed training time prediction method according to an embodiment of the present application is shown;

[0027] Figure 2 Show according to Figure 1 A schematic diagram of a communication network simulation model entity in a distributed training time prediction method of an embodiment;

[0028] Figure 3 A schematic diagram showing a flow chart of a distributed training time prediction method according to another embodiment of the present application;

[0029] Figure 4 A schematic diagram showing a flow chart of a distributed training time prediction method according to another embodiment of the present application;

[0030] Figure 5 Show according to Figure 4 A schematic diagram of the structure of a network model in a distributed training time prediction method of an embodiment;

[0031] Figure 6 A schematic diagram showing a flow chart of a distributed training time prediction method according to another embodiment of the present application;

[0032] Figure 7 A schematic diagram showing a flow chart of a distributed training time prediction method according to another embodiment of the present application;

[0033] Figure 8 A schematic diagram showing the structure of a distributed training time prediction device according to an embodiment of the present application is shown;

[0034] Fig. 9A schematic diagram of the structure of a computing device according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0035] In order to make those skilled in the art understand the concept and thought of the present application more clearly, the present application is described in detail below in conjunction with specific embodiments. It should be understood that the embodiments provided herein are only a part of all embodiments that the present application may have. After reading the specification of the present application, those skilled in the art have the ability to make improvements, transformations, or replacements to part or all of the following embodiments, and these improvements, transformations, or replacements are also included in the scope of the present application.

[0036] In this document, the terms "one", "an" and other similar words are not intended to indicate that there is only one of the things described, but rather that the relevant description is only for one of the things described, and the things described may have one or more. In this document, the terms "comprises", "includes" and other similar words are intended to indicate logical relationships, and cannot be regarded as indicating spatial structural relationships. For example, "A includes B" is intended to indicate that B logically belongs to A, but does not mean that B is spatially located inside A. In addition, the meanings of the terms "comprises", "includes" and other similar words should be regarded as open, rather than closed. For example, "A includes B" is intended to indicate that B belongs to A, but B does not necessarily constitute the whole of A, and A may also include other elements such as C, D, and E.

[0037] In this document, the terms "first", "second" and other similar words are not intended to imply any order, quantity and importance, but are only used to distinguish different elements. In this document, the terms "embodiment", "present embodiment", "one embodiment" and "an embodiment" do not mean that the relevant description is only applicable to a specific embodiment, but rather that these descriptions may also be applicable to one or more other embodiments. Those skilled in the art should understand that in this document, any description made for a certain embodiment can be replaced, combined, or otherwise combined with the relevant description in one or more other embodiments, and the new embodiment produced by the replacement, combination, or other combination is easily conceivable by those skilled in the art and belongs to the scope of protection of this application.

[0038] In the embodiments of the present application, distributed training may refer to the use of computing units such as GPUs (graphics processing units) on one or more machines for parallel computing to train artificial intelligence models or other models. For example, distributed training may refer to the training of a model by multiple processes working together, and these processes can run on a single machine or on multiple machines. For example, in distributed training, the workload used to train the model is split and shared among multiple microprocessors, which can be called worker nodes. These worker nodes work in parallel to accelerate model training. Distributed training can be used for traditional machine learning models, and is also suitable for computationally and time-intensive tasks, such as deep learning for training deep neural networks.

[0039] Distributed training time may refer to the total amount of time required to complete the distributed training task of the entire model. Distributed training time prediction may refer to the total model training time predicted by simulation or emulation based on the structure, algorithm, scale and other information of the AI ​​model and the device and software used to train the AI ​​model, without actually completing the entire model training task.

[0040] In AI (artificial intelligence) model training, the training time is mainly divided into computing time and communication time. The estimation of computing time can be completed by modeling the computing performance of a single card, while the time estimation of the communication part requires modeling the network cluster to predict the time consumption of the communication algorithm. As the AI ​​model grows, a larger cluster is needed to train the AI ​​model, which leads to the need to evaluate the cluster communication performance of large-scale network clusters.

[0041] Large-scale application developers need to perform network simulation to evaluate the performance of the communication part during algorithm design, performance bottleneck analysis, and optimization, so as to guide development or optimization. Cluster scheduling also needs to evaluate the time consumption of model training based on the number of cards rented by customers, and the time consumption of the communication part also needs to be obtained through network simulation. In these scenarios, network simulation is needed to provide communication time estimates in a relatively short time.

[0042] In this field, the prediction of cluster network communication time mainly adopts simulation modeling method, which is divided into two parts. The first part is to model the communication algorithm, that is, to simulate the network traffic behavior generated by the collective communication task such as AllReduce communication operator according to the communication algorithm adopted in its implementation, such as Ring algorithm. The second part is to model the network, and then use the network traffic behavior in the first part as input to estimate the time required to complete the entire network traffic behavior of the task by simulating the traffic transmission process.

[0043] The communication algorithm modeling technology in this field requires developers who have mastered the relevant capabilities of the communication algorithm to develop a communication algorithm model based on the algorithm implementation details. The model needs to be able to complete the conversion from communication task input to network traffic behavior output.

[0044] The problem with these technologies is that communication algorithms are updated and iterated quickly, and the same communication operator may have different optimal algorithm implementations in different product forms. Therefore, when the product is iterated, the adapted communication algorithm will also be different. This requires frequent modification of the model when modeling the communication algorithm, which requires a lot of time and manpower investment.

[0045] The network modeling technology in this field mainly uses a network simulation platform to model network details. It is necessary to implement all network cards, switch models, and details of each layer of the protocol stack in the corresponding network. Each time data is sent or received, each component needs to act in sequence to generate corresponding events to complete the simulation of data transmission actions close to the real network. In this process, a data transmission between two nodes is divided into the transmission of multiple data packets, and a data packet is divided into the transmission of multiple data slices. By simulating the behavior of each slice in all links and node devices along the way, the simulation of a data transmission is achieved.

[0046] The problem with these technologies is that network modeling emphasizes the simulation of the network. A single data transmission will generate a large number of network packets and network slices. A single packet or data slice needs to be forwarded, parsed, and scheduled by multiple network components. In the scenario of AI large model training, due to the huge network scale and the huge amount of network data, using this network modeling method to simulate the traffic behavior of the collective communication operator will generate a huge amount of network slice transmission actions, which will be very time-consuming.

[0047] Some embodiments of the present application are aimed at solving two problems when predicting the communication time in AI training through simulation: 1) The communication algorithm is iteratively updated quickly, and the time and manpower costs of frequently modeling and updating the communication operators are high; 2) In the communication simulation of large-scale AI training, the network scale is large and the data volume is large, resulting in too long simulation time for traditional network modeling solutions.

[0048] Figure 1 A schematic diagram of the architecture of a distributed training time prediction method according to an embodiment of the present application is shown.

[0049] like Figure 1As shown, the time consumed by distributed AI model training includes the communication time between distributed computing nodes. The prediction of communication time is an important aspect of distributed training time prediction. The prediction of communication time is completed through the communication network simulation model. By inputting the communication task into the communication network simulation model, the communication time required for distributed training to communicate can be obtained. Information about communication tasks can be parsed from the training task information. Specific AI model training tasks will have specific training task information, which will include communication task (such as communication operator) information. The communication network simulation model is constructed by the user based on the information obtained from the actual testing of the training device and / or training software actually used for distributed training. The communication network simulation model constructed by this method has a simple structure, avoids complex calculations, and is in line with the actual training situation. It can ensure the high accuracy of simulation, thereby greatly improving the efficiency of estimating or predicting communication time.

[0050] Figure 2 Show according to Figure 1 A schematic diagram of an example of a communication network simulation model in the distributed training time prediction method of an embodiment.

[0051] like Figure 2 The distributed training tasks shown include computing tasks and communication tasks. The simulation of computing tasks is implemented through a computing model. By inputting the computing tasks into the computing model, the time spent on computing can be obtained. The simulation of communication tasks is implemented through a communication network simulation model. By inputting the communication tasks into the communication network simulation model, the time spent on communication can be obtained. In this example, the communication network simulation model includes a communication algorithm model and a network model. The communication algorithm model simulates the communication algorithm and converts the communication tasks (such as communication operators) into traffic behaviors (for example, transmitting a certain amount of data from one node to another). The network model simulates the structure and transmission behavior of the communication network. By inputting the traffic behavior into the network model, the time required for the traffic transmission to be completed can be calculated, thereby obtaining the communication time.

[0052] Figure 3 A flow chart of a distributed training time prediction method according to an embodiment of the present application is shown.

[0053] According to this embodiment, the distributed training time prediction method includes steps S130 to S340, and each step is described in detail below.

[0054] S310: Construct a communication network simulation model based on measured information of an actual training scenario for distributed training.

[0055] In this embodiment, the actual training scenario of distributed training may refer to the training equipment and / or training software actually used in the distributed training, as well as other training configurations or conditions that may be included. Among them, the training equipment can be any computing device suitable for large-scale AI model training. Such a device can be a device dedicated to model training, or it can be a general-purpose computing device, which can be used for other computing tasks in addition to model training. For example, the computing equipment used in distributed training may include a network card, a GPU, an artificial intelligence chip, and the like. The training software may refer to software used in the art for building and training artificial intelligence models, such as a commercially available artificial intelligence framework.

[0056] In this embodiment, the measured information for the actual training scenario may refer to information about the training scenario obtained by using the hardware and / or software of the actual training scenario to perform actual model training, or actual testing of the hardware and / or software of the actual training scenario. In this embodiment, the measured information for the actual training scenario may also include initial parameters or configuration information that can be determined when the training device or training software is shipped or purchased.

[0057] In this embodiment, the communication network simulation model may refer to a model used to simulate the communication behavior or communication-related behavior required in distributed training. Through the simulation and emulation of the communication network simulation model, the time required for the communication behavior or operation in the distributed training can be calculated, which in turn helps to estimate the total time required for the entire distributed training.

[0058] S320: Receive training task information, and parse the communication task from the training task information.

[0059] In this embodiment, the training task information may refer to information related to the distributed model training task. The model training task usually includes information such as model parameters and scale, model type, required computing amount, adopted algorithm, required training effect, etc. Those skilled in the art may also think of other information related to the training task, and such information is included in the scope of this application.

[0060] In this embodiment, the communication task may refer to the communication task or related task that needs to be implemented between distributed nodes in distributed training. Because there are many computing nodes in distributed training, nodes need to communicate with each other and share data to complete the training task. These data communications and synchronous sharing belong to the scope of communication tasks. Those skilled in the art can also think of other types of communication tasks. In some embodiments of the present application, the communication task may refer to the communication primitive or communication operator in the collective communication. Collective communication usually refers to the global communication operation in which all processes of a process group participate. The most basic operations are sending, receiving, copying, intra-group process barrier synchronization and inter-node process synchronization. These most basic operations are combined to form a group of communication templates also called communication primitives, such as one-to-many Broadcast, many-to-one Gather, many-to-many AllGather, one-to-many Scatter, many-to-one Reduce, many-to-many AllReduce, combined ReduceScatter (reduction and divergence), many-to-many AlltoAll, etc. Broadcast, Gather, AllReduce, etc. are all different communication tasks, or communication operators.

[0061] In this embodiment, the communication task is parsed out from the training task information, which can be obtained by parsing the communication-related information contained in the training task information itself, or by calculating the communication task information required to achieve the training task from the training task information based on specific experience or algorithms. Other ways of parsing out the communication task are also conceivable.

[0062] S330, input the communication task into the communication network simulation model, and calculate the communication time.

[0063] In this embodiment, the communication time may refer to the time required to complete a communication task.

[0064] In this embodiment, inputting the communication task into the communication network simulation model may refer to inputting the information contained in the communication task itself into the communication network simulation model, or may refer to inputting the information and parameters related to the communication task into the communication network simulation model. Such information may be input into the communication network simulation model together, or may be input into different parts of the communication network simulation model step by step as required.

[0065] S340: predict the time of distributed training according to the communication time.

[0066] In this embodiment, the communication time is a part of the total time consumed by the distributed training. In order to predict the time of the distributed training based on the communication time, the time required for other tasks (such as computing tasks, scheduling tasks, etc.) can be calculated based on the communication time, and these times can be calculated comprehensively to obtain the final distributed training time.

[0067] Figure 4 A flow chart of a distributed training time prediction method according to an embodiment of the present application is shown.

[0068] According to this embodiment, the distributed training time prediction method includes steps S410 to S450, and each step is described in detail below.

[0069] S410: Construct a communication algorithm model for generating a traffic transmission task according to a communication task based on the record file generated during the actual measurement process.

[0070] In this embodiment, the record files generated during the actual measurement process may refer to record files automatically generated in the actual test environment. These record files may be log files or operation status record files automatically generated in the software or network used for actual testing, which are used to record the operating status of the equipment and software to facilitate subsequent analysis and troubleshooting.

[0071] In this embodiment, traffic may refer to information or data packets sent in a single time. In distributed training, different collective communication operators will produce different traffic behaviors, and different information, messages or data packets need to be sent to different objects. For example, each content sent to a specific object can be counted as a traffic flow. Sending multiple contents to multiple objects can be counted as multiple traffic flows on multiple links. A traffic transmission task may refer to a task of sending specific content from a specific node to another specific node. A traffic transmission task usually includes multiple transmissions of multiple traffic flows. These multiple transmission tasks all belong to traffic transmission tasks in the overall sense.

[0072] In this embodiment, the communication algorithm may refer to an algorithm used to implement a specific communication task (or communication operator). For example, for the AllReduce task, the Ring algorithm may be used, or the halving and doubling algorithm, the butterfly algorithm, etc. may be used. The communication algorithm model may refer to a model composed of a communication algorithm, whose input is a communication task and whose output is a traffic transmission task or behavior. The communication algorithm model itself may contain communication algorithm information, or may only contain the correspondence between the communication task and the traffic transmission task, as long as a specific traffic transmission task can be output according to a specific communication task.

[0073] As an example, S410 may be specifically implemented as follows: determining the correspondence between the communication task and the traffic transmission task according to the record file generated during the actual measurement process; and constructing a communication algorithm model according to the correspondence.

[0074] In this example, the correspondence between the communication task and the traffic transmission task may mean that a specific communication task always generates a specific traffic transmission task, and there is a correspondence between such a communication task and such a traffic transmission task. Generally speaking, the correspondence depends on the algorithm used to implement such a communication task. For example, in order to implement the AllReduce task, the Ring algorithm may be used, and the Ring algorithm determines the correspondence between the AllReduce task and the specific traffic transmission task. If other algorithms are used, the correspondence changes accordingly.

[0075] As an example, the log files include trace files.

[0076] In this example, a trace file can be a file used to record system operation information. It can be used to collect debugging information, performance indicators, error messages, and other information, which can be used to analyze system performance, behavior, and problems. A trace file usually contains many events and data points. Each data point has a timestamp and event name. The event name helps to understand the meaning of each data point. For example, a trace file can refer to a trace file automatically generated in a training scenario (including training equipment or training software architecture, etc.) used in a distributed training task. For example, a trace file can refer to a file automatically generated during the operation of a collective communication library such as NCCL (NVIDIA collective communications library) used in distributed training. Trace files of other forms or sources are also conceivable.

[0077] In this embodiment, a communication operator modeling technology with trace as input is adopted. By parsing the measured communication operator trace file and parsing the type of data sent and received, the amount of data, the communication object and other information, the complete traffic behavior of the corresponding operator can be obtained, and the dependencies between these traffic behaviors are recorded. Then, when the same communication operator is input, the corresponding traffic behavior can be output. In this embodiment, the communication algorithm model is automatically generated by parsing the measured trace in AI communication simulation modeling, which can greatly reduce the cost of communication algorithm modeling and make the communication algorithm model update more efficient and quick. In this embodiment, the communication algorithm is modeled by parsing the traffic dependencies in the measured trace file, which can automate the modeling process, reduce the workload of communication algorithm modeling, and improve the model update speed.

[0078] S420: Construct a network model for simulating the completion of a traffic transmission task based on the measured network data.

[0079] In this embodiment, the measured network data may refer to data obtained by measuring or testing the network in an actual scenario of distributed training. These data are used to display the network conditions in the distributed training environment, such as network communication rate, network delay, etc.

[0080] In this embodiment, the network model may refer to a model used to simulate the network actually used for distributed training. This model can simulate various details in the network in a refined manner, or it can be simulated using a simple structure. In some embodiments of the present application, the network model may be a network model with an undirected graph structure, which does not have the details of each protocol stack that can be simulated by the usual network simulation software, but only includes nodes and connection edges. Such a network model is conducive to speeding up the simulation calculation, quickly obtaining the communication time, and is suitable for rapid prediction of the training time of large models.

[0081] As an example, S420 may be specifically implemented as follows: determining network parameters that affect traffic transmission time based on measured network data; and constructing a network model based on the network parameters.

[0082] In this embodiment, the network parameters may refer to various indicators used to display network performance, such as network transmission rate, number of network nodes, number of network links, network scale, network delay, etc.

[0083] As an example, in order to determine the network parameters that affect traffic transmission time based on the measured network data, we can first determine multiple network attributes that affect traffic transmission time based on the structure of the network model; then, by fitting the measured network data, we can determine the values ​​of multiple network attributes as network parameters.

[0084] In this example, the structure of the network model may refer to the structure of a virtual network model used to simulate a real network, which is usually an artificially constructed and designed structure used to simulate the actual running network as realistically as possible while simplifying calculations. In some embodiments of the present application, the structure of the network model may refer to a simplified network model structure, such a network model does not contain the layer-by-layer segmentation characteristics and various details of the protocol stack in the real network, but only contains nodes and edges. Other types of network structures are also conceivable.

[0085] In this example, network attributes can refer to the abstraction of structural characteristics and operating factors of a virtual network model, summarizing multiple characteristics or features that can affect traffic transmission time. When these characteristics or features have values ​​of different sizes, the traffic transmission time will change, and the transmission performance of the entire virtual network will also change, thus getting closer and closer to the actual performance of the real network. For example, network attributes can include task startup delay, data copy fixed overhead, data copy bandwidth, aggregate computing fixed overhead, aggregate computing bandwidth, process synchronization overhead, intra-node network bandwidth, intra-node network fixed overhead, inter-node network bandwidth, inter-node network fixed overhead, etc. These network attributes need to be fitted and calibrated to match the actual environment.

[0086] In this example, in order to determine the values ​​of multiple network attributes by fitting the measured network data, the approximate range of the value of each network attribute can be roughly determined based on the measured network data, and then the value of the network attribute can be continuously adjusted according to experience or algorithms based on the specific network data, so that the simulation results are continuously close to the measured results, and finally the optimal value of each network attribute is determined.

[0087] According to this embodiment, the details of the protocol stack on the network card and the switch are hidden in the network modeling, the network data transmission is simulated using flow granularity, the key attributes of the network traffic transmission time are abstracted and fit with the measured values, the bandwidth of the conflicting data streams is reallocated, and the data flow completion time is estimated in combination with the measured parameters and the allocated bandwidth, thereby greatly improving the simulation speed while ensuring a certain degree of accuracy.

[0088] In this embodiment, a data flow simulation technology that fits the software and hardware network information is used to simplify the network into an undirected graph containing only nodes and connecting edges. By collecting measured network data and fitting the network transmission parameters, the protocol stack overhead is estimated. The data flow is used as the main object of network simulation, and its transmission time is divided into a fixed delay part calculated by the fitting parameters and a transmission delay part affected by the link bandwidth. For conflicts among multiple flows, the link bandwidth is reallocated to simulate the multi-flow competing links. In this embodiment, by simplifying the network model and simulating the network at the flow granularity, the network rate is greatly improved. In this embodiment, by fitting the software and hardware network details, the network protocol stack overhead is simulated to ensure a high network simulation accuracy in the simplified network model.

[0089] S430: Receive training task information, and parse the communication task from the training task information.

[0090] For details about S430, see above Figure 3 The detailed description of Example S320 is not repeated here.

[0091] S440: Input the communication task into the communication algorithm model to generate a traffic transmission task.

[0092] In this embodiment, the communication operator and other information included in the communication task may be input into the communication algorithm model, and the communication algorithm model calculates the corresponding traffic transmission task according to the correspondence model between the communication task and the traffic transmission task included therein.

[0093] S450: Input the traffic transmission task into the network model and calculate the communication time.

[0094] In this embodiment, various details of the traffic transmission tasks can be input into the network model, and the network model obtains the network transmission time or communication time required to complete these traffic transmission tasks by simulating and fitting the real network.

[0095] As an example, S450 can be specifically implemented as follows: determining a fixed delay according to a traffic transmission task and network parameters; determining a transmission delay according to the traffic transmission task and link bandwidth information; and calculating a communication time according to the fixed delay and the transmission delay.

[0096] In this example, fixed delay can refer to a delay that is independent of the amount of data or information contained in the traffic. For example, the waiting time, sending time, etc. of the traffic in the network node can be counted as fixed delay. Transmission delay can refer to the delay that the traffic takes different time to transmit in the network due to the different data volumes. For example, the time it takes for a traffic of a certain size to be transmitted in a transmission path with a certain bandwidth can be regarded as transmission delay.

[0097] In this example, the link bandwidth information may refer to information about the bandwidth size of the transmission link, or information about the bandwidth size of multiple transmission paths between multiple nodes included in the transmission link, or other information related to the bandwidth of the transmission link.

[0098] In this example, in order to calculate the communication time based on the fixed delay and the transmission delay, all the calculated fixed delays and transmission delays can be added together to calculate the total communication time. Alternatively, the total communication time can be finally obtained after performing other calculations based on the fixed delay and the transmission delay.

[0099] S460: Predict the time of distributed training according to the communication time.

[0100] For details about S460, see above Figure 3 The detailed description of Example S340 is not repeated here.

[0101] Figure 5 Show according to Figure 4A schematic diagram of the structure of a network model in a distributed training time prediction method of an embodiment.

[0102] like Figure 5 As shown, the network model can be an undirected graph containing only nodes and edges, which is used to simplify the network model in the traditional network simulation software, hiding various details such as the protocol stack and data slices in the network, so that the real network used in the actual training can be simulated and emulated according to the simple undirected graph network structure, so as to speed up the simulation speed and quickly obtain the network transmission consumption time (i.e., communication consumption time).

[0103] exist Figure 5 In the figure, black dots represent network nodes, and straight lines between nodes represent edges (i.e., network transmission paths). For example, nodes 1 and 2 are connected by edge 1, nodes 1 and 3 are connected by edge 2, and nodes 1 and 4 are connected by edge 3. Edge 1, edge 2, and edge 3 are all network transmission paths connected to node 1. One or more edges connected end to end can form a link, for example Figure 5 The link represented by the bold black line is the transmission link from node 1 to node 3, from node 3 to node 5, and then from node 5 to node 6. The time that traffic stays on each node, including the time that network nodes (such as computers, switches, etc.) process traffic data and the time that traffic queues on the node waiting to be sent, is a fixed delay. The time that traffic is transmitted on the link is determined by the amount of traffic data and the bandwidth of each edge on the link, which is a transmission delay. Link bandwidth affects the data transmission rate, which in turn affects the transmission delay.

[0104] Figure 6 A flow chart of a distributed training time prediction method according to an embodiment of the present application is shown.

[0105] like Figure 6As shown, the information of the distributed training task is input by the user, such as the MoE (mixture of experts) large model training performed with 1024 network cards. The prediction of the distributed training time is made by the AI ​​training performance evaluation platform. The AI ​​training performance evaluation platform can be an independent computing platform with independent hardware and software, or it can be a part or application of other platforms. The evaluation platform includes a scheduling simulation module for simulating scheduling operations in training, a computing simulation module for simulating computing tasks, and a communication network simulation model for simulating communication behavior to obtain communication time, such as the high-speed network simulation module in this embodiment. The high-speed network simulation is input by the platform scheduling communication tasks, such as the AllReduce operation of 100MB (megabyte) data performed with 1024 network cards. The output of the high-speed network simulation is the predicted communication time. The high-speed network simulation includes a communication algorithm model and a network model. See above for its functions and other details. Figure 4 After the AI ​​training performance evaluation platform completes the time prediction for scheduling, computing, and communication, it calculates the training time and outputs it, that is, it outputs the estimated result for the total training duration.

[0106] Figure 7 A flow chart of a distributed training time prediction method according to an embodiment of the present application is shown.

[0107] like Figure 7 As shown, the prediction of distributed training time is implemented by the AI ​​training performance evaluation platform. The input of the AI ​​training performance evaluation platform is the training task information, and the output is the training time estimate or prediction. After receiving the training task information, the model parsing module parses the training task information to obtain the calculation operator and the communication operator. The model parsing module sends the calculation operator to the calculation simulation module, which simulates the calculation process to obtain the estimated result of the calculation time required for the calculation operator, and the calculation simulation module feeds back the calculation time to the model parsing module. The model parsing module sends the communication operator to the communication network simulation model such as the high-speed network simulation module in this embodiment, and the communication operator is received by the communication algorithm model in the high-speed network simulation module. After the high-speed network simulation module completes the time estimation of the communication operator, it feeds back the estimation result to the model parsing module. The model parsing module calculates the training time based on the feedback from the calculation simulation module and the high-speed network simulation module, and outputs the training time prediction result.

[0108] like Figure 7As shown, the high-speed network simulation includes a communication algorithm model and a network model. The communication algorithm model is constructed according to the trace file input by the configuration. The high-speed network simulation module includes a trace parsing module, which parses the trace file input by the configuration to obtain information related to the communication algorithm, and thus constructs the communication algorithm model accordingly. The network model is constructed according to the network fitting parameters input by the configuration. The high-speed network simulation module includes a network configuration module, which constructs the network model according to the network fitting parameters. The communication algorithm model converts the issued communication operator into a single traffic transmission task and inputs it into the network model. The network model calculates the time required for the transmission to complete based on the information of the traffic transmission task and feeds it back to the communication algorithm model. After the simulation of all traffic transmission tasks is completed, the time required for all communications in the training task is obtained, that is, the communication operator time estimation result. The result is fed back to the model parsing module, and finally the training time prediction result is formed.

[0109] Specifically, in Figure 7 In the illustrated embodiment, taking the estimation of the MoE model training time for 1024 network cards under a specific data set as an example, the process is as follows: 1) read in the trace file, and the trace parsing module parses its communication algorithm information to model / update the communication algorithm model; 2) read in the network fitting information collected according to the measured network, and reconfigure the network model according to the fitting parameters; 3) input the training task information and estimate the training time.

[0110] like Figure 7 As shown in the figure, after the platform inputs the training task information, it needs to include the corresponding data set, training scale, and model. The model parsing module issues the operator task generated according to the model information, among which the computing operator task is completed by the corresponding computing simulation module. The communication operator task is issued to the network simulation module. The issued communication task description includes the communication scale such as 1024 network cards, the communication data volume such as 100MB data, and the collective communication operator type such as AllReduce task. The communication algorithm model generates a series of traffic transmission tasks issued by 1024 network cards according to the communication task information. These traffic transmission tasks calculate the fixed delay part through the network fitting parameters, and calculate the actual occupied bandwidth through the bandwidth competition between the traffic, so as to obtain the traffic completion time. When all traffic simulations are completed, the communication algorithm model completes the time consumption prediction of the AllReduce task and informs the model parser of the results. The model parses it and issues the subsequent operators. Repeat the above operator issuance process until all operators are executed, and then output the training time estimate.

[0111] Based on the aforementioned Figure 3 The method embodiment described above, the present application embodiment also provides a distributed training time prediction device, the structural diagram of which is as follows Figure 8 The device is used to perform the above Figure 3 The various steps in .

[0112] According to this embodiment, the distributed training time prediction device 800 includes a construction module 810, a receiving module 820, an input module 830 and a prediction module 840. The construction module 810 is used to construct a communication network simulation model based on the measured information of the actual training scenario for distributed training. The receiving module 820 is used to receive the training task information and parse the communication task information from the training task information. The input module 830 is used to input the communication task information into the communication network simulation model to calculate the communication time. The prediction module 840 is used to predict the time of distributed training based on the communication time.

[0113] In one embodiment, the building block is further configured to:

[0114] According to the record files generated during the actual measurement process, a communication algorithm model is constructed for generating traffic transmission tasks according to the communication tasks;

[0115] Based on the measured network data, a network model is constructed to simulate the completion of the traffic transmission task.

[0116] In one embodiment, the root building block is further configured to:

[0117] According to the record files generated during the actual measurement process, determine the correspondence between the communication tasks and the traffic transmission tasks;

[0118] According to the corresponding relationship, a communication algorithm model is constructed.

[0119] In one embodiment, the record file includes a trace file.

[0120] In one embodiment, the building block is further configured to:

[0121] Determine the network parameters that affect traffic transmission time based on measured network data;

[0122] Construct a network model based on network parameters.

[0123] In one embodiment, the building block is further configured to:

[0124] Based on the structure of the network model, multiple network attributes that affect traffic transmission time are determined;

[0125] By fitting the measured network data, the values ​​of multiple network attributes are determined as network parameters.

[0126] In one embodiment, the input module is further configured to:

[0127] Input the communication task into the communication algorithm model to generate the traffic transmission task;

[0128] The traffic transmission task is input into the network model and the communication time is calculated.

[0129] In one embodiment, the network model includes network parameters and link bandwidth information, and the input module is further configured to:

[0130] Determine the fixed delay based on the traffic transmission task and network parameters;

[0131] Determine the transmission delay based on the traffic transmission task and link bandwidth information;

[0132] The communication time is calculated based on the fixed delay and transmission delay.

[0133] It should be noted that Figure 8 The distributed training time prediction device 800 provided in the embodiment shown in the figure only uses the division of the above-mentioned functional modules as an example when executing the distributed training time prediction method. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. Figure 3 The distributed training time prediction method embodiments shown belong to the same concept, and their specific implementation processes are detailed in the method embodiments, which will not be repeated here.

[0134] Fig. 9 It is a schematic diagram of the hardware structure of a computing device 900 provided in an embodiment of the present application.

[0135] See also Fig. 9 The computing device 900 includes a processor 910, a memory 920, a communication interface 930, and a bus 940, wherein the processor 910, the memory 920, and the communication interface 930 are connected to each other via the bus 940. The processor 910, the memory 920, and the communication interface 930 may also be connected in other connection modes besides the bus 940.

[0136] Among them, the memory 920 can be various types of storage media, such as random access memory (RAM), read-only memory (ROM), non-volatile RAM (NVRAM), programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), flash memory, optical storage, hard disk, etc.

[0137] The processor 910 may be a general-purpose processor, which may be a processor that performs specific steps and / or operations by reading and executing the contents stored in a memory (e.g., the memory 920). For example, the general-purpose processor may be a central processing unit (CPU). The processor 910 may include at least one circuit to execute Figure 3 The illustrated embodiment provides all or part of the steps of the distributed training time prediction method.

[0138] Among them, the communication interface 930 includes input / output (I / O) interfaces, physical interfaces, and logical interfaces for interconnecting devices within the computing device 900, as well as interfaces for interconnecting the computing device 900 with other devices (such as other computing devices or user devices). The physical interface can be an Ethernet interface, an optical fiber interface, an ATM interface, etc. The communication interface 930 can be connected to an external input device and an output device. For example, the input device can be a microphone or a microphone array for capturing voice input signals; it can be a communication network connector for receiving collected input signals from the cloud or other devices; it can also include, for example, a keyboard, a mouse, etc. The output device can output various information to the outside, including determined distance information, direction information, etc. The output device can include, for example, a display, a speaker, a printer, a communication network and a remote output device connected thereto, etc.

[0139] The bus 940 may be any type of communication bus for interconnecting the processor 910 , the memory 920 , and the communication interface 930 , such as a system bus.

[0140] The above devices may be arranged on independent chips, or at least partially or completely on the same chip. Whether to arrange each device independently on different chips or to integrate them on one or more chips often depends on the needs of product design. The embodiments of the present application do not limit the specific implementation form of the above devices.

[0141] Fig. 9 The computing device 900 shown is merely exemplary. During implementation, the computing device 900 may further include other components, which are not listed one by one herein.

[0142] An embodiment of the present application may also be a computer-readable storage medium having computer program instructions stored thereon, which, when executed by a processor, causes the processor to execute the steps of the distributed training time prediction method according to various embodiments of the present application described above in this specification.

[0143] The computer readable storage medium can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can include, for example, but is not limited to, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0144] The concepts, principles and ideas of the present application are described in detail above in conjunction with specific implementation methods (including embodiments and examples). Those skilled in the art should understand that the implementation methods of the present application are more than the several forms given above. After reading the application documents, those skilled in the art can make any possible improvements, replacements and equivalent forms to the steps, methods, devices and components in the above-mentioned implementation methods, and these improvements, replacements and equivalent forms should be deemed to fall within the scope of the present application. The scope of protection of the present application shall be subject only to the claims.

Claims

1. A distributed training time prediction method, characterized in that: The method comprises: constructing a communication network simulation model based on measured information of an actual training scenario for the distributed training; Receiving training task information, and parsing communication tasks from the training task information; Inputting the communication task into the communication network simulation model to calculate the communication time; The time of the distributed training is predicted according to the communication time.

2. The distributed training time prediction method according to claim 1, characterized in that: The constructing of a communication network simulation model according to measured information of an actual training scenario for the distributed training includes: According to the record files generated during the actual measurement process, a communication algorithm model is constructed for generating a traffic transmission task according to the communication task; Based on the measured network data, a network model is constructed to simulate the completion of the traffic transmission task.

3. The distributed training time prediction method according to claim 2, characterized in that: The step of constructing a communication algorithm model for generating a traffic transmission task according to the communication task based on the record file generated during the actual measurement process includes: According to the record files generated during the actual measurement process, determine the correspondence between the communication tasks and the traffic transmission tasks; According to the corresponding relationship, the communication algorithm model is constructed.

4. The distributed training time prediction method according to claim 3, characterized in that: The record file includes a trace file.

5. The distributed training time prediction method according to claim 2, characterized in that: The step of constructing a network model for simulating the completion of the traffic transmission task based on the measured network data includes: Determine the network parameters that affect traffic transmission time based on measured network data; The network model is constructed according to the network parameters.

6. The distributed training time prediction method according to claim 5, characterized in that: Determining the network parameters that affect the traffic transmission time based on the measured network data includes: Determining, based on the structure of the network model, a plurality of network attributes that affect traffic transmission time; By fitting the measured network data, the values ​​of the plurality of network attributes are determined as the network parameters.

7. The distributed training time prediction method according to claim 2, characterized in that: The step of inputting the communication task into the communication network simulation model and calculating the communication time comprises: Inputting the communication task into the communication algorithm model to generate the traffic transmission task; The traffic transmission task is input into the network model to calculate the communication time.

8. The distributed training time prediction method according to claim 7, characterized in that: The network model includes network parameters and link bandwidth information, and the step of inputting the traffic transmission task into the network model and calculating the communication time includes: Determining a fixed delay according to the traffic transmission task and the network parameters; Determining a transmission delay according to the traffic transmission task and the link bandwidth information; The communication time is calculated according to the fixed delay and the transmission delay.

9. A distributed training time prediction device, characterized in that: The device comprises: A construction module, configured to construct a communication network simulation model based on measured information of an actual training scenario for the distributed training; A receiving module, used for receiving training task information and parsing communication task information from the training task information; An input module, used to input the communication task information into the communication network simulation model to calculate the communication time; A prediction module is used to predict the time of the distributed training according to the communication time.

10. A computing device, characterized in that The computing device includes a processor and a memory, wherein the processor is configured to execute a computer program stored in the memory to implement the distributed training time prediction method according to any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and the computer program is used to execute the distributed training time prediction method according to any one of claims 1 to 8.

Images