Communication delay time prediction method, system, electronic device and storage medium

Abstract

The present invention provides a communication delay time prediction method, system, electronic device and storage medium, which relates to the field of computer technology. The method includes: fitting the measured network communication parameters of the sample communication system obtained by actual measurement to obtain the target startup delay coefficient corresponding to each data jump transmission in the communication system to be predicted; performing startup delay time analysis according to the target startup delay coefficient and the total number of data jump transmissions in the communication system to be predicted to obtain the startup delay time of the communication system to be predicted; performing transmission delay time analysis according to the network configuration parameters of the communication system to be predicted and the amount of data transmitted by the communication system to be predicted to obtain the transmission delay time of the communication system to be predicted; performing total delay time prediction according to the startup delay time and the transmission delay time to obtain the total delay time prediction value of the communication system to be predicted. The present invention realizes accurate prediction of the end-to-end delay time in the communication process while reducing the configuration cost.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a communication delay time prediction method, system, electronic device and storage medium. Background Art

[0002] Collective communication refers to a communication mode in parallel computing and distributed computing in a communication system, which involves data exchange and coordinated operations between multiple processing units. Since the parallel computing of multiple processing units will bring additional communication delay time, how to accurately predict the communication delay time of multiple processing units to optimize the design of parallel communication algorithms of multiple processing units, reduce the proportion of communication, and thus greatly improve computing efficiency is an important issue that needs to be solved in the industry.

[0003] In the related art, when obtaining the end-to-end delay time for a certain scenario, it is necessary to build a physical model for the scenario, and use the communication test tool of the communication library on the physical model to design a fixed topology for a specific data block size, and conduct actual experimental tests on the physical model to obtain the actual end-to-end delay data for the scenario. When it comes to obtaining the end-to-end delay time for multiple scenarios, it is necessary to build a corresponding physical model for each scenario, which has a high configuration cost.

[0004] Therefore, there is an urgent need to provide a communication delay time prediction method, system, electronic device and storage medium to solve the above technical problems. Summary of the invention

[0005] The present invention provides a communication delay time prediction method, system, electronic device and storage medium, which are used to solve the defects in the prior art that corresponding physical models need to be built for each scene one by one, and the configuration cost is relatively high, so as to achieve accurate prediction of communication delay time and reduce configuration cost.

[0006] The present invention provides a communication delay time prediction method, comprising:

[0007] The network communication parameters of the sample communication system are measured to obtain the measured network communication parameters of the sample communication system, and the startup delay coefficient is fitted according to the measured network communication parameters to obtain the target startup delay coefficient corresponding to each data jump transmission in the communication system to be predicted; the sample communication system and the communication system to be predicted are both communication systems constructed by multiple processing nodes;

[0008] Performing a startup delay time analysis according to the target startup delay coefficient and the total number of data jump transmissions in the communication system to be predicted to obtain a startup delay time of the communication system to be predicted;

[0009] Performing transmission delay time analysis according to the network configuration parameters of the communication system to be predicted and the amount of data transmitted by the communication system to be predicted to obtain the transmission delay time of the communication system to be predicted;

[0010] A total delay time prediction is performed according to the startup delay time and the transmission delay time to obtain a total delay time prediction value of the communication system to be predicted.

[0011] According to a communication delay time prediction method provided by the present invention, the transmission delay time analysis is performed according to the network configuration parameters of the communication system to be predicted and the amount of data transmitted by the communication system to be predicted to obtain the transmission delay time of the communication system to be predicted, including:

[0012] Performing parameter parsing on the network configuration parameters to obtain network topology information of the communication system to be predicted and bandwidth data of each communication link in the communication system to be predicted;

[0013] The transmission delay time of the communication system to be predicted is calculated according to the network topology information, the bandwidth data and the amount of data transmitted by the communication system to be predicted.

[0014] According to a communication delay time prediction method provided by the present invention, the transmission delay time of the communication system to be predicted is calculated based on the network topology information, the bandwidth data and the amount of data transmitted by the communication system to be predicted, including:

[0015] Perform statistics on the number of communication links and the number of processing nodes according to the network topology information to obtain the number of communication links and the number of processing nodes in the communication system to be predicted;

[0016] Calculating the product of the number of communication links and the bandwidth data to obtain the total bandwidth data of the communication system to be predicted;

[0017] Calculating the ratio between the amount of data transmitted by the communication system to be predicted and the total bandwidth data to obtain the bandwidth utilization rate of the communication system to be predicted;

[0018] The transmission delay time of the communication system to be predicted is calculated according to the number of processing nodes and the bandwidth utilization.

[0019] According to a communication delay time prediction method provided by the present invention, the transmission delay time of the communication system to be predicted is calculated according to the number of processing nodes and the bandwidth utilization rate, including:

[0020] Calculating the fault tolerance performance parameter of the communication system to be predicted according to the number of processing nodes;

[0021] The product of the fault tolerance performance parameter and the bandwidth utilization is calculated to obtain the transmission delay time.

[0022] According to a communication delay time prediction method provided by the present invention, the total number of data jump transmissions in the communication system to be predicted is obtained based on the following steps:

[0023] Performing statistics on the number of communication links and the number of processing nodes according to the network topology information of the communication system to be predicted, to obtain the number of communication links and the number of processing nodes in the communication system to be predicted;

[0024] The total number of data jump transmissions is calculated based on the number of data blocks transmitted by the communication system to be predicted, the number of communication links and the number of processing nodes.

[0025] According to a communication delay time prediction method provided by the present invention, the total number of data jump transmissions is calculated according to the number of data blocks transmitted by the communication system to be predicted, the number of communication links and the number of processing nodes, including:

[0026] Calculating a logarithmic value between the number of communication links and the number of processing nodes;

[0027] The product of the number of data blocks and the logarithmic value is calculated to obtain the total number of data jump transmissions.

[0028] According to a communication delay time prediction method provided by the present invention, the startup delay coefficient fitting is performed according to the measured network communication parameters to obtain the target startup delay coefficient corresponding to each data jump transmission in the communication system to be predicted, including:

[0029] Based on the target fitting model, the measured network communication parameters are fitted to obtain a fitting loss value corresponding to the target fitting model;

[0030] With the goal of minimizing the fitting loss value, the target fitting model is optimized to obtain an initial startup delay coefficient corresponding to each data jump transmission in the communication system to be predicted;

[0031] The target startup delay coefficient is obtained according to the initial startup delay coefficient.

[0032] According to a communication delay time prediction method provided by the present invention, obtaining the target startup delay coefficient according to the initial startup delay coefficient includes:

[0033] Calculating an average value of initial startup delay coefficients corresponding to each data jump transmission in the sample communication system under a plurality of different topological structures to obtain an average startup delay coefficient;

[0034] The target startup delay coefficient is obtained according to the average startup delay coefficient.

[0035] According to a communication delay time prediction method provided by the present invention, the measured network communication parameters are fitted based on a target fitting model to obtain a fitting loss value corresponding to the target fitting model, including:

[0036] In the measured network communication parameters, the data volume and total delay time measured values ​​of a plurality of sample data blocks of different sizes are obtained;

[0037] For the sample data block of any size, inputting the data amount of the sample data block into the target fitting model to obtain a first total delay time prediction value of the sample data block;

[0038] Determine a loss value corresponding to the sample data block according to the total delay time actually measured value of the sample data block and a deviation between the first total delay time predicted value of the sample data block and the total delay time actually measured value of the sample data block;

[0039] The sample data blocks of various sizes are traversed to obtain loss values ​​corresponding to the multiple sample data blocks, and the loss values ​​corresponding to the multiple sample data blocks are merged to obtain a fitting loss value corresponding to the target fitting model.

[0040] According to a communication delay time prediction method provided by the present invention, obtaining the target startup delay coefficient according to the initial startup delay coefficient includes:

[0041] Randomly updating the initial startup delay coefficient to obtain a plurality of updated startup delay coefficients;

[0042] According to each of the updated start delay coefficients, obtaining second total delay time prediction values ​​corresponding to the sample data blocks of different sizes;

[0043] Obtaining fitting performance parameters corresponding to each of the updated start-up delay coefficients according to the second total delay time prediction value, data volume and total delay time measured value corresponding to the sample data blocks of different sizes;

[0044] According to the fitting performance parameter, obtaining a startup delay coefficient with the best fitting performance from a plurality of the updated startup delay coefficients;

[0045] The target startup delay coefficient is obtained according to the startup delay coefficient with the best fitting performance.

[0046] According to a communication delay time prediction method provided by the present invention, obtaining the fitting performance parameters corresponding to each of the updated startup delay coefficients according to the second total delay time prediction value, data volume and total delay time measured value corresponding to the sample data blocks of different sizes, includes:

[0047] Obtaining average delay times corresponding to the sample data blocks of different sizes according to a ratio between a second total delay time prediction value corresponding to the sample data blocks of different sizes and a data amount;

[0048] Obtaining delay time errors corresponding to the sample data blocks of different sizes according to deviations between total delay time measured values ​​corresponding to the sample data blocks of different sizes and the average delay time;

[0049] The fitting performance parameters corresponding to each of the updated start-up delay coefficients are obtained according to the total delay time measured values ​​corresponding to the sample data blocks of different sizes and the delay time error.

[0050] According to a communication delay time prediction method provided by the present invention, the startup delay time analysis is performed according to the target startup delay coefficient and the total number of data jump transmissions in the communication system to be predicted to obtain the startup delay time of the communication system to be predicted, including:

[0051] In the measured network communication parameters, the data volume corresponding to each target data block and the measured value of the total delay time corresponding to each target data block are obtained; the target data block is a sample data block in the measured network communication parameters whose data volume is less than the target value;

[0052] According to the data amount corresponding to each of the target data blocks and the measured value of the total delay time corresponding to each of the target data blocks, the startup delay offset corresponding to the data jump transmission in the communication system to be predicted is fitted to obtain the startup delay offset corresponding to the data jump transmission in the communication system to be predicted;

[0053] The target startup delay coefficient and the total number of data jump transmissions are multiplied, and the multiplication result is added to the startup delay offset to obtain the startup delay time.

[0054] The present invention also provides a communication delay time prediction system, comprising:

[0055] A fitting unit is used to measure the network communication parameters of the sample communication system to obtain the measured network communication parameters of the sample communication system, and to fit the startup delay coefficient according to the measured network communication parameters to obtain the target startup delay coefficient corresponding to each data jump transmission in the communication system to be predicted; the sample communication system and the communication system to be predicted are both communication systems constructed by multiple processing nodes;

[0056] A first acquisition unit, configured to perform a startup delay time analysis according to the target startup delay coefficient and the total number of data jump transmissions in the communication system to be predicted, to obtain the startup delay time of the communication system to be predicted;

[0057] A second acquisition unit is used to perform transmission delay time analysis according to the network configuration parameters of the communication system to be predicted and the amount of data transmitted by the communication system to be predicted, so as to obtain the transmission delay time of the communication system to be predicted;

[0058] The prediction unit is used to predict the total delay time according to the startup delay time and the transmission delay time to obtain a predicted value of the total delay time of the communication system to be predicted.

[0059] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, a communication delay time prediction method as described above is implemented.

[0060] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the communication delay time prediction method as described in any one of the above is implemented.

[0061] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the communication delay time prediction method as described above is implemented.

[0062] The communication delay time prediction method, system, electronic device and storage medium provided by the present invention obtain a target startup delay coefficient by fitting with the measured network communication parameters of a sample communication system, and on the basis of the target startup delay coefficient, adaptive prediction of the end-to-end total delay time of the communication system to be predicted in different scenarios is performed in combination with multi-dimensional characteristics such as the total number of data jump transmissions and network configuration parameters, and the amount of transmitted data of the communication system to be predicted in different scenarios, thereby avoiding the need to build a physical model for each scenario for actual experimental testing, and predicting the total delay time of the communication system to be predicted in multiple scenarios, thereby effectively reducing the configuration cost while achieving accurate prediction of the communication delay time, and effectively solving the problem of high cost of obtaining end-to-end delay time in related technologies. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0064] Figure 1 It is a flow chart of the communication delay time prediction method provided by the present invention.

[0065] Figure 2 It is a schematic diagram of the structure of a sample communication system provided by the present invention.

[0066] Figure 3 It is a distribution scatter plot of actually measured network communication parameters under different numbers of processing nodes provided by the present invention.

[0067] Figure 4 It is a flowchart of the ring communication provided by the present invention.

[0068] Figure 5 This is one of the simulation schematic diagrams of the communication delay time prediction method provided by the present invention.

[0069] Figure 6 This is the second simulation schematic diagram of the communication delay time prediction method provided by the present invention.

[0070] Figure 7 This is the third simulation schematic diagram of the communication delay time prediction method provided by the present invention.

[0071] Figure 8 It is a structural schematic diagram of the communication delay time prediction system provided by the present invention.

[0072] Fig. 9 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0073] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0074] Collective communication refers to a communication mode in parallel computing and distributed computing in communication systems, which involves data exchange and coordination operations between multiple participants (usually processing units). Unlike point-to-point communication, collective communication involves synchronization and data sharing operations among multiple participants, and is widely used in scientific computing, high performance computing (HPC), machine learning and other fields. In the traditional central processing unit (CPU) high performance computing field, collective communication is usually implemented by specialized communication libraries, such as the message passing interface (MPI) library and the open multi-processing (OpenMP) library. These communication libraries provide efficient collective communication primitives, allowing developers to focus more on the algorithm itself when writing parallel programs without having to worry too much about the underlying communication details.

[0075] Collective communication in the field of deep learning occurs between multiple computing units (also known as computing cards or computing nodes). Due to the rich connection topology between computing units, many computing equipment manufacturers have developed their own underlying collective communication libraries, such as NVIDIA's cluster communication library (NCCL). Collective communication includes multiple operation modes (also known as communication primitives). In the field of deep learning parallel computing, the two most important operations are global reduction operation (also known as Allreduce) and full exchange operation (also known as all-to-all). Allreduce is a common reduction operation that combines the input data of multiple participants according to a certain reduction operation (such as summation, maximum value, minimum value, etc.) and returns the result to all participants. Specifically, each participant first sends its own data to all other participants, and then the data is merged in the reduction operation to finally obtain a global result, which all participants will obtain. All-to-All is a more complex collective communication mode. In the All-to-All operation, each participant sends data to all other participants and receives data from all other participants. The result of this operation is that each participant obtains information from all other participants.

[0076] To simplify the description, the following Allreduce collective communication operation for deep learning parallel training is used as an actual application scenario to describe the communication delay time prediction method provided by the implementation of this application. For other scenarios, you can refer to this scenario and make adaptive reference modifications.

[0077] Allreduce operation plays a key role in deep learning and large-scale model training, especially in distributed training. Its main function is to synchronize and share data between various computing units in a distributed computing environment, ensuring the consistency of model parameters and the effectiveness of training. The parallel methods that use Allreduce collective communication in deep learning include: data parallelism and model parallelism. For data parallelism, it is used to ensure that each computing unit has the same copy of the model but processes different data subsets. Each node independently calculates the gradient of its data subset, and then uses the Allreduce operation to average or merge the gradients of all nodes. Finally, all nodes update their model parameters so that the model of each node is consistent. For model parallelism, it is used to divide the model into different parts, and each node is responsible for calculating the forward and backward propagation of a part of the model; tensor parallel computing in model parallelism requires frequent use of Allreduce operations to synchronize checkpoints.

[0078] However, since parallel computing of multiple processing units will bring additional communication delay time, how to accurately predict the communication delay time of multiple processing units in order to optimize the design of parallel communication algorithms for multiple processing units, reduce the communication ratio, and greatly improve computing efficiency, is an important issue that needs to be urgently solved in the industry.

[0079] In the related art, when obtaining the end-to-end delay time for a certain scenario, it is necessary to build a physical model for the scenario, and use the communication test tool of the communication library on the physical model to design a fixed topology for a specific data block size, and perform actual experimental tests on the physical model to obtain the actual end-to-end delay data for the scenario. Although the use of the test tool can very objectively reflect the end-to-end delay data of the current system, it must physically implement the delay data test in the scenario, and it is necessary to perform experimental tests for all data block sizes. When we need to perform a total delay performance test for different scenarios, because it is implemented on the physical model, it is impossible to quickly modify the physical model and perform delay data tests for other scenarios based on the physical model built in a certain scenario, but it is necessary to re-build the corresponding physical models for other scenarios one by one, which leads to high configuration costs.

[0080] In this regard, the embodiment of the present application provides a communication delay time prediction method, which is based on a modern GPU and uses the NCCL library to test different scale data of different sample communication systems to obtain end-to-end total delay data. The end-to-end total delay data is fitted to obtain the relevant parameters of the cost model, and then a set of cost model related parameters methodology and a cost model for a computing device with a fixed topology are formed, so as to accurately evaluate the total delay time corresponding to different data blocks based on this cost model, while avoiding the construction of an actual physical model, but modifying the parameters on the cost model to simulate and predict the end-to-end total delay of different topologies and hardware systems, so as to effectively reduce the configuration cost.

[0081] Figure 1 is a flow chart of the communication delay time prediction method provided by the present invention; Figure 1 As shown, the method includes: step 110, step 120, step 130 and step 140.

[0082] Step 110, measuring the network communication parameters of the sample communication system to obtain the measured network communication parameters of the sample communication system, and fitting the startup delay coefficient according to the measured network communication parameters to obtain the target startup delay coefficient corresponding to each data jump transmission in the communication system to be predicted; the sample communication system and the communication system to be predicted are both communication systems constructed by multiple processing nodes.

[0083] Figure 2 Schematic diagram of the structure of the sample communication system provided by the present invention. Figure 2 As shown, the sample communication system here includes multiple processing units (also called processing nodes), and the multiple processing units are fully interconnected through a switch, such as an InfiniBand Switch (IB Switch); the processing unit here can be an AMD graphics processing unit (AMD GPU), a CPU, a computing node, etc., and this embodiment does not specifically limit this. Here, the processing unit is an AMD GPU as an example for description. The configuration model of the AMD GPU here can be NF6488A5 and A800.

[0084] It should be noted that during the testing of the sample communication system, since the sample communication system will provide its own parallel communication library when completing parallel computing, such as the NCCL library configured for NVIDIA GPU and the RCCL library configured for AMD GPU, during the testing of the sample communication system, the parallel communication library configured by the sample communication system itself can be used, and its Application Programming Interface (API) can be used to test the measured network communication parameters of the sample communication system under fixed communication operators and data blocks of different sizes under fixed topology.

[0085] Optionally, a suitable test tool may be selected to test the end-to-end delay data of the sample communication system; during the test, the end-to-end delay data of the test sample communication system may be actually measured for different network topology information and sample data blocks of different sizes.

[0086] Figure 3 It is a distribution scatter plot of actually measured network communication parameters under different numbers of processing nodes provided by the present invention.

[0087] Exemplarily, the end-to-end delay data of the sample communication system is tested using the NCCL test tool. 16 GPUs are used in the test, the collective communication operator uses the Allreduce operator, and the in-place operation time (In-place OperationTime) is used as the total delay time measured value of data blocks of different sizes, thereby obtaining the delay time measured values ​​under sample data blocks of different sizes, and thus constructing the measured network communication parameters of the sample communication system according to the data volume corresponding to the sample data blocks of different sizes and the corresponding delay time measured values; exemplarily, the distribution scatter plot of the measured network communication parameters under different processing units (such as GPUs) is as follows: Figure 3 shown.

[0088] The Allreduce operator here can be a bidirectional tree communication operator (also called bidirectional tree-allreduce), a ring communication operator (also called ring-allreduce), a topology-aware operator, etc., which can be determined according to actual needs, and this embodiment does not specifically limit this.

[0089] Figure 4 It is a schematic diagram of the flow of the ring communication provided by the present invention; Figure 4As shown in the figure, ring-allreduce is a distributed algorithm that implements the Allreduce operation, which is particularly suitable for use in systems with ring topology or fully interconnected networks. It completes the protocol and broadcast by arranging all participants into a logical ring and exchanging data through multiple communication rounds. Each node sends a data block to the next node in each round and receives a data block from the previous node. This method has the advantage of communication load balancing, and the communication workload of each node is similar.

[0090] After obtaining the measured network communication parameters of the sample communication system, parameter fitting can be performed based on the measured network communication parameters of the sample communication system to obtain the target startup delay coefficient corresponding to each data jump transmission in the communication system to be predicted. The target startup delay coefficient here is used to characterize the startup delay coefficient corresponding to the startup time such as protocol unpacking and network device startup generated during each data jump transmission.

[0091] After obtaining the target startup delay coefficient, an optimized cost model can be constructed based on the target startup delay coefficient; for example, the optimized cost model can be obtained by updating the cost model to be optimized (i.e., the target fitting model) based on the target startup delay coefficient, or the optimized cost model can be obtained by updating the cost model to be optimized based on the target startup delay coefficient and then fine-tuning it. This embodiment does not specifically limit this. The cost model (Cost-model) here refers to a calculation model that can calculate the end-to-end total delay time given a fixed-size data block. When the collective communication algorithm transmits fixed-size data, it divides the data into small blocks. Each time a small block is transmitted, the total delay time includes at least the startup delay time (indicating the fixed overhead of a message transmission, including the initialization of the message and the startup time of the network device) and the data transmission delay time (indicating the actual transmission time when transmitting data, generally related to the actual interconnection bandwidth). Therefore, based on the optimized cost model, the predicted value of the end-to-end total delay time of the communication system to be predicted in each scenario can be adaptively estimated.

[0092] The communication system to be predicted here can be a communication system that has been mass-produced or a non-mass-produced communication system, such as a communication system for which a physical model is to be built. By calculating the predicted values ​​of the delay time under different topologies and different data blocks, the communication proportion of the communication system during deep learning training and reasoning can be determined, and then the optimal topology and the optimal parallel communication algorithm of the communication system can be optimized and determined. Based on the optimal topology and the optimal parallel communication algorithm, a physical model of the communication system can be constructed to improve the actual transmission efficiency and stability of the communication system and reduce development costs.

[0093] The fitting here may be obtained based on the least squares algorithm fitting, or may be obtained based on the least squares algorithm fitting and then optimized based on the Monte Carlo algorithm, etc. This embodiment does not specifically limit this.

[0094] Step 120: Perform a startup delay time analysis based on the target startup delay coefficient and the total number of data jump transmissions in the communication system to be predicted to obtain the startup delay time of the communication system to be predicted.

[0095] The total number of data jump transmissions here refers to the number of intermediate jump transmissions that the data undergoes when it is transmitted from the starting processing node to the ending processing node of the communication system to be predicted; specifically, the path passed by the data transmission process and the number of intermediate processing nodes contained in this path can be determined based on the network topology information of the communication system to be predicted, and the total number of data jump transmissions can be inferred from this.

[0096] Optionally, after obtaining the target startup delay coefficient, the optimized cost model can be combined with the target startup delay coefficient and the total number of data jump transmissions in the communication system to be predicted to estimate the startup delay time of the communication system to be predicted in startup scenarios such as protocol unpacking and network equipment startup.

[0097] The estimation method of the startup delay time here includes performing logical calculation on the target startup delay coefficient and the total number of data jump transmissions in the communication system to be predicted to obtain the startup delay time of the communication system to be predicted, such as non-linear calculation; or, based on a machine learning algorithm, applying the target startup delay coefficient and the total number of data jump transmissions in the communication system to be predicted to evaluate and obtain the startup delay time of the communication system to be predicted. This embodiment does not specifically limit this.

[0098] Step 130, performing transmission delay time analysis according to the network configuration parameters of the communication system to be predicted and the amount of data transmitted by the communication system to be predicted, to obtain the transmission delay time of the communication system to be predicted;

[0099] The network configuration parameters here refer to a series of parameters used to characterize and optimize network performance, structure and behavior. These parameters are crucial to ensure the effective operation of the network, optimize data transmission efficiency and improve overall system performance, and include but are not limited to network topology information, bandwidth information and throughput, and protocol parameters, which are not specifically limited in this embodiment.

[0100] The amount of data transmitted by the communication system to be predicted here refers to the number or size of data blocks actually required to be transmitted by the communication system to be predicted; specifically, it can be obtained by actually counting or measuring the data blocks.

[0101] Optionally, the optimized cost model may be combined with network configuration parameters of the communication system to be predicted and the amount of data transmitted by the communication system to be predicted to calculate the transmission delay time generated by the communication system to be predicted during data transmission.

[0102] The method for determining the transmission delay time here includes performing logical calculations on the network configuration parameters of the communication system to be predicted and the amount of data transmitted by the communication system to be predicted to obtain the transmission delay time, such as nonlinear calculation; or based on a machine learning algorithm, applying the network configuration parameters of the communication system to be predicted and the amount of data transmitted by the communication system to be predicted to evaluate and obtain the transmission delay time of the communication system to be predicted. This embodiment does not make specific limitations on this.

[0103] Step 140: predicting the total delay time according to the startup delay time and the transmission delay time to obtain a predicted total delay time value of the communication system to be predicted.

[0104] Optionally, after obtaining the startup delay time and the transmission delay time, the optimized cost model can perform fusion calculation operations such as direct addition or weighted addition on the startup delay time and the transmission delay time to predict the end-to-end total delay time prediction value of the communication system to be predicted.

[0105] In the prior art, bandwidth data and corresponding communication algorithms are usually used to estimate the total delay time. For example, for the ring-allreduce operator, the general delay estimation method is: , where N is the number of processing units, M is the size of the transmitted data block, and Bandwidth is the unidirectional bandwidth on the link. However, there are many factors that affect the determination of the delay time of the communication system, such as the topology structure, the total number of data jump transmissions, etc. If the total delay time is estimated only based on bandwidth data, the end-to-end total delay time cannot be accurately estimated. Especially when the transmission data block is small, using bandwidth data to estimate the end-to-end total delay time will produce a large deviation, which makes it difficult to guarantee the accuracy of the calculation of the end-to-end total delay time.

[0106] In the embodiment of the present application, the startup delay time of the communication system to be predicted is estimated by fitting the target startup delay coefficient obtained by combining the measured network communication parameters of the sample communication system and the total number of data jump transmissions in the communication system to be predicted, and the transmission delay time of the communication system to be predicted is estimated by combining the network configuration parameters of the communication system to be predicted and the amount of data transmitted by the communication system to be predicted. Finally, the startup delay time and the transmission delay time are combined to estimate the end-to-end total delay time. By comprehensively considering multiple dimensions such as the measured network communication parameters, the total number of data jump transmissions, the network configuration parameters, and the data transmission volume, compared with the existing technology that only relies on bandwidth data, it is possible to more comprehensively and accurately estimate the end-to-end total delay time of the communication system, especially when the transmission data block is small or there is a complex network structure, and the accuracy of the delay time estimation is significantly improved.

[0107] The method provided by the implementation of the present application obtains a target startup delay coefficient by fitting with the measured network communication parameters of a sample communication system, and on the basis of the target startup delay coefficient, the total end-to-end delay time of the communication system to be predicted in different scenarios is adaptively predicted in combination with multi-dimensional characteristics such as the total number of data jump transmissions and network configuration parameters, and the amount of transmitted data of the communication system to be predicted in different scenarios, thereby avoiding the need to build a physical model for each scenario for actual experimental testing, and thus predicting the total delay time of the communication system to be predicted in multiple scenarios, thereby effectively reducing the configuration cost while achieving accurate prediction of the communication delay time, and effectively solving the problem of high cost of acquiring end-to-end delay time in related technologies.

[0108] In some embodiments, the total number of data jump transmissions in the communication system to be predicted is obtained based on the following steps:

[0109] Performing statistics on the number of communication links and the number of processing nodes according to the network topology information of the communication system to be predicted, to obtain the number of communication links and the number of processing nodes in the communication system to be predicted;

[0110] The total number of data jump transmissions is calculated based on the number of data blocks transmitted by the communication system to be predicted, the number of communication links and the number of processing nodes.

[0111] The processing node here includes but is not limited to a central processing unit (CPU) core, a graphics processing unit (GPU) or a computing node, and this implementation does not specifically limit this.

[0112] Optionally, the step of acquiring the total number of data jump transmissions in the communication system to be predicted in step 120 includes:

[0113] The optimized cost model calculates the number of communication links between processing nodes in the communication system to be predicted based on the network topology information of the communication system to be predicted, and counts the number of processing nodes. The total number of data jump transmissions is estimated by combining the number of data blocks transmitted by the communication system to be predicted, the number of communication links, and the number of processing nodes.

[0114] The estimation method of the total number of data jump transmissions here includes performing logical calculations on the number of data blocks transmitted by the predicted communication system, the number of communication links, and the number of processing nodes to obtain the total number of data jump transmissions, such as nonlinear calculations; or based on a machine learning algorithm, applying the number of data blocks transmitted by the predicted communication system, the number of communication links, and the number of processing nodes to be predicted, to evaluate and obtain the total number of data jump transmissions of the predicted communication system, which is not specifically limited in this embodiment.

[0115] Exemplarily, in some embodiments, the step of obtaining the total number of data jump transmissions specifically includes:

[0116] Calculating a logarithmic value between the number of communication links and the number of processing nodes;

[0117] The product of the number of data blocks and the logarithmic value is calculated to obtain the total number of data jump transmissions.

[0118] Optionally, the logarithmic value between the number of communication links and the number of processing nodes may be calculated using the optimized cost model, and the product between the number of data blocks and the logarithmic value may be calculated to obtain the total number of data jump transmissions.

[0119] The method provided in this embodiment accurately estimates the total number of data jump transmissions by utilizing multi-dimensional characteristics of the communication system to be predicted, such as the network topology information, number of data blocks, number of communication links, and number of processing nodes. The total number of data jump transmissions obtained thereby can effectively improve the accuracy and efficiency of the communication system delay time prediction.

[0120] In some embodiments, step 130 specifically includes:

[0121] Performing parameter parsing on the network configuration parameters to obtain network topology information of the communication system to be predicted and bandwidth data of each communication link in the communication system to be predicted;

[0122] The transmission delay time of the communication system to be predicted is calculated according to the network topology information, the bandwidth data and the amount of data transmitted by the communication system to be predicted.

[0123] Optionally, the step of acquiring the transmission delay time specifically includes:

[0124] The network configuration parameters are parsed by the optimized cost model to obtain network topology information of the communication system to be predicted and bandwidth data of each communication link in the communication system to be predicted.

[0125] Furthermore, the optimized cost model is combined with network topology information, bandwidth data, and the amount of data transmitted by the communication system to be predicted to estimate the transmission delay time.

[0126] The transmission delay time estimation method here includes performing logical calculations on network topology information, bandwidth data, and the amount of data transmitted by the communication system to be predicted to obtain the transmission delay time, such as nonlinear calculation; or based on a machine learning algorithm, applying network topology information, bandwidth data, and the amount of data transmitted by the communication system to be predicted to evaluate and obtain the transmission delay time of the communication system to be predicted. This embodiment does not specifically limit this.

[0127] Exemplarily, in some embodiments, the step of acquiring the transmission delay time specifically includes:

[0128] Perform statistics on the number of communication links and the number of processing nodes according to the network topology information to obtain the number of communication links and the number of processing nodes in the communication system to be predicted;

[0129] Calculating the product of the number of communication links and the bandwidth data to obtain the total bandwidth data of the communication system to be predicted;

[0130] Calculating the ratio between the amount of data transmitted by the communication system to be predicted and the total bandwidth data to obtain the bandwidth utilization rate of the communication system to be predicted;

[0131] The transmission delay time of the communication system to be predicted is calculated according to the number of processing nodes and the bandwidth utilization.

[0132] Optionally, the optimized cost model calculates the number of communication links between processing nodes in the communication system to be predicted based on the network topology information of the communication system to be predicted, and counts the number of processing nodes. The product between the number of communication links and the bandwidth data is calculated to obtain the total bandwidth data of the communication system to be predicted; the ratio between the amount of data transmitted by the communication system to be predicted and the total bandwidth data is calculated to obtain the bandwidth utilization of the communication system to be predicted; and the number of processing nodes and the bandwidth utilization are combined to estimate the corresponding transmission delay time.

[0133] The method for obtaining the transmission delay time here includes: performing logical calculation on the number of processing nodes and bandwidth utilization to obtain the transmission delay time, such as nonlinear calculation; or based on a machine learning algorithm, applying the number of processing nodes and bandwidth utilization to evaluate and obtain the transmission delay time of the communication system to be predicted. This embodiment does not specifically limit this.

[0134] Exemplarily, in some embodiments, the step of acquiring the transmission delay time further includes:

[0135] Calculating the fault tolerance performance parameter of the communication system to be predicted according to the number of processing nodes;

[0136] The product of the fault tolerance performance parameter and the bandwidth utilization is calculated to obtain the transmission delay time.

[0137] Optionally, the optimized cost model subtracts the number of processing nodes from the preset fault tolerance number and then divides the result by the number of processing nodes to obtain a fault tolerance performance parameter of the communication system to be predicted; and the product between the fault tolerance performance parameter and the bandwidth utilization is calculated to obtain the transmission delay time. The preset fault tolerance number here can be set according to actual needs, such as the preset fault tolerance number is 1, etc., and this embodiment does not specifically limit this.

[0138] The method provided in this embodiment uses network configuration parameters to obtain network topology information and bandwidth data, and comprehensively and adaptively estimates the transmission delay time in combination with the amount of data transmitted by the communication system to be predicted, so as to comprehensively and accurately evaluate the transmission delay time of the communication system, thereby ensuring the accuracy of the end-to-end total delay time estimation of the communication system.

[0139] In some embodiments, step 110 specifically includes:

[0140] Based on the target fitting model, the measured network communication parameters are fitted to obtain a fitting loss value corresponding to the target fitting model;

[0141] With the goal of minimizing the fitting loss value, the target fitting model is optimized to obtain an initial startup delay coefficient corresponding to each data jump transmission in the communication system to be predicted;

[0142] The target startup delay coefficient is obtained according to the initial startup delay coefficient.

[0143] The target fitting model here can be a cost model to be optimized, in which the startup delay coefficient is a parameter to be optimized, which can be obtained by initializing, refining, updating and correcting the Allreduce collective communication based on the communication library. Since the Allreduce algorithm implemented by the collective communication library is generally a topology-aware communication algorithm, the corresponding Allreduce operator can be selected according to the size of the data block and the network topology information of the actual communication system to implement the initialization cost model construction. Since the ring-allreduce operator and the bidirectional tree-allreduce operator are generally used in deep learning scenarios for modern computing devices, the following initialization cost model can be constructed to simulate the end-to-end total delay time of different data block sizes:

[0144] ;

[0145] In the formula, The startup delay time is the total delay time of protocol unpacking, network device startup, etc. generated when each communication data block is generated. is the inverse of the communication bandwidth, and H is the amount of data communicated; is the total end-to-end delay corresponding to the transmission of H data volume.

[0146] Furthermore, the initialization cost model is refined to obtain the following updated cost model:

[0147] ;

[0148] in, The startup delay coefficient required to be optimized corresponding to each data jump transmission in the communication system; It is the total number of data jump transmissions in the communication system.

[0149] Furthermore, the update cost model can be modified according to the general implementation of Allreduce in the communication library to obtain the following cost model to be optimized:

[0150] ;

[0151] in, The number of data blocks transmitted by the communication system, which can be based on Calculated; is the number of communication links (also called link number) between processing nodes in the communication system; is the number of processing nodes in the communication system, that is, the total number of computing devices; It is the bandwidth data of each communication link in the communication system, that is, the one-way bandwidth of each link; The amount of data transmitted by a communication system.

[0152] It should be noted that due to the startup delay coefficient in the cost model to be optimized is the parameter to be optimized; in the related art, although the parameter to be optimized can be determined based on the ideal data in the specification provided by the communication system manufacturer, there is often a certain deviation between the data actually used and the ideal data, so it is impossible to accurately determine the parameter to be optimized; in this regard, the embodiment of the present application determines the startup delay coefficient by fitting the measured data , thereby improving the accuracy of parameter optimization and ensuring the accuracy of end-to-end total delay time prediction.

[0153] Optionally, the target fitting model is applied to fit the measured network communication parameters by the least square method with weighted coefficients, thereby obtaining a fitting loss value corresponding to the target fitting model. The fitting loss value here is used to characterize the difference between the total delay time predicted by the target fitting model and the total delay time measured in the measured network communication parameters.

[0154] Exemplarily, in some embodiments, the step of obtaining the fitting loss value specifically includes:

[0155] In the measured network communication parameters, the data volume and total delay time measured values ​​of a plurality of sample data blocks of different sizes are obtained;

[0156] For the sample data block of any size, inputting the data amount of the sample data block into the target fitting model to obtain a first total delay time prediction value of the sample data block;

[0157] Determine a loss value corresponding to the sample data block according to the total delay time actually measured value of the sample data block and a deviation between the first total delay time predicted value of the sample data block and the total delay time actually measured value of the sample data block;

[0158] The sample data blocks of various sizes are traversed to obtain loss values ​​corresponding to the multiple sample data blocks, and the loss values ​​corresponding to the multiple sample data blocks are merged to obtain a fitting loss value corresponding to the target fitting model.

[0159] Optionally, the measured network communication parameters are parsed to obtain the data volume and total delay time measured values ​​corresponding to sample data blocks of different sizes to form a test sample pair. For example, the test sample pair can be characterized as: ,in, For the sample data block The corresponding data volume; For the sample data block The corresponding total delay time is measured.

[0160] After obtaining the test sample pairs, the data volume corresponding to the sample data blocks of different sizes can be input into the target fitting model, and the target fitting model predicts the first total delay time prediction value corresponding to the sample data blocks of different sizes based on the data volume corresponding to the sample data blocks of different sizes.

[0161] In addition, the deviation value between the first total delay time prediction value corresponding to the sample data blocks of different sizes and the total delay time actual value corresponding to the sample data blocks of different sizes is calculated, and the deviation value is multiplied by the inverse of the total delay time actual value corresponding to the sample data blocks of different sizes to obtain the loss value corresponding to the sample data blocks of different sizes; and the loss values ​​corresponding to multiple sample data blocks of different sizes are summed to obtain the fitting loss value.

[0162] Exemplarily, the specific calculation formula of the fitting loss value is as follows:

[0163] ;

[0164] in, is the fitting loss value; is the number of sample data blocks; For the sample data block The corresponding total delay time is measured.

[0165] After obtaining the fitting loss value according to the above-mentioned fitting loss function calculation formula, the fitting loss value can be minimized as the goal, and the parameters to be optimized (that is, the startup delay coefficient) in the target fitting model can be optimized and adjusted until the fitting loss value converges to the preset threshold. The adjusted startup delay coefficient that makes the fitting loss value converge to the preset threshold is used as the initial startup delay coefficient corresponding to each data jump transmission in the communication system to be predicted.

[0166] After obtaining the initial startup delay coefficient corresponding to each data jump transmission in the communication system to be predicted, the final startup delay coefficient, that is, the target startup delay coefficient, can be obtained based on the initial startup delay coefficient corresponding to each data jump transmission in the communication system to be predicted.

[0167] Here, methods for determining the target startup delay coefficient include directly using the initial startup delay coefficient as the target startup delay coefficient, or correcting the initial startup delay coefficient, such as using a Monte Carlo algorithm to correct the initial startup delay coefficient, or obtaining initial startup delay coefficients corresponding to multiple sets of network configuration parameters and fusing them to obtain the target startup delay coefficient. This embodiment does not specifically limit this.

[0168] The method provided in this embodiment optimizes the startup delay coefficient by fitting measured data, thereby improving the accuracy of parameter optimization, thereby accurately predicting the end-to-end total delay time in the communication system to be predicted.

[0169] In some embodiments, the step of obtaining the target startup delay coefficient specifically further includes:

[0170] Calculating an average value of initial startup delay coefficients corresponding to each data jump transmission in the sample communication system under a plurality of different topological structures to obtain an average startup delay coefficient;

[0171] The target startup delay coefficient is obtained according to the average startup delay coefficient.

[0172] By changing the number of processing nodes and the number of communication links of the sample communication system, a plurality of sample communication systems under different topological structures are formed, and the measured network communication parameters of the sample communication systems under multiple groups of different topological structures are collected. According to the fitting steps mentioned above, the measured network communication parameters of the sample communication system under each topological structure are fitted to obtain the initial startup delay coefficient corresponding to each data jump transmission in the sample communication systems under multiple different topological structures. Since the startup delay time is basically the same regardless of the number of processing nodes, the initial startup delay coefficient corresponding to each data jump transmission in the sample communication systems under multiple groups of different topological structures can be averaged to obtain the average startup delay coefficient, and the average startup delay coefficient is used as the target startup delay coefficient.

[0173] The method provided in this embodiment constructs multiple groups of sample communication systems with different topological structures and collects measured network communication parameters, then fits the initial startup delay coefficient of each data jump transmission under each topological structure, and calculates the average value of these coefficients as the target startup delay coefficient, thereby effectively achieving accurate acquisition of the target startup delay coefficient, thereby accurately predicting the end-to-end total delay time in the communication system to be predicted.

[0174] In some embodiments, the step of obtaining the target startup delay coefficient specifically further includes:

[0175] Randomly updating the initial startup delay coefficient to obtain a plurality of updated startup delay coefficients;

[0176] According to each of the updated start delay coefficients, obtaining second total delay time prediction values ​​corresponding to the sample data blocks of different sizes;

[0177] Obtaining fitting performance parameters corresponding to each of the updated start-up delay coefficients according to the second total delay time prediction value, data volume and total delay time measured value corresponding to the sample data blocks of different sizes;

[0178] According to the fitting performance parameter, obtaining a startup delay coefficient with the best fitting performance from a plurality of the updated startup delay coefficients;

[0179] The target startup delay coefficient is obtained according to the startup delay coefficient with the best fitting performance.

[0180] It should be noted that, based on the initial startup delay coefficient obtained based on the least squares algorithm, the Monte Carlo method can also be used to correct the initial startup delay coefficient to further improve the accuracy of parameter estimation and enhance the robustness of the cost model.

[0181] Optionally, based on the initial startup delay coefficient, the initial startup delay coefficient is randomly updated to obtain multiple updated startup delay coefficients; and based on the updated startup delay coefficient, the startup delay coefficient in the cost model to be optimized is updated, so as to predict the second total delay time prediction value corresponding to the sample data blocks of different sizes by applying the data amount corresponding to the sample data blocks of different sizes based on the updated cost model to be optimized.

[0182] After obtaining the second total delay time prediction value corresponding to sample data blocks of different sizes, the second total delay time prediction value corresponding to sample data blocks of different sizes, the data volume and the actual measured value of the total delay time can be combined to evaluate the fitting performance corresponding to each updated startup delay coefficient to obtain the fitting performance parameters corresponding to each updated startup delay coefficient.

[0183] The fitting performance parameters here can be obtained by performing logical calculations on the second total delay time prediction value, the data volume and the total delay time actual measured value, such as nonlinear calculations, or by evaluating the second total delay time prediction value, the data volume and the total delay time actual measured value based on a machine learning model. This embodiment does not specifically limit this.

[0184] Exemplarily, in some embodiments, the step of acquiring the fitting performance parameter specifically includes:

[0185] Obtaining average delay times corresponding to the sample data blocks of different sizes according to a ratio between a second total delay time prediction value corresponding to the sample data blocks of different sizes and a data amount;

[0186] Obtaining delay time errors corresponding to the sample data blocks of different sizes according to deviations between total delay time measured values ​​corresponding to the sample data blocks of different sizes and the average delay time;

[0187] The fitting performance parameters corresponding to each of the updated start-up delay coefficients are obtained according to the total delay time measured values ​​corresponding to the sample data blocks of different sizes and the delay time error.

[0188] Optionally, when calculating the fitting performance parameters, the ratio between the second total delay time prediction value corresponding to sample data blocks of different sizes and the data amount corresponding to the sample data blocks of different sizes can be calculated first to obtain the average delay time corresponding to the sample data blocks of different sizes; then, the sum of the deviations between the actual measured values ​​of the total delay time corresponding to the sample data blocks of different sizes and the average delay time corresponding to the sample data blocks of different sizes is calculated, and the mean square error of the actual measured values ​​of the total delay time corresponding to the sample data blocks of different sizes is calculated, and the fitting performance parameters corresponding to each updated startup delay coefficient are determined based on the ratio between the sum of the deviations and the mean square error.

[0189] Exemplarily, the calculation formula of the fitting performance parameter is as follows:

[0190] ;

[0191] ;

[0192] in, is the average delay time; For the sample data block a corresponding second total delay time prediction value; For the sample data block The corresponding data volume; For the sample data block The corresponding measured value of the total delay time; is the fitting performance parameter.

[0193] After obtaining the fitting performance parameters corresponding to each updated startup delay coefficient, the currently updated startup delay coefficient can be compared and screened with the previously updated startup delay coefficient based on the fitting performance parameters, and only the updated startup delay coefficient with smaller fitting performance parameters is retained. By repeating this process several times, the startup delay coefficient with the smallest fitting performance parameters under Monte Carlo conditions can be obtained as the startup delay coefficient with the best fitting performance, and then the target startup delay coefficient can be obtained based on the startup delay coefficient with the best fitting performance.

[0194] The methods for obtaining the target startup delay coefficient here include directly using the startup delay coefficient with the best fitting performance as the target startup delay coefficient, or obtaining the startup delay coefficient with the best fitting performance under different network topology information, and averaging the startup delay coefficients with the best fitting performance under different network topology information to obtain the target startup delay coefficient. This embodiment does not make any specific limitations on this.

[0195] The method provided in this embodiment randomly updates the initial startup delay coefficient, and uses the Monte Carlo method in combination with sample data blocks of different sizes to predict the total delay time, and then evaluates the fitting performance of each updated coefficient, and finally selects the startup delay coefficient with the best fitting performance as the target startup delay coefficient, which significantly improves the accuracy of parameter estimation and the robustness of the cost model.

[0196] In some embodiments, step 120 specifically includes:

[0197] In the measured network communication parameters, the data volume corresponding to each target data block and the measured value of the total delay time corresponding to each target data block are obtained; the target data block is a sample data block in the measured network communication parameters whose data volume is less than the target value;

[0198] According to the data amount corresponding to each of the target data blocks and the measured value of the total delay time corresponding to each of the target data blocks, the startup delay offset corresponding to the data jump transmission in the communication system to be predicted is fitted to obtain the startup delay offset corresponding to the data jump transmission in the communication system to be predicted;

[0199] The target startup delay coefficient and the total number of data jump transmissions are multiplied, and the multiplication result is added to the startup delay offset to obtain the startup delay time.

[0200] Figure 5 FIG. 1 is one of the simulation schematic diagrams of the communication delay time prediction method provided by the present invention; Figure 5 As shown, in the simulation verification process, by comparing the scatter plot formed by the end-to-end total delay time prediction value output by the updated cost model obtained by directly updating the start delay coefficient based on the target start delay coefficient with the scatter plot formed by the measured value of the total delay time, it can be seen that in the end-to-end total delay time prediction process of small-sized sample data blocks, there is a certain deviation between the end-to-end total delay time prediction value output by the updated cost model obtained by directly updating the target start delay coefficient and the measured value of the total delay time. Therefore, in order to further improve the prediction performance, this embodiment may further correct the updated cost model obtained by updating the target start delay coefficient to obtain the final optimized cost model, so as to accurately obtain the end-to-end total delay time prediction value based on the final optimized cost model.

[0201] Optionally, the updated cost model is used to predict and generate a series of end-to-end total delay time prediction values ​​based on the data volume of different sample data blocks, and the scatter plot formed by the end-to-end total delay time prediction values ​​output by the updated cost model is compared and analyzed with the scatter plot formed by the total delay time measured value to determine the turning point where a significant deviation occurs between the predicted value and the measured value. The turning point here usually means that when the data volume is reduced to a certain specific value, the prediction accuracy of the cost model begins to decrease. After obtaining the turning point, the target quantity value (i.e., the target value) can be adaptively determined based on the data volume of the data block corresponding to the turning point, so that the cost model can be effectively corrected based on this target value in the future, the prediction accuracy of the cost model is improved, and the adaptive ability and robustness of the system are enhanced.

[0202] After obtaining the target value, a sample data block with a data volume less than the target value can be obtained as a target data block in the measured network communication parameters, and the data volume and the measured value of the total delay time corresponding to each target data block can be obtained. Based on the data volume corresponding to each target data block and the measured value of the total delay time corresponding to each target data block, parameter fitting is performed to obtain the startup delay offset corresponding to the data jump transmission in the communication system to be predicted, and based on the startup delay offset, the updated cost model obtained by directly updating the startup delay coefficient based on the target startup delay coefficient is further corrected to obtain the final optimized cost model. The specific calculation formula of the final optimized cost model is as follows:

[0203] ;

[0204] in, is the start delay offset; Used to calculate the startup delay time; Used to calculate transmission delay time; ( ) is used to calculate the total number of data jump transmissions.

[0205] Accordingly, when calculating the startup delay time, the target startup delay coefficient and the total number of data jump transmissions may be multiplied and then added to the startup delay offset to obtain the corresponding startup delay time.

[0206] The method provided in this embodiment adaptively determines the target value that affects the prediction performance based on the verification results to optimize the model, and ultimately achieves high-precision prediction of the end-to-end total delay time, significantly improves the prediction performance, and enhances the adaptability and robustness of the system.

[0207] Figure 6 FIG. 2 is a second simulation diagram of the communication delay time prediction method provided by the present invention. Figure 6As shown, the gap between the predicted value and the measured value of the optimized cost model provided in the embodiment of the present application is very small, which effectively verifies the prediction accuracy of the optimized cost model provided in the embodiment of the present application.

[0208] Figure 7 This is the third simulation diagram of the communication delay time prediction method provided by the present invention. In order to further verify the prediction performance of the optimized cost model provided by the embodiment of the present application, this embodiment uses the target startup delay coefficient in the optimized cost model and the total number of data jump transmissions (also called the number of hops) in the sample communication system of the minimum topology network structure to obtain the Pareto optimal curve of the allreduce algorithm under the topology, so as to verify the correctness of the prediction of the optimized cost model through Pareto optimality. Figure 7 As shown, the fitting curve corresponding to the total delay time prediction value output by the optimized cost model and the total delay time actual value in the measured data should be above the Pareto optimal curve, which can effectively verify that the optimized cost model provided in this embodiment has good prediction performance.

[0209] Based on the above embodiments, the communication delay time prediction method provided in this embodiment has at least the following advantages:

[0210] Advantage 1: Through test data, a more accurate cost model can be calculated under any given communication system. This cost model can calculate the end-to-end communication delay data under a given data block size.

[0211] Advantage 2: The number of links (that is, the number of communication links) can be adaptively changed to achieve different topological connections between processing units. At this time, the cost model only needs to change the constant parameters, and can continue to use the parameter values ​​fitted based on the measured data to adaptively complete the delay data estimation under different topologies without rebuilding the physical model, thereby effectively ensuring the accurate prediction of communication delay time while reducing configuration costs and increasing the scope of application.

[0212] Advantage 3: The cost model can be used to evaluate the delay time of Allreduce communication in deep learning training. The entire iteration time can be evaluated through the delay time and the effective computing utilization of the device in a fixed model structure. This evaluation operation is also applicable to multi-card inference.

[0213] Advantage 4: Using the cost model, we can effectively determine the changes in the amount of data in the effective bandwidth for different data block sizes when the delay changes, and then guide manufacturers to optimize the end-to-end delay time to a reasonable time range, thereby reducing costs.

[0214] Advantage 5: The Pareto optimal curve obtained according to the cost model can guide the limits of communication algorithms and topology optimization to ensure that the optimal communication solution can be quickly found in complex communication systems.

[0215] The communication delay time prediction system provided by the present invention is described below. The communication delay time prediction system described below and the communication delay time prediction method described above can be referenced to each other.

[0216] Figure 8 is a schematic diagram of the structure of the communication delay time prediction system provided by the present invention; Figure 8 As shown, the system includes: a fitting unit 810 , a first acquisition unit 820 , a second acquisition unit 830 and a prediction unit 840 .

[0217] The fitting unit 810 is used to measure the network communication parameters of the sample communication system to obtain the measured network communication parameters of the sample communication system, and to fit the startup delay coefficient according to the measured network communication parameters to obtain the target startup delay coefficient corresponding to each data jump transmission in the communication system to be predicted; the sample communication system and the communication system to be predicted are both communication systems constructed by multiple processing nodes;

[0218] The first acquisition unit 820 is used to perform a startup delay time analysis according to the target startup delay coefficient and the total number of data jump transmissions in the communication system to be predicted, so as to obtain the startup delay time of the communication system to be predicted;

[0219] The second acquisition unit 830 is used to perform transmission delay time analysis according to the network configuration parameters of the communication system to be predicted and the amount of data transmitted by the communication system to be predicted, so as to obtain the transmission delay time of the communication system to be predicted;

[0220] The prediction unit 840 is used to predict the total delay time according to the startup delay time and the transmission delay time, so as to obtain a predicted value of the total delay time of the communication system to be predicted.

[0221] The system provided by the implementation of the present application obtains a target startup delay coefficient by fitting with the measured network communication parameters of a sample communication system, and on this basis, combines the total number of data jump transmissions of the communication system to be predicted in different scenarios with multi-dimensional characteristics such as the network configuration parameters and the amount of transmitted data, to perform adaptive prediction of the end-to-end total delay time of the communication system to be predicted in different scenarios, thereby avoiding the need to build a physical model for each scenario for actual experimental testing, and thus predicting the total delay time of the communication system to be predicted in multiple scenarios, thereby effectively reducing the configuration cost while achieving accurate prediction of the communication delay time, and effectively solving the problem of high cost of acquiring end-to-end delay time in related technologies.

[0222] The system provided by the present invention is used to execute the above-mentioned method embodiments. Please refer to the above-mentioned embodiments for the specific processes and detailed contents, which will not be repeated here.

[0223] Fig. 9 An example of a physical structure diagram of an electronic device is shown in FIG. Fig. 9 As shown, the electronic device may include: a processor (processor) 910 , a communication interface (Communications Interface) 920 , a memory (memory) 930 and a communication bus 940 , wherein the processor 910 , the communication interface 920 , and the memory 930 communicate with each other through the communication bus 940 . The processor 910 can call the logic instructions in the memory 930 to execute the communication delay time prediction method, which includes: measuring the network communication parameters of the sample communication system to obtain the measured network communication parameters of the sample communication system, and fitting the startup delay coefficient according to the measured network communication parameters to obtain the target startup delay coefficient corresponding to each data jump transmission in the communication system to be predicted; the sample communication system and the communication system to be predicted are both communication systems constructed by multiple processing nodes; the startup delay time is analyzed according to the target startup delay coefficient and the total number of data jump transmissions in the communication system to be predicted to obtain the startup delay time of the communication system to be predicted; the transmission delay time is analyzed according to the network configuration parameters of the communication system to be predicted and the amount of data transmitted by the communication system to be predicted to obtain the transmission delay time of the communication system to be predicted; the total delay time is predicted according to the startup delay time and the transmission delay time to obtain the total delay time prediction value of the communication system to be predicted.

[0224] In addition, the logic instructions in the above-mentioned memory 930 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0225] On the other hand, the present invention also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the communication delay time prediction method provided by the above-mentioned methods, which method includes: measuring the network communication parameters of the sample communication system to obtain the measured network communication parameters of the sample communication system, and fitting the startup delay coefficient according to the measured network communication parameters to obtain the target startup delay coefficient corresponding to each data jump transmission in the communication system to be predicted; the sample communication system and the communication system to be predicted are both communication systems constructed by multiple processing nodes; the startup delay time is analyzed according to the target startup delay coefficient and the total number of data jump transmissions in the communication system to be predicted to obtain the startup delay time of the communication system to be predicted; the transmission delay time is analyzed according to the network configuration parameters of the communication system to be predicted and the amount of data transmitted by the communication system to be predicted to obtain the transmission delay time of the communication system to be predicted; the total delay time is predicted according to the startup delay time and the transmission delay time to obtain the total delay time prediction value of the communication system to be predicted.

[0226] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the communication delay time prediction method provided by the above-mentioned methods, the method comprising: measuring the network communication parameters of a sample communication system to obtain the measured network communication parameters of the sample communication system, and fitting the startup delay coefficient according to the measured network communication parameters to obtain a target startup delay coefficient corresponding to each data jump transmission in the communication system to be predicted; the sample communication system and the communication system to be predicted are both communication systems constructed by multiple processing nodes; a startup delay time analysis is performed according to the target startup delay coefficient and the total number of data jump transmissions in the communication system to be predicted to obtain the startup delay time of the communication system to be predicted; a transmission delay time analysis is performed according to the network configuration parameters of the communication system to be predicted and the amount of data transmitted by the communication system to be predicted to obtain the transmission delay time of the communication system to be predicted; a total delay time prediction is performed according to the startup delay time and the transmission delay time to obtain a total delay time prediction value of the communication system to be predicted.

[0227] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0228] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0229] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A communication delay time prediction method, characterized in that: include: The network communication parameters of the sample communication system are measured to obtain the measured network communication parameters of the sample communication system, and the startup delay coefficient is fitted according to the measured network communication parameters to obtain the target startup delay coefficient corresponding to each data jump transmission in the communication system to be predicted; the sample communication system and the communication system to be predicted are both communication systems constructed by multiple processing nodes; Performing a startup delay time analysis according to the target startup delay coefficient and the total number of data jump transmissions in the communication system to be predicted to obtain a startup delay time of the communication system to be predicted; Performing transmission delay time analysis according to the network configuration parameters of the communication system to be predicted and the amount of data transmitted by the communication system to be predicted to obtain the transmission delay time of the communication system to be predicted; Perform a total delay time prediction based on the startup delay time and the transmission delay time to obtain a total delay time prediction value of the communication system to be predicted; The total number of data jump transmissions in the communication system to be predicted is obtained based on the following steps: Performing statistics on the number of communication links and the number of processing nodes according to the network topology information of the communication system to be predicted, to obtain the number of communication links and the number of processing nodes in the communication system to be predicted; The total number of data jump transmissions is calculated based on the number of data blocks transmitted by the communication system to be predicted, the number of communication links and the number of processing nodes; The calculating the total number of data jump transmission times according to the number of data blocks transmitted by the communication system to be predicted, the number of communication links and the number of processing nodes comprises: Calculating a logarithmic value between the number of communication links and the number of processing nodes; Calculate the product of the number of data blocks and the logarithmic value to obtain the total number of data jump transmissions; The step of fitting the startup delay coefficient according to the measured network communication parameters to obtain a target startup delay coefficient corresponding to each data jump transmission in the communication system to be predicted includes: Based on the target fitting model, fitting the measured network communication parameters to obtain a fitting loss value corresponding to the target fitting model; With the goal of minimizing the fitting loss value, the target fitting model is optimized to obtain an initial startup delay coefficient corresponding to each data jump transmission in the communication system to be predicted; According to the initial startup delay coefficient, obtaining the target startup delay coefficient; The performing startup delay time analysis according to the target startup delay coefficient and the total number of data jump transmissions in the communication system to be predicted to obtain the startup delay time of the communication system to be predicted includes: In the measured network communication parameters, the data volume corresponding to each target data block and the measured value of the total delay time corresponding to each target data block are obtained; the target data block is a sample data block in the measured network communication parameters whose data volume is less than the target value; According to the data amount corresponding to each of the target data blocks and the measured value of the total delay time corresponding to each of the target data blocks, the startup delay offset corresponding to the data jump transmission in the communication system to be predicted is fitted to obtain the startup delay offset corresponding to the data jump transmission in the communication system to be predicted; The target startup delay coefficient and the total number of data jump transmissions are multiplied, and the multiplication result is added to the startup delay offset to obtain the startup delay time.

2. The communication delay time prediction method according to claim 1, characterized in that: The step of performing transmission delay time analysis according to the network configuration parameters of the communication system to be predicted and the amount of data transmitted by the communication system to be predicted to obtain the transmission delay time of the communication system to be predicted includes: Performing parameter parsing on the network configuration parameters to obtain network topology information of the communication system to be predicted and bandwidth data of each communication link in the communication system to be predicted; The transmission delay time of the communication system to be predicted is calculated according to the network topology information, the bandwidth data and the amount of data transmitted by the communication system to be predicted.

3. The communication delay time prediction method according to claim 2, characterized in that: The calculating, according to the network topology information, the bandwidth data and the amount of data transmitted by the communication system to be predicted, the transmission delay time of the communication system to be predicted comprises: Perform statistics on the number of communication links and the number of processing nodes according to the network topology information to obtain the number of communication links and the number of processing nodes in the communication system to be predicted; Calculating the product of the number of communication links and the bandwidth data to obtain the total bandwidth data of the communication system to be predicted; Calculating the ratio between the amount of data transmitted by the communication system to be predicted and the total bandwidth data to obtain the bandwidth utilization rate of the communication system to be predicted; The transmission delay time of the communication system to be predicted is calculated according to the number of processing nodes and the bandwidth utilization.

4. The communication delay time prediction method according to claim 3, characterized in that: The calculating, according to the number of processing nodes and the bandwidth utilization rate, the transmission delay time of the communication system to be predicted comprises: Calculating the fault tolerance performance parameter of the communication system to be predicted according to the number of processing nodes; The product of the fault tolerance performance parameter and the bandwidth utilization is calculated to obtain the transmission delay time.

5. The communication delay time prediction method according to claim 1, characterized in that: The acquiring the target startup delay coefficient according to the initial startup delay coefficient includes: Calculating an average value of initial startup delay coefficients corresponding to each data jump transmission in the sample communication system under a plurality of different topological structures to obtain an average startup delay coefficient; The target startup delay coefficient is obtained according to the average startup delay coefficient.

6. The communication delay time prediction method according to claim 1, characterized in that: The step of fitting the measured network communication parameters based on the target fitting model to obtain a fitting loss value corresponding to the target fitting model includes: In the measured network communication parameters, the data volume and total delay time measured values ​​of a plurality of sample data blocks of different sizes are obtained; For the sample data block of any size, inputting the data amount of the sample data block into the target fitting model to obtain a first total delay time prediction value of the sample data block; Determine a loss value corresponding to the sample data block according to the total delay time actually measured value of the sample data block and a deviation between the first total delay time predicted value of the sample data block and the total delay time actually measured value of the sample data block; The sample data blocks of various sizes are traversed to obtain loss values ​​corresponding to the multiple sample data blocks, and the loss values ​​corresponding to the multiple sample data blocks are merged to obtain a fitting loss value corresponding to the target fitting model.

7. The communication delay time prediction method according to claim 6, characterized in that: The acquiring the target startup delay coefficient according to the initial startup delay coefficient includes: Randomly updating the initial startup delay coefficient to obtain a plurality of updated startup delay coefficients; According to each of the updated start delay coefficients, obtaining second total delay time prediction values ​​corresponding to the sample data blocks of different sizes; Obtaining fitting performance parameters corresponding to each of the updated start-up delay coefficients according to the second total delay time prediction value, data volume and total delay time measured value corresponding to the sample data blocks of different sizes; According to the fitting performance parameter, obtaining a startup delay coefficient with the best fitting performance from a plurality of the updated startup delay coefficients; The target startup delay coefficient is obtained according to the startup delay coefficient with the best fitting performance.

8. The communication delay time prediction method according to claim 7, characterized in that: The obtaining, according to the second total delay time prediction value, data amount and total delay time actual measured value corresponding to the sample data blocks of different sizes, the fitting performance parameter corresponding to each updated startup delay coefficient comprises: Obtaining average delay times corresponding to the sample data blocks of different sizes according to a ratio between a second total delay time prediction value corresponding to the sample data blocks of different sizes and a data amount; Obtaining delay time errors corresponding to the sample data blocks of different sizes according to deviations between total delay time measured values ​​corresponding to the sample data blocks of different sizes and the average delay time; The fitting performance parameters corresponding to each of the updated start-up delay coefficients are obtained according to the total delay time measured values ​​corresponding to the sample data blocks of different sizes and the delay time error.

9. A communication delay time prediction system, characterized in that: include: A fitting unit is used to measure the network communication parameters of the sample communication system to obtain the measured network communication parameters of the sample communication system, and to fit the startup delay coefficient according to the measured network communication parameters to obtain the target startup delay coefficient corresponding to each data jump transmission in the communication system to be predicted; the sample communication system and the communication system to be predicted are both communication systems constructed by multiple processing nodes; A first acquisition unit, configured to perform a startup delay time analysis according to the target startup delay coefficient and the total number of data jump transmissions in the communication system to be predicted, to obtain the startup delay time of the communication system to be predicted; A second acquisition unit is used to perform transmission delay time analysis according to the network configuration parameters of the communication system to be predicted and the amount of data transmitted by the communication system to be predicted, so as to obtain the transmission delay time of the communication system to be predicted; A prediction unit, configured to predict the total delay time according to the startup delay time and the transmission delay time, and obtain a predicted value of the total delay time of the communication system to be predicted; The total number of data jump transmissions in the communication system to be predicted is obtained based on the following steps: Performing statistics on the number of communication links and the number of processing nodes according to the network topology information of the communication system to be predicted, to obtain the number of communication links and the number of processing nodes in the communication system to be predicted; The total number of data jump transmissions is calculated based on the number of data blocks transmitted by the communication system to be predicted, the number of communication links and the number of processing nodes; The calculating the total number of data jump transmission times according to the number of data blocks transmitted by the communication system to be predicted, the number of communication links and the number of processing nodes comprises: Calculating a logarithmic value between the number of communication links and the number of processing nodes; Calculate the product of the number of data blocks and the logarithmic value to obtain the total number of data jump transmissions; The fitting unit is specifically used for: Based on the target fitting model, fitting the measured network communication parameters to obtain a fitting loss value corresponding to the target fitting model; With the goal of minimizing the fitting loss value, the target fitting model is optimized to obtain an initial startup delay coefficient corresponding to each data jump transmission in the communication system to be predicted; According to the initial startup delay coefficient, obtaining the target startup delay coefficient; The first acquisition unit is specifically used to: In the measured network communication parameters, the data volume corresponding to each target data block and the measured value of the total delay time corresponding to each target data block are obtained; the target data block is a sample data block in the measured network communication parameters whose data volume is less than the target value; According to the data amount corresponding to each of the target data blocks and the measured value of the total delay time corresponding to each of the target data blocks, the startup delay offset corresponding to the data jump transmission in the communication system to be predicted is fitted to obtain the startup delay offset corresponding to the data jump transmission in the communication system to be predicted; The target startup delay coefficient and the total number of data jump transmissions are multiplied, and the multiplication result is added to the startup delay offset to obtain the startup delay time.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the communication delay time prediction method according to any one of claims 1 to 8 is implemented.

11. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the communication delay time prediction method according to any one of claims 1 to 8 is implemented.

Images