Method and device for predicting training time consumption in heterogeneous computing power based on CXL

Abstract

The invention discloses a CXL-based training time consumption prediction method and device in heterogeneous computing power, and relates to the technical field of heterogeneous computing, and the method comprises the steps: obtaining task description information of a distributed training task; determining a neural layer calculation sequence of a neural network in the distributed training task and each heterogeneous computing power corresponding to the distributed training task according to the task description information; according to the neural layer calculation sequence, initial calculation process diagrams corresponding to the heterogeneous calculation power are constructed; adding a CXL-based parameter synchronization process to a backward transmission calculation node in the initial calculation process diagram to obtain each target calculation process diagram; respectively obtaining neural layer calculation time consumption and memory access time consumption of each node in the target calculation process graph; and according to the calculation time consumption of each neural layer and the access time consumption of each memory, predicting the total time consumption of training the distributed training task. The method solves the problems of long time consumption of parameter synchronization, low time consumption prediction accuracy and the like, reduces the time consumption of model parameter synchronization, and improves the accuracy of training time consumption prediction.

Description

Technical Field

[0001] The present invention relates to the field of heterogeneous computing technology, and in particular to a method and device for predicting training time consumption in heterogeneous computing power based on CXL. Background Art

[0002] In order to solve the problem of large-scale neural network training, distributed training is usually used to split the training data or large network models, and deploy the split data and sub-training tasks to multiple computing forces.

[0003] The current training time prediction method in heterogeneous computing power mainly synchronizes communication by setting up communication nodes, and then synchronizes model parameters, and uses the synchronized model parameters for model training. Communication and model parameter synchronization take a long time, and the accuracy of training time prediction is low.

[0004] In summary, how to effectively solve the problems of long communication time, long model parameter synchronization time, and low accuracy in predicting training time is an issue that technical personnel in this field urgently need to solve. Summary of the invention

[0005] An object of the present invention is to provide a method for predicting training time consumption in heterogeneous computing power based on CXL, which method greatly reduces the time consumption of model parameter synchronization and improves the accuracy of training time consumption prediction; another object of the present invention is to provide a device, computer-readable storage medium and computer program product for predicting training time consumption in heterogeneous computing power based on CXL.

[0006] In order to solve the above technical problems, the present invention provides the following technical solutions:

[0007] A method for predicting training time consumption in heterogeneous computing power based on CXL, comprising:

[0008] Get the task description information of the distributed training task;

[0009] Determine, according to the task description information, a neural layer calculation order of the neural network in the distributed training task and the heterogeneous computing power corresponding to the distributed training task;

[0010] Constructing initial calculation process diagrams corresponding to the heterogeneous computing power according to the neural layer calculation sequence;

[0011] Add the parameter synchronization process based on CXL to the backward transmission computing nodes in each initial computing process graph to obtain each target computing process graph;

[0012] Respectively obtain the neural layer computing time and memory access time of each node in the target computing process diagram; wherein each node includes a forward transmission computing node, a backward transmission computing node and a parameter synchronization node;

[0013] According to the computation time of each neural layer and the memory access time, the total time required for training the distributed training task is predicted.

[0014] In a specific implementation of the present invention, a parameter synchronization process based on CXL is added to the backward transmission computing node in each initial computing process graph, including:

[0015] Add the parameter synchronization node between the backward transmission calculation nodes of the same neural layer in each initial calculation process graph;

[0016] Obtain the neural layer to which each parameter synchronization node belongs respectively;

[0017] A parameter synchronization connection is established between each parameter synchronization node and a forward transmission computing node in the corresponding neural layer; wherein the parameter transmission direction of the parameter synchronization connection is from the parameter synchronization node to the forward transmission computing node.

[0018] In a specific implementation of the present invention, respectively obtaining the memory access time consumption of each node in the target computing process graph includes:

[0019] For each parameter synchronization node, obtain the communication delay of each parameter synchronization node accessing each heterogeneous computing power through CXL;

[0020] Obtain the neural layer parameter data of each backward transmission computing node in each target computing process graph respectively;

[0021] Obtain the communication bandwidth of each parameter synchronization node to access each heterogeneous computing power through CXL;

[0022] For each heterogeneous computing power, the amount of neural layer parameter data is divided by the communication bandwidth, and the obtained quotient is summed with the communication delay to obtain the sub-memory access time.

[0023] The access time consumption of each sub-memory is summed up and calculated to obtain the memory access time consumption.

[0024] In a specific implementation of the present invention, respectively obtaining the neural layer computing time and memory access time of each node in the target computing process graph includes:

[0025] For each forward transmission computing node and each backward transmission computing node, respectively obtain the neural layer computing complexity of each node in each target computing process graph;

[0026] Obtain the computing power of each heterogeneous computing power separately;

[0027] Calculate the neural layer calculation time of each node according to the computing power and the neural layer calculation complexity;

[0028] Respectively obtain the accessible memory space preset by each heterogeneous computing power for the distributed training task;

[0029] Obtain the target access latency and target access bandwidth for each heterogeneous computing power to read each accessible memory space;

[0030] Respectively obtain the neural layer parameter data volume and neural layer input data volume of each node in each target calculation process graph;

[0031] The memory access time consumption of each node is calculated respectively according to the target access delay, the target access bandwidth, the neural layer parameter data volume and the neural layer input data volume.

[0032] In a specific implementation of the present invention, obtaining target access latency and target access bandwidth for each heterogeneous computing power to read each accessible memory space includes:

[0033] When the accessible memory space is a local memory, obtaining a local access latency and a local access bandwidth of the heterogeneous computing power to read the local memory;

[0034] The local access delay is determined as the target access delay, and the local access bandwidth is determined as the target access bandwidth.

[0035] In a specific implementation of the present invention, obtaining target access latency and target access bandwidth for each heterogeneous computing power to read each accessible memory space includes:

[0036] When the accessible memory space is a CXL memory, obtaining a CXL access latency and a CXL access bandwidth of the heterogeneous computing power reading the CXL memory;

[0037] The CXL access latency is determined as the target access latency, and the CXL access bandwidth is determined as the target access bandwidth.

[0038] In a specific implementation of the present invention, the memory access time consumption of each node is calculated respectively according to the target access delay, the target access bandwidth, the neural layer parameter data volume and the neural layer input data volume, including:

[0039] For each forward transmission computing node and each backward transmission computing node, the amount of neural layer input data and the amount of neural layer parameter data are summed up to obtain a total amount of data;

[0040] The data access time is obtained by dividing the total data volume by the target access bandwidth;

[0041] The data access time consumption and the target access delay are summed up to obtain the memory access time consumption.

[0042] In a specific embodiment of the present invention, predicting the total time required for training the distributed training task according to the computation time of each neural layer and the memory access time includes:

[0043] For each forward transmission computing node and each backward transmission computing node, the larger of the neural layer computing time and the memory access time of each node is determined as the current neural layer processing time corresponding to the node;

[0044] For each parameter synchronization node, the memory access time of each node is determined as the current neural layer processing time corresponding to the node;

[0045] The total time required for training the distributed training task is predicted according to the processing time of each current neural layer.

[0046] In a specific embodiment of the present invention, predicting the total time required for training the distributed training task according to the processing time of each current neural layer includes:

[0047] Determine the initial node in each target calculation process graph as the currently running node;

[0048] Construct a time-consuming list containing the processing time of the current neural layer corresponding to each currently running node;

[0049] Select the minimum value of the processing time of each current neural layer from the time-consuming list;

[0050] Add the minimum value to the current training time;

[0051] Calculate the difference between the processing time of each current neural layer in the time-consuming list and the minimum value to obtain each time difference;

[0052] Deleting the zero value in each time difference from the time-consuming list, and updating the time-consuming list according to each target calculation process diagram;

[0053] Determine whether the current neural layer processing time corresponding to each initial node exists in the updated time-consuming list;

[0054] If not, repeat the step of selecting the minimum value of the processing time of each current neural layer from the time consumption list;

[0055] If yes, then obtaining the number of times the current neural layer processing time corresponding to each pre-recorded initial node is added to the time consumption list, and selecting the minimum number of additions from each number of additions;

[0056] When the minimum number of additions reaches a preset value, the current training time is obtained; wherein the preset value is 1 greater than the preset number of iterations;

[0057] The current training time is determined as the total time.

[0058] In a specific implementation of the present invention, updating the time-consuming list according to each target calculation process diagram includes:

[0059] Determine the node pointed to by the currently running node corresponding to the zero time difference in the target computing process graph as the new currently running node;

[0060] The current neural layer processing time corresponding to the new currently running node is added to the time consumption list.

[0061] In a specific implementation of the present invention, adding the current neural layer processing time corresponding to the new currently running node to the time consumption list includes:

[0062] Determine whether the new currently running node meets the running conditions;

[0063] If yes, then executing the step of adding the current neural layer processing time corresponding to the new currently running node to the time consumption list;

[0064] If not, wait for a preset time period and repeat the step of determining whether the new currently running node meets the running conditions.

[0065] In a specific implementation of the present invention, determining whether the new currently running node meets the running condition includes:

[0066] When the new currently running node is a forward transmission computing node and is not an initial node, determine whether the current iteration number corresponding to the new currently running node is the same as the current iteration number of the parameter synchronization node pointing to the new currently running node, and is 1 less than the current iteration number of the forward transmission computing node pointing to the new currently running node;

[0067] When the new currently running node is the initial node, determine whether the current iteration number corresponding to the new currently running node is the same as the current iteration number of each node pointing to the new currently running node;

[0068] When the new currently running node is a backward transmission computing node, determining whether the current iteration number corresponding to the new currently running node is less than 1 of the current iteration number of the forward transmission computing node or the backward transmission computing node pointing to the new currently running node;

[0069] When the new currently running node is a parameter synchronization node, it is determined whether the current iteration number corresponding to the new currently running node is less than 1 of the current iteration number of the backward transmission calculation node pointing to the new currently running node.

[0070] A CXL-based heterogeneous computing power training time prediction device, comprising:

[0071] Memory for storing computer programs;

[0072] A processor is used to implement the steps of the training time prediction method in heterogeneous computing power based on CXL as described above when executing the computer program.

[0073] A computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the method for predicting training time consumption in heterogeneous computing power based on CXL as described above.

[0074] A computer program product includes a computer program, which, when executed by a processor, implements the steps of the method for predicting training time consumption in heterogeneous computing power based on CXL as described above.

[0075] The method for predicting training time consumption in heterogeneous computing power based on CXL provided by the present invention obtains task description information of distributed training tasks; determines the neural layer calculation order of the neural network in the distributed training tasks and the heterogeneous computing powers corresponding to the distributed training tasks according to the task description information; constructs initial calculation process graphs corresponding to the heterogeneous computing powers according to the neural layer calculation order; adds a parameter synchronization process based on CXL to the backward transmission calculation node in the initial calculation process graph to obtain each target calculation process graph; obtains the neural layer calculation time consumption and memory access time consumption of each node in the target calculation process graph respectively; wherein each node includes a forward transmission calculation node, a backward transmission calculation node and a parameter synchronization node; and predicts the total time consumption required for training the distributed training task according to the calculation time consumption of each neural layer and the memory access time consumption.

[0076] The beneficial effect of the present invention is that a distributed training time-consuming prediction framework in heterogeneous computing power based on CXL is proposed. By obtaining the task description information of the distributed training task, the neural layer calculation order of the neural network in the distributed training task and the heterogeneous computing power corresponding to the distributed training task are determined according to the task description information, the computing performance of the heterogeneous computing power and the characteristics of CXL are considered, and the time-consuming prediction of the distributed training task is realized in the heterogeneous computing system based on CXL. By constructing the initial calculation process graph corresponding to each heterogeneous computing power according to the neural layer calculation order, and adding the parameter synchronization process based on CXL to the backward transmission calculation node in the initial calculation process graph, direct access to the memory of each heterogeneous computing power is realized through CXL, and then model parameter synchronization is realized, and communication synchronization is not required, which greatly reduces the time consumption of model parameter synchronization. By combining the calculation time consumption of each neural layer and the time consumption of each memory access to predict the total time consumption required for training the distributed training task, the influence of the memory access time consumption is considered, and the accuracy of the training time consumption prediction is greatly improved.

[0077] Correspondingly, the present invention also provides a CXL-based heterogeneous computing power training time prediction device, a computer-readable storage medium and a computer program product corresponding to the above-mentioned CXL-based heterogeneous computing power training time prediction method, which have the above-mentioned technical effects and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0078] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or related technologies, the drawings required for use in the embodiments or related technical descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0079] Figure 1 It is a flowchart of an implementation method of a training time-consuming prediction method in heterogeneous computing power based on CXL in an embodiment of the present invention;

[0080] Figure 2 It is a flowchart of another implementation method of a training time-consuming prediction method in heterogeneous computing power based on CXL in an embodiment of the present invention;

[0081] Figure 3 is a structural block diagram of a heterogeneous computing system based on CXL in an embodiment of the present invention;

[0082] Figure 4 This is an architecture diagram of a training time-consuming prediction system in heterogeneous computing power based on CXL in an embodiment of the present invention;

[0083] Figure 5 A schematic diagram of a calculation process diagram in an embodiment of the present invention;

[0084] Figure 6 is a schematic diagram of another calculation process diagram in an embodiment of the present invention;

[0085] Figure 7 It is a structural block diagram of a training time-consuming prediction device in heterogeneous computing power based on CXL in an embodiment of the present invention;

[0086] Figure 8 It is a structural block diagram of a training time-consuming prediction device in heterogeneous computing power based on CXL in an embodiment of the present invention;

[0087] Fig. 9 A schematic diagram of the specific structure of a CXL-based training time-consuming prediction device in heterogeneous computing power provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0088] In order to enable those skilled in the art to better understand the scheme of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific implementation methods. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0089] See also Figure 1 , Figure 1 The following is a flowchart of an implementation method of a method for predicting training time consumption in heterogeneous computing power based on CXL in an embodiment of the present invention. The method may include the following steps:

[0090] S101: Obtain task description information of a distributed training task.

[0091] Predefine the task description information of the distributed training task and obtain the task description information of the distributed training task. The task description information may include the data size allocated to each heterogeneous computing power, the training precision P (single precision P=1, double precision P=2), the number of training iterations (steps) required for prediction, and the neural network structure of the model to be trained. According to the neural network structure of the model to be trained, the neural layers contained in the model to be trained can be determined, and the parameter data volume of each neural layer, the computational complexity FLOPs, and the input data volume of the neural layer can also be determined.

[0092] S102: Determine the neural layer calculation order of the neural network in the distributed training task and the heterogeneous computing power corresponding to the distributed training task according to the task description information.

[0093] After obtaining the task description information of the distributed training task, the neural layer calculation order of the neural network in the distributed training task and the heterogeneous computing power corresponding to the distributed training task are determined according to the task description information.

[0094] Heterogeneous computing power can include graphics processing units (GPUs), central processing units (CPUs), deep learning processing units (DPUs), etc.

[0095] Each neural layer may include a convolutional layer, a fully connected layer, a pooling layer, etc.

[0096] S103: Construct the initial calculation process diagram corresponding to each heterogeneous computing power according to the calculation sequence of the neural layer.

[0097] After determining the neural layer calculation order and the heterogeneous computing power, the initial calculation process diagram corresponding to each heterogeneous computing power is constructed according to the neural layer calculation order. For each heterogeneous computing power, the neural layers can be connected in sequence according to the determined neural layer calculation order to obtain the initial calculation process diagram.

[0098] S104: adding a CXL-based parameter synchronization process to the backward transmission computing nodes in each initial computing process graph to obtain each target computing process graph.

[0099] After constructing the initial calculation process graph, add the parameter synchronization process based on CXL (Compute Express Link) to the backward transmission calculation nodes in each initial calculation process graph to obtain each target calculation process graph. By adding the parameter synchronization process based on CXL, the model parameter synchronization of the corresponding heterogeneous computing power memory can be directly accessed through CXL, without the need for communication synchronization process, which improves the efficiency of model parameter synchronization.

[0100] S105: Obtain the neural layer computing time and memory access time of each node in the target computing process graph respectively.

[0101] Each node includes a forward transmission computing node, a backward transmission computing node and a parameter synchronization node.

[0102] After adding the CXL-based parameter synchronization process to each initial calculation process graph, each node in each target calculation process graph includes a forward transmission calculation node, a backward transmission calculation node, and a parameter synchronization node. The calculation process of each node in the target calculation process graph includes a model parameter and input data acquisition process (i.e., a memory access process), and also includes a neural layer calculation process based on the acquired model parameters and input data. The model parameter and input data acquisition is performed in parallel with the neural layer calculation, and the neural layer calculation time and memory access time of each node in the target calculation process graph are obtained respectively, so as to respectively count the model parameter and input data acquisition time and the neural layer calculation time.

[0103] S106: Predict the total time required for training the distributed training task based on the computation time of each neural layer and the memory access time.

[0104] After obtaining the neural layer computing time and memory access time of each node, the total time required for distributed training tasks is predicted based on the neural layer computing time and memory access time. By combining the neural layer computing time and memory access time to predict the total time required for distributed training tasks, the impact of memory access time is taken into account, which greatly improves the accuracy of training time prediction.

[0105] It can be seen from the above technical scheme that a distributed training time prediction framework based on CXL in heterogeneous computing power is proposed. By obtaining the task description information of the distributed training task, the neural layer calculation order of the neural network in the distributed training task and the heterogeneous computing power corresponding to the distributed training task are determined according to the task description information. The computing performance of the heterogeneous computing power and the characteristics of CXL are considered, and the time prediction of the distributed training task in the heterogeneous computing system based on CXL is realized. By constructing the initial calculation process graph corresponding to each heterogeneous computing power according to the neural layer calculation order, and adding the CXL-based parameter synchronization process to the backward transmission calculation node in the initial calculation process graph, direct access to the memory of each heterogeneous computing power is realized through CXL, and then model parameter synchronization is realized, without the need for communication synchronization, which greatly reduces the model parameter synchronization time. By combining the calculation time of each neural layer and the memory access time prediction for the total time required for distributed training tasks, the influence of memory access time is considered, and the accuracy of training time prediction is greatly improved.

[0106] It should be noted that, based on the above embodiment, the embodiment of the present invention also provides corresponding improved solutions. In the subsequent embodiments, the same steps or corresponding steps as those in the above embodiment can be referenced to each other, and the corresponding beneficial effects can also be referenced to each other, which will not be repeated one by one in the following improved embodiments.

[0107] See also Figure 2 , Figure 2 The following is a flowchart of another method for predicting training time consumption in heterogeneous computing power based on CXL in an embodiment of the present invention. The method may include the following steps:

[0108] S201: Obtain task description information of a distributed training task.

[0109] S202: Determine the neural layer calculation order of the neural network in the distributed training task and the heterogeneous computing power corresponding to the distributed training task according to the task description information.

[0110] The heterogeneous computing power selected by the distributed training task in the heterogeneous computing system can be represented by the Internet Protocol (IP) or other unique identifiers.

[0111] See also Figure 3 , Figure 3 This is a structural block diagram of a heterogeneous computing system based on CXL in an embodiment of the present invention. Since the system contains heterogeneous computing power, and the interconnection between heterogeneous computing power is composed of a complex data center network. In addition, when the heterogeneous computing system deploys the CXL consistent data access protocol, all extended memory (such as non-local memory devices) and local memory (such as local video memory) of heterogeneous computing power in the system can be called by other heterogeneous computing power and used directly as local memory resources, connected using the CXL switching network.

[0112] Due to the existence of heterogeneous computing power, the computing capabilities of each computing power are different, and the coordination between them will affect the execution efficiency of distributed training tasks. At the same time, due to the introduction of CXL, the execution efficiency of distributed training can be optimized through CXL technology.

[0113] See also Figure 4 , Figure 4 This is an architecture diagram of a training time prediction system in heterogeneous computing power based on CXL in an embodiment of the present invention. It is divided into four modules, and their names and functions are as follows:

[0114] (1) Modeling module for distributed training tasks: Based on the user’s task description information, the distributed training tasks that the user needs to predict the time consumption are modeled.

[0115] (2) CXL information collection module: used to collect CXL exchange network information of multiple heterogeneous computing systems.

[0116] (3) Computing power information collection module: used to record and store the computing power and memory bandwidth information of heterogeneous computing power in a multi-heterogeneous computing system.

[0117] (4) Time consumption prediction module for distributed training tasks: Based on the modeling of distributed training tasks and the information collected by each module, the time consumption of distributed training tasks is predicted.

[0118] S203: Construct initial calculation process diagrams corresponding to each heterogeneous computing power according to the calculation sequence of the neural layer.

[0119] See also Figure 5 , Figure 5 It is a schematic diagram of a calculation process diagram in an embodiment of the present invention. Figure 5The initial calculation process graph is constructed by taking a distributed training task that uses two heterogeneous computing forces to train three neural layers as an example. The initial calculation process graph can be constructed using graph modeling tools such as networkx or neo4j. In the initial calculation process graph, nodes represent calculation processes, and edges represent dependencies between calculation processes. Dependency means that for a node, the node can only be executed after the node pointing to the node is executed. In the initial calculation process graph, each node needs to identify the amount of parameter data of the neural layer, the computational complexity, the amount of data input to the neural layer, and whether it is an initial graph node. Each heterogeneous computing force needs to be given an allocated memory location.

[0120] S204: Add parameter synchronization nodes between backward transmission calculation nodes of the same neural layer in each initial calculation process graph.

[0121] See also Figure 6 , Figure 6 Schematic diagram of another calculation process graph in an embodiment of the present invention. After constructing the initial calculation process graph, a parameter synchronization node is added between the backward transmission calculation nodes of the same neural layer in each initial calculation process graph. Figure 6 In the example, parameter synchronization node 1 is added between the two backward transmission computing nodes including the backward transmission computing layer 1, parameter synchronization node 2 is added between the two backward transmission computing nodes including the backward transmission computing layer 2, and parameter synchronization node 3 is added between the two backward transmission computing nodes including the backward transmission computing layer 3.

[0122] Distributed training mainly includes two methods: parameter server and allreduce. The purpose of both methods is to allow each heterogeneous computing power to obtain updated model parameters. Therefore, with the help of the CXL consistent data access protocol, there is no need to collect parameters through the network. You only need to let a parameter synchronization node in the CXL exchange network directly access the memory space of each heterogeneous computing power to directly perform parameter synchronization. Therefore, when there is a CXL consistent data access protocol, the calculation process constructed by the two distributed training methods of parameter server and allreduce is Figure 1 Their parameter synchronization process can be used Figure 6 It is represented by the parameter synchronization node in .

[0123] S205: Obtain the neural layer to which each parameter synchronization node belongs respectively.

[0124] After adding the parameter synchronization nodes, obtain the neural layers to which each parameter synchronization node belongs. Figure 6 As shown, parameter synchronization node 1 belongs to backward transmission calculation layer 1, parameter synchronization node 2 belongs to backward transmission calculation layer 2, and parameter synchronization node 3 belongs to backward transmission calculation layer 3.

[0125] S206: Establishing parameter synchronization connections between each parameter synchronization node and the forward transmission calculation node in the corresponding neural layer to obtain each target calculation process diagram.

[0126] The parameter transmission direction of the parameter synchronization connection is from the parameter synchronization node to the forward transmission computing node.

[0127] After obtaining the neural layer to which each parameter synchronization node belongs, a parameter synchronization connection is established between each parameter synchronization node and the forward transmission computing node in the neural layer to which it belongs, and the parameter transmission direction of the parameter synchronization connection is from the parameter synchronization node to the forward transmission computing node, thereby obtaining each target calculation process diagram.

[0128] like Figure 6 As shown, a parameter synchronization connection is established between parameter synchronization node 1 and two forward transmission computing nodes including forward transmission computing layer 1, a parameter synchronization connection is established between parameter synchronization node 2 and two forward transmission computing nodes including forward transmission computing layer 2, and a parameter synchronization connection is established between parameter synchronization node 3 and two forward transmission computing nodes including forward transmission computing layer 3.

[0129] S207: For each parameter synchronization node, obtain the communication delay of each parameter synchronization node accessing each heterogeneous computing power through CXL.

[0130] After establishing the calculation process graphs for each target, for each parameter synchronization node, the communication delay of each parameter synchronization node accessing each heterogeneous computing power through CXL is obtained.

[0131] Since there is often a gap between the actual communication capacity of CXL and the nominal communication capacity of CXL, it is necessary to measure the actual communication capacity between CXLs in heterogeneous computing systems before making time consumption predictions. This process can be obtained using common memory bandwidth and latency testing tools such as tinymembench. It is necessary to collect information such as bandwidth and latency between CXL nodes (all heterogeneous computing power that uses the CXL switching network to interconnect reads remote memory), which can be represented and stored using an adjacency matrix or adjacency table composed of IPs.

[0132] S208: Obtain the neural layer parameter data volume of each backward transmission calculation node in each target calculation process diagram respectively.

[0133] Obtain the neural layer parameter data of each backward transmission computing node in each target computing process diagram. The parameter of each neural layer can be obtained using the structure of the neural layer. For example, for a fully connected layer with a weight matrix of [x, x], its parameters include the weight matrix and bias, and the parameter data of the fully connected layer is (x*x+x)*P *4 bytes.

[0134] S209: Obtain the communication bandwidth of each parameter synchronization node to access each heterogeneous computing power through CXL.

[0135] Obtain the communication bandwidth of each parameter synchronization node to access each heterogeneous computing power through CXL.

[0136] S210: For each heterogeneous computing power, the amount of neural layer parameter data is divided by the communication bandwidth, and the obtained quotient is summed with the communication delay to obtain the sub-memory access time.

[0137] After obtaining the communication delay and communication bandwidth of each parameter synchronization node accessing each heterogeneous computing power through CXL, and obtaining the neural layer parameter data volume of each backward transmission computing node, for each heterogeneous computing power, the neural layer parameter data volume is divided by the communication bandwidth, and the obtained quotient is summed with the communication delay to obtain the sub-memory access time.

[0138] S211: Sum up the access time of each sub-memory to obtain the memory access time.

[0139] After obtaining the sub-memory access time corresponding to each neural layer, the sub-memory access time is summed up to obtain the memory access time. It can be calculated using the following formula:

[0140] ;

[0141] in, is the memory access time of the parameter synchronization node, n is the number of heterogeneous computing power XPUs participating in distributed training, It is a processor used to handle parameter synchronization in the CXL switching network. for Access via CXL The delay, Heterogeneous computing power middle The computation task calculates the amount of parameter data of the neural layer, that is, during the synchronization process The amount of memory access data for each XPU involved in training, for Access via CXL communication bandwidth.

[0142] S212: For each forward transmission computing node and each backward transmission computing node, respectively obtain the neural layer computing complexity of each node in each target computing process graph.

[0143] After obtaining the memory access time, for each forward transmission computing node and each backward transmission computing node, the neural layer computational complexity of each node in each target computation process graph is obtained respectively.

[0144] The computational complexity of the neural layer of each node can be obtained from the computational characteristics of the neural layer. For example, for the fully connected layer with a weight matrix of [x, x], since the computational complexity of matrix multiplication of the shape of [a, b]*[b, c] is 2abc floating point operations (FLOPs), the computational complexity of the fully connected layer with a shape of [b, s, x]*[x, x] is 2bsx^2 floating point operations, where b is the data size (batch_size) allocated to each heterogeneous computing power, s is the first dimension of the input matrix of the fully connected layer, and x is the matrix side length of the square matrix.

[0145] S213: Obtain the computing capabilities of each heterogeneous computing power respectively.

[0146] Obtain the computing power of each heterogeneous computing power separately. The computing power of heterogeneous computing power can be recorded according to the computing power declared when purchasing each heterogeneous computing power, or it can be measured and recorded through some benchmarks or measurement tools.

[0147] S214: Calculate the neural layer calculation time of each node according to the computing power and the neural layer calculation complexity.

[0148] After obtaining the neural layer computational complexity of each node and the computing power of each heterogeneous computing power, the neural layer computational time of each node is calculated separately according to the computing power and the neural layer computational complexity.

[0149] For the forward transmission computing node in the computing process graph, Heterogeneous computing power Computational neural layer The time required is calculated using the formula:

[0150] ;

[0151] in, Heterogeneous computing power Computational tasks compute neural layers The computational complexity of Heterogeneous computing power The computing power of Heterogeneous computing power Computational tasks compute neural layers The amount of input data for the neural layer. Heterogeneous computing power middle The computation task calculates the amount of neural layer parameter data. and ,if For this training task, the local memory space of the heterogeneous computing power is opened up. and They represent the latency and bandwidth of heterogeneous computing power reading local memory. For this training task, the extended memory based on CXL interconnect is opened. and Respectively represent The latency and bandwidth of heterogeneous computing power reading remote CXL memory.

[0152] For the backward transmission computing node in the computing process graph, Heterogeneous computing power Computational neural layer The time required is obtained in a similar way and will not be described in detail. The calculation formula is:

[0153] ;

[0154] in, Heterogeneous computing power Computational tasks compute neural layers The computational complexity of Heterogeneous computing power Computational tasks compute neural layers The amount of neural layer input data, Heterogeneous computing power middle The calculation task calculates the parameter data volume of the neural layer, and the value is equal to .

[0155] S215: Obtain the accessible memory space preset by each heterogeneous computing power for the distributed training task respectively.

[0156] Each heterogeneous computing power may contain other processing tasks that are running. The other processing tasks that are running occupy part of the memory of the heterogeneous computing power. The accessible memory space preset by each heterogeneous computing power for the distributed training task is obtained separately.

[0157] S216: Obtain target access latency and target access bandwidth for each heterogeneous computing power to read each accessible memory space.

[0158] The memory space location opened by each heterogeneous computing power for distributed training tasks is recorded as local memory or CXL extended memory. After obtaining the preset accessible memory space of each heterogeneous computing power for distributed training tasks, obtain the target access latency and target access bandwidth of each heterogeneous computing power to read each accessible memory space.

[0159] S217: Obtain the neural layer parameter data volume and neural layer input data volume of each node in each target calculation process graph respectively.

[0160] Obtain the neural layer parameter data volume and neural layer input data volume of each node in each target calculation process diagram respectively. The input data volume of each neural layer can be obtained from the input structure of the neural layer. For example, for a fully connected layer with a weight matrix of [x, x], the neural layer input data volume is [b, s, x]*P*8 bytes. For other neural layers such as convolution and pooling, similar mathematical methods can be used to estimate the above parameters based on the characteristics of the neural layer, which will not be elaborated here.

[0161] S218: Calculate the memory access time of each node according to the target access delay, the target access bandwidth, the neural layer parameter data volume and the neural layer input data volume.

[0162] After obtaining the target access latency and target access bandwidth for each heterogeneous computing power to read each accessible memory space, and obtaining the neural layer parameter data volume and neural layer input data volume of each node, the memory access time of each node is calculated based on the target access latency, target access bandwidth, neural layer parameter data volume and neural layer input data volume.

[0163] In a specific implementation of the present invention, step S216 may include the following steps:

[0164] Step 1: When the accessible memory space is local memory, obtain the local access latency and local access bandwidth of the heterogeneous computing power reading the local memory;

[0165] Step 2: Determine the local access delay as the target access delay, and determine the local access bandwidth as the target access bandwidth.

[0166] For the convenience of description, the above two steps can be combined for explanation.

[0167] When the accessible memory space is local memory, obtain the local access latency and local access bandwidth of the heterogeneous computing power reading the local memory, determine the local access latency as the target access latency, and determine the local access bandwidth as the target access bandwidth. Therefore, the memory access time can be calculated based on the local access latency and local access bandwidth.

[0168] The memory bandwidth and latency of the computing power reading the local memory can be recorded according to the computing power declared when purchasing each heterogeneous computing power, or can be measured and recorded through some benchmarks or measurement tools.

[0169] In a specific implementation of the present invention, step S216 may include the following steps:

[0170] Step 1: When the accessible memory space is CXL memory, obtain the CXL access latency and CXL access bandwidth of heterogeneous computing power reading CXL memory;

[0171] Step 2: Determine the CXL access delay as the target access delay, and determine the CXL access bandwidth as the target access bandwidth.

[0172] For the convenience of description, the above two steps can be combined for explanation.

[0173] When the accessible memory space is CXL memory, obtain the CXL access latency and CXL access bandwidth of heterogeneous computing power reading CXL memory, determine the CXL access latency as the target access latency, and determine the CXL access bandwidth as the target access bandwidth. Therefore, the memory access time can be calculated based on the CXL access latency and CXL access bandwidth.

[0174] In a specific implementation of the present invention, step S218 may include the following steps:

[0175] Step 1: For each forward transmission computing node and each backward transmission computing node, the amount of neural layer input data and the amount of neural layer parameter data are summed up to obtain the total amount of data;

[0176] Step 2: Divide the total data volume by the target access bandwidth to obtain the data access time.

[0177] Step 3: Sum the data access time and target access latency to get the memory access time.

[0178] For the convenience of description, the above three steps can be combined for explanation.

[0179] For each forward transmission computing node and each backward transmission computing node, the neural layer input data volume and the neural layer parameter data volume are summed to obtain the total data volume, and the total data volume is divided by the target access bandwidth to obtain the data access time consumption, and the data access time consumption and the target access delay are summed to obtain the memory access time consumption. Thus, the memory access time consumption corresponding to each forward transmission computing node and each backward transmission computing node is obtained.

[0180] S219: For each forward transmission computing node and each backward transmission computing node, the larger of the neural layer computing time and the memory access time of each node is determined as the current neural layer processing time corresponding to the node.

[0181] For each forward transmission computing node and each backward transmission computing node, the larger of the neural layer computing time and the memory access time of each node is determined as the current neural layer processing time corresponding to the node. By using the larger of the neural layer computing time and the memory access time as the current neural layer processing time, it is ensured that the input data and model parameters of each sub-neural layer in the neural layer are obtained and the model calculations involved in each sub-neural layer are calculated and completed.

[0182] S220: For each parameter synchronization node, the memory access time of each node is determined as the current neural layer processing time corresponding to the node.

[0183] For each parameter synchronization node, since the synchronization process is only simple operations such as addition and averaging, the calculation amount of the parameter synchronization node can be ignored, and the memory access time of each parameter synchronization node is determined as the processing time of the current neural layer corresponding to the node.

[0184] S221: Predict the total time required for training the distributed training task based on the processing time of each current neural layer.

[0185] After obtaining the processing time of each current neural layer, the total time required for training the distributed training task is predicted based on the processing time of each current neural layer. For example, by summing up the processing time of each current neural layer, the calculated sum is determined as the total time. The embodiment of the present invention achieves the time prediction of distributed training tasks in a heterogeneous computing system based on CXL by modeling, describing and simulating distributed training tasks in a heterogeneous computing system based on CXL, and simultaneously considers the computing performance of heterogeneous computing power and the characteristics of CXL. The training time of multiple iterations of distributed training tasks in a heterogeneous computing system based on CXL can be accurately predicted.

[0186] In a specific implementation of the present invention, step S221 may include the following steps:

[0187] Step 1: Determine the initial node in each target calculation process graph as the currently running node;

[0188] Step 2: Construct a time-consuming list containing the processing time of the current neural layer corresponding to each currently running node;

[0189] Step 3: Select the minimum value of the processing time of each current neural layer from the time consumption list;

[0190] Step 4: Add the minimum value to the current training time;

[0191] Step 5: Calculate the difference between the processing time of each current neural layer in the time consumption list and the minimum value to obtain each time difference;

[0192] Step 6: Delete the zero value in each time difference from the time consumption list, and update the time consumption list according to each target calculation process diagram;

[0193] Step 7: Determine whether the current neural layer processing time corresponding to each initial node exists in the updated time-consuming list. If not, return to step 3. If yes, execute step 8.

[0194] Step 8: Obtain the number of times the current neural layer processing time corresponding to each initial node is added to the time consumption list, and select the minimum number of additions from each number of additions;

[0195] Step 9: When the minimum number of additions reaches a preset value, the current training time is obtained; wherein the preset value is 1 greater than the preset number of iterations;

[0196] Step 10: Determine the current training time as the total training time.

[0197] For the convenience of description, the above ten steps can be combined for explanation.

[0198] Determine the initial node in each target calculation process graph as the currently running node, construct a time consumption list containing the current neural layer processing time corresponding to each currently running node, select the minimum value of each current neural layer processing time from the time consumption list, add the minimum value to the current training time, calculate the difference between each current neural layer processing time in the time consumption list and the minimum value, and obtain each time difference, delete the zero value in each time difference from the time consumption list, and update the time consumption list according to each target calculation process graph.

[0199] Determine whether the current neural layer processing time corresponding to each initial node exists in the updated time consumption list. If not, continue to select the minimum value of each current neural layer processing time from the time consumption list to further accumulate the current training time. If so, obtain the number of additions of the pre-recorded current neural layer processing time corresponding to each initial node to the time consumption list, and select the minimum number of additions from each number of additions. When the minimum number of additions reaches the preset value, obtain the current training time. The preset value is 1 greater than the preset number of iterations, and the current training time is determined as the total time. By adding the processing time of each neural layer to the time consumption list according to the arrangement order of each node in the target calculation process diagram, and calculating the current training time in an accumulated manner, the time consumption list is updated by performing difference calculation, thereby promoting the continuous iteration of the model training process, realizing the automatic prediction of the total time consumption of distributed training tasks, and improving the accuracy of the prediction of the total time consumption of distributed training tasks.

[0200] In a specific implementation of the present invention, updating the time-consuming list according to each target calculation process diagram may include the following steps:

[0201] Step 1: Determine the node pointed to by the currently running node corresponding to the zero time difference in the target computing process graph as the new currently running node;

[0202] Step 2: Add the current neural layer processing time corresponding to the new currently running node to the time consumption list.

[0203] For the convenience of description, the above two steps can be combined for explanation.

[0204] After deleting the zero value in each time difference from the time-consuming list, the node pointed to by the currently running node corresponding to the zero time difference in the target computing process graph is determined as the new currently running node, such as Figure 6 As shown, after the forward transmission computing node 1 corresponding to the heterogeneous computing power 1 and the forward transmission computing layer 1 in the time consumption list is deleted, the forward transmission computing node 2 corresponding to the heterogeneous computing power 1 and the forward transmission computing layer 2 is determined as the new currently running node, the node pointed to by the currently running node corresponding to the zero time difference in the target computing process graph is determined as the new currently running node, and the current neural layer processing time corresponding to the new currently running node is added to the time consumption list. By determining the node pointed to by the currently running node corresponding to the zero time difference in the target computing process graph as the new currently running node, the training time is gradually accumulated according to the order of the neural layers, and the accuracy of the obtained training time is improved.

[0205] In a specific implementation of the present invention, adding the current neural layer processing time corresponding to the new currently running node to the time consumption list may include the following steps:

[0206] Step 1: Determine whether the new currently running node meets the running conditions. If yes, execute step 2; if not, execute step 3;

[0207] Step 2: Add the current neural layer processing time corresponding to the new currently running node to the time consumption list;

[0208] Step 3: Wait for the preset time and return to step 1.

[0209] For the convenience of description, the above three steps can be combined for explanation.

[0210] Pre-set the node running conditions. After the node pointed to by the currently running node corresponding to the zero time difference in the target calculation process diagram is determined as the new currently running node, before adding the current neural layer processing time corresponding to the new currently running node to the time consumption list, determine whether the new currently running node meets the running conditions. If so, add the current neural layer processing time corresponding to the new currently running node to the time consumption list. If not, wait for a preset time and determine again whether the new currently running node meets the running conditions. By pre-setting the node running conditions, the current neural layer processing time corresponding to the node is added to the time consumption list only when it is determined that the running conditions are met, thereby further ensuring the smooth execution of distributed training tasks and improving the accuracy of model training.

[0211] In a specific implementation of the present invention, determining whether a new currently running node meets the running conditions may include the following steps:

[0212] Step 1: When the new currently running node is a forward transmission computing node and is not an initial node, determine whether the current iteration number corresponding to the new currently running node is the same as the current iteration number of the parameter synchronization node pointing to the new currently running node, and is 1 less than the current iteration number of the forward transmission computing node pointing to the new currently running node;

[0213] Step 2: When the new currently running node is the initial node, determine whether the current iteration number corresponding to the new currently running node is the same as the current iteration number of each node pointing to the new currently running node;

[0214] Step 3: When the new currently running node is a backward transmission computing node, determine whether the current iteration number corresponding to the new currently running node is less than 1 of the current iteration number of the forward transmission computing node or the backward transmission computing node pointing to the new currently running node;

[0215] Step 4: When the new currently running node is a parameter synchronization node, determine whether the current iteration number corresponding to the new currently running node is less than 1 of the current iteration number of the backward transmission calculation node pointing to the new currently running node.

[0216] For the convenience of description, the above four steps can be combined for explanation.

[0217] The operating conditions are set in advance according to the relationship between the iteration numbers of the new currently running node and other nodes in the target calculation process diagram. When the new currently running node is a forward transmission calculation node and is not an initial node, it is determined whether the current iteration number corresponding to the new currently running node is the same as the current iteration number of the parameter synchronization node pointing to the new currently running node, and is 1 less than the current iteration number of the forward transmission calculation node pointing to the new currently running node; when the new currently running node is an initial node, it is determined whether the current iteration number corresponding to the new currently running node is the same as the current iteration number of each node pointing to the new currently running node. The number of times is the same; when the new currently running node is a backward transmission computing node, determine whether the current number of iterations corresponding to the new currently running node is 1 less than the current number of iterations of the forward transmission computing node or the backward transmission computing node pointing to the new currently running node; when the new currently running node is a parameter synchronization node, determine whether the current number of iterations corresponding to the new currently running node is 1 less than the current number of iterations of the backward transmission computing node pointing to the new currently running node; when the number of completed iterations is 0, the forward transmission computing node of the target computing process graph does not need to consider the parameter synchronization node and the backward transmission computing node on which it depends. Only when the conditions for the number of iterations are met, the current neural layer processing time corresponding to the node is added to the time-consuming list, thereby avoiding the situation where the execution order of the training tasks is confused, improving the accuracy of model training, and improving the efficiency of model training.

[0218] In distributed training, the training data and training subtasks used to train the model are split and shared among multiple computing resources. Distributed training can be used for traditional machine learning models, which are more suitable for computationally and time-intensive training tasks, such as training large deep neural network models.

[0219] Currently, the commonly used distributed training methods mainly include parameter server (PS) and allreduce. In the parameter server training method, each heterogeneous computing power will train an identical and complete neural network model, and multiple heterogeneous computing powers rely on a parameter server to synchronize the parameters of the neural network model. In an iteration of distributed training, each heterogeneous computing power will use the batch_size data allocated to it to train its own neural network. After a heterogeneous computing power completes training, it will send (push) the gradient of its own neural network to the parameter server. When the parameter server receives the gradients of all heterogeneous computing powers, it will synchronize the gradients and update the parameters, and then return (pull) the updated parameters to each heterogeneous computing power. After each heterogeneous computing power receives the parameters, it can proceed to the next training iteration. In the Allreduce training, similarly, each heterogeneous computing power will train an identical and complete neural network model, and in each iteration of distributed training, the parameters of the neural network in each heterogeneous computing power will be synchronized based on the collective communication of allreduce (such as ring-allreduce, tree-allreduce), thereby completing the distributed training, and no longer requiring a parameter server.

[0220] Corresponding to the above method embodiments, the present invention also provides a device for predicting training time consumption in heterogeneous computing power based on CXL. The device for predicting training time consumption in heterogeneous computing power based on CXL described below and the method for predicting training time consumption in heterogeneous computing power based on CXL described above can be referred to each other.

[0221] See also Figure 7 , Figure 7 The structure block diagram of a device for predicting training time consumption in heterogeneous computing power based on CXL in an embodiment of the present invention may include:

[0222] A task description information acquisition module 71 is used to obtain task description information of a distributed training task;

[0223] A calculation order and heterogeneous computing power determination module 72, used to determine the neural layer calculation order of the neural network in the distributed training task and the heterogeneous computing power corresponding to the distributed training task according to the task description information;

[0224] An initial process graph construction module 73 is used to construct initial calculation process graphs corresponding to the heterogeneous computing power according to the neural layer calculation sequence;

[0225] A target process diagram acquisition module 74 is used to add a parameter synchronization process based on CXL to the backward transmission calculation node in each initial calculation process diagram to obtain each target calculation process diagram;

[0226] The time-consuming acquisition module 75 is used to respectively obtain the neural layer computing time and memory access time of each node in the target computing process diagram; wherein each node includes a forward transmission computing node, a backward transmission computing node and a parameter synchronization node;

[0227] The total time consumption prediction module 76 is used to predict the total time consumption required for training the distributed training task according to the calculation time consumption of each neural layer and the memory access time consumption.

[0228] It can be seen from the above technical scheme that a distributed training time prediction framework based on CXL in heterogeneous computing power is proposed. By obtaining the task description information of the distributed training task, the neural layer calculation order of the neural network in the distributed training task and the heterogeneous computing power corresponding to the distributed training task are determined according to the task description information. The computing performance of the heterogeneous computing power and the characteristics of CXL are considered, and the time prediction of the distributed training task in the heterogeneous computing system based on CXL is realized. By constructing the initial calculation process graph corresponding to each heterogeneous computing power according to the neural layer calculation order, and adding the CXL-based parameter synchronization process to the backward transmission calculation node in the initial calculation process graph, direct access to the memory of each heterogeneous computing power is realized through CXL, and then model parameter synchronization is realized, without the need for communication synchronization, which greatly reduces the model parameter synchronization time. By combining the calculation time of each neural layer and the memory access time prediction for the total time required for distributed training tasks, the influence of memory access time is considered, and the accuracy of training time prediction is greatly improved.

[0229] In a specific embodiment of the present invention, the target process diagram obtaining module 74 includes:

[0230] A parameter synchronization node adding submodule is used to add parameter synchronization nodes between backward transmission calculation nodes of the same neural layer in each initial calculation process graph;

[0231] A neural layer acquisition submodule is used to respectively acquire the neural layer to which each parameter synchronization node belongs;

[0232] The parameter synchronization connection establishment submodule is used to establish a parameter synchronization connection between each parameter synchronization node and the forward transmission calculation node in the corresponding neural layer; wherein the parameter transmission direction of the parameter synchronization connection is from the parameter synchronization node to the forward transmission calculation node.

[0233] In a specific embodiment of the present invention, the time consumption obtaining module 75 includes:

[0234] The communication delay acquisition submodule is used to obtain the communication delay of each parameter synchronization node accessing each heterogeneous computing power through CXL for each parameter synchronization node;

[0235] The parameter data volume acquisition submodule is used to respectively acquire the neural layer parameter data volume of each backward transmission computing node in each target computing process graph;

[0236] The communication bandwidth acquisition submodule is used to obtain the communication bandwidth of each parameter synchronization node to access each heterogeneous computing power through CXL;

[0237] The sub-memory access time acquisition sub-module is used to divide the neural layer parameter data volume by the communication bandwidth for each heterogeneous computing power, and sum the obtained quotient with the communication delay to obtain the sub-memory access time;

[0238] The memory access time acquisition submodule is used to sum up the memory access time of each submodule to obtain the memory access time.

[0239] In a specific embodiment of the present invention, the time consumption obtaining module 75 includes:

[0240] A computational complexity acquisition submodule is used to obtain the neural layer computational complexity of each node in each target computational process graph for each forward transmission computational node and each backward transmission computational node;

[0241] The computing power acquisition submodule is used to obtain the computing power of each heterogeneous computing power;

[0242] The computation time calculation submodule is used to calculate the neural layer computation time of each node according to the computation power and neural layer computation complexity;

[0243] The accessible memory space acquisition submodule is used to respectively obtain the accessible memory space preset by each heterogeneous computing power for the distributed training task;

[0244] The target latency and target bandwidth acquisition submodule is used to obtain the target access latency and target access bandwidth of each heterogeneous computing power to read each accessible memory space;

[0245] The parameter data volume and input data volume acquisition submodule is used to respectively acquire the neural layer parameter data volume and the neural layer input data volume of each node in each target calculation process graph;

[0246] The memory access time consumption calculation submodule is used to calculate the memory access time consumption of each node according to the target access delay, target access bandwidth, neural layer parameter data volume and neural layer input data volume.

[0247] In a specific implementation of the present invention, the target delay and target bandwidth acquisition submodule includes:

[0248] A local latency and local bandwidth acquisition unit, used to acquire the local access latency and local access bandwidth of heterogeneous computing power reading local memory when the accessible memory space is local memory;

[0249] The first target delay and target bandwidth determining unit is configured to determine the local access delay as the target access delay, and determine the local access bandwidth as the target access bandwidth.

[0250] In a specific implementation of the present invention, the target delay and target bandwidth acquisition submodule includes:

[0251] A CXL latency and CXL bandwidth acquisition unit, used for acquiring the CXL access latency and CXL access bandwidth of heterogeneous computing power reading the CXL memory when the accessible memory space is the CXL memory;

[0252] The second target delay and target bandwidth determining unit is configured to determine the CXL access delay as the target access delay, and determine the CXL access bandwidth as the target access bandwidth.

[0253] In a specific embodiment of the present invention, the memory access time-consuming calculation submodule includes:

[0254] A total data volume obtaining unit is used to sum up the neural layer input data volume and the neural layer parameter data volume for each forward transmission computing node and each backward transmission computing node to obtain the total data volume;

[0255] A data access time acquisition unit, used for obtaining the data access time by dividing the total data volume by the target access bandwidth;

[0256] The memory access time acquisition unit is used to calculate the sum of the data access time and the target access delay to obtain the memory access time.

[0257] In a specific embodiment of the present invention, the total time consumption prediction module 76 includes:

[0258] The first current neural layer processing time determination submodule is used to determine the larger of the neural layer computing time and the memory access time of each node as the current neural layer processing time corresponding to the node for each forward transmission computing node and each backward transmission computing node;

[0259] The second current neural layer processing time-consuming determination submodule is used to determine the memory access time of each node as the current neural layer processing time corresponding to the node for each parameter synchronization node;

[0260] The total time consumption prediction submodule is used to predict the total time consumption required for training the distributed training task according to the processing time consumption of each current neural layer.

[0261] In a specific embodiment of the present invention, the total time consumption prediction submodule includes:

[0262] A currently running node determination unit is used to determine the initial node in each target calculation process graph as the currently running node;

[0263] A time-consuming list construction unit, used to construct a time-consuming list containing the current neural layer processing time corresponding to each currently running node;

[0264] A minimum value selection unit, used for selecting the minimum value of the processing time of each current neural layer from the time consumption list;

[0265] A training time accumulation unit is used to accumulate the minimum value to the current training time;

[0266] A time difference obtaining unit, used for calculating the difference between the processing time of each current neural layer in the time consumption list and the minimum value, to obtain each time difference;

[0267] A time-consuming list updating unit is used to delete the zero value in each time difference from the time-consuming list, and update the time-consuming list according to each target calculation process diagram;

[0268] A judging unit, used for judging whether the current neural layer processing time corresponding to each initial node exists in the updated time-consuming list;

[0269] A repeating execution unit, configured to repeat the step of selecting the minimum value of each current neural layer processing time from the time consumption list if it is determined that the current neural layer processing time corresponding to each initial node does not exist in the updated time consumption list;

[0270] A minimum addition times selection unit is used to obtain the number of additions of the current neural layer processing time corresponding to each initial node to the time consumption list, and select the minimum number of additions from each number of additions if it is determined that the current neural layer processing time corresponding to each initial node exists in the updated time consumption list;

[0271] The current training time acquisition unit is used to acquire the current training time when the minimum number of additions reaches a preset value; wherein the preset value is 1 greater than the preset number of iterations;

[0272] The total time consumption determining unit is used to determine the current training time consumption as the total time consumption.

[0273] In a specific implementation of the present invention, the time-consuming list updating unit includes:

[0274] A node determination subunit, used for determining the node pointed to by the currently running node corresponding to the zero time difference in the target computing process graph as a new currently running node;

[0275] The time-consuming adding subunit is used to add the current neural layer processing time corresponding to the new currently running node to the time-consuming list.

[0276] In a specific embodiment of the present invention, the time-consuming adding sub-unit is specifically for determining whether a new currently running node meets the running conditions; if so, executing the step of adding the current neural layer processing time corresponding to the new currently running node to the time-consuming list; if not, waiting for a preset time, and repeating the step of determining whether the new currently running node meets the running conditions.

[0277] In a specific embodiment of the present invention, the time-consuming addition subunit includes:

[0278] A first judgment subunit is used to judge whether the current number of iterations corresponding to the new currently running node is the same as the current number of iterations of the parameter synchronization node pointing to the new currently running node, and is 1 less than the current number of iterations of the forward transmission computing node pointing to the new currently running node, when the new currently running node is a forward transmission computing node and is not an initial node;

[0279] The second judgment subunit is used to judge whether the current iteration number corresponding to the new currently running node is the same as the current iteration number of each node pointing to the new currently running node when the new currently running node is the initial node;

[0280] a third judgment subunit, for judging, when the new currently running node is a backward transmission computing node, whether the current iteration number corresponding to the new currently running node is less than 1 by the current iteration number of the forward transmission computing node or the backward transmission computing node pointing to the new currently running node;

[0281] The fourth judgment subunit is used to judge whether the current iteration number corresponding to the new currently running node is less than 1 of the current iteration number of the backward transmission calculation node pointing to the new currently running node when the new currently running node is a parameter synchronization node.

[0282] Corresponding to the above method embodiment, see Figure 8 , Figure 8 This is a schematic diagram of a training time-consuming prediction device in heterogeneous computing power based on CXL provided by the present invention. The device may include:

[0283] A memory 332, for storing computer programs;

[0284] The processor 322 is used to implement the steps of the method for predicting the training time consumption in heterogeneous computing power based on CXL in the above method embodiment when executing the computer program.

[0285] For details, please refer to Fig. 9 , Fig. 9A schematic diagram of the specific structure of a training time prediction device in heterogeneous computing power based on CXL provided in this embodiment, the training time prediction device in heterogeneous computing power based on CXL may have relatively large differences due to different configurations or performances, and may include a processor (central processing units, CPU) 322 (for example, one or more processors) and a memory 332, and the memory 332 stores one or more computer programs 342 or data 344. Among them, the memory 332 can be a short-term storage or a persistent storage. The program stored in the memory 332 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations in the data processing device. Furthermore, the processor 322 can be configured to communicate with the memory 332, and execute a series of instruction operations in the memory 332 on the training time prediction device 301 based on CXL in heterogeneous computing power.

[0286] The training time prediction device 301 in heterogeneous computing power based on CXL may also include one or more power supplies 326 , one or more wired or wireless network interfaces 350 , one or more input and output interfaces 358 , and / or one or more operating systems 341 .

[0287] The steps in the method for predicting the training time consumption in heterogeneous computing power based on CXL described above can be implemented by the structure of the device for predicting the training time consumption in heterogeneous computing power based on CXL.

[0288] Corresponding to the above method embodiment, the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the following steps can be implemented:

[0289] Obtain task description information of the distributed training task; determine the neural layer calculation order of the neural network in the distributed training task and the heterogeneous computing power corresponding to the distributed training task according to the task description information; construct the initial calculation process graph corresponding to each heterogeneous computing power according to the neural layer calculation order; add a CXL-based parameter synchronization process to the backward transmission calculation node in the initial calculation process graph to obtain each target calculation process graph; obtain the neural layer calculation time and memory access time of each node in the target calculation process graph respectively; wherein each node includes a forward transmission calculation node, a backward transmission calculation node and a parameter synchronization node; predict the total time required for training the distributed training task according to the calculation time of each neural layer and the memory access time.

[0290] The computer-readable storage medium may include: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and other media that can store program codes.

[0291] For an introduction to the computer-readable storage medium provided by the present invention, please refer to the above method embodiment, and the present invention will not be elaborated here.

[0292] Corresponding to the above method embodiment, the present invention further provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of the training time prediction method in heterogeneous computing power based on CXL.

[0293] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device, computer-readable storage medium, and computer program product disclosed in the embodiment, since they correspond to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part description.

[0294] The principles and implementation methods of the present invention are described in this article using specific examples. The description of the above embodiments is only used to help understand the technical solution and core ideas of the present invention. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present invention, the present invention can also be improved and modified, and these improvements and modifications also fall within the scope of protection of the present invention.

Claims

1. A method for predicting training time in heterogeneous computing power based on CXL, characterized in that: include: Get the task description information of the distributed training task; Determine, according to the task description information, a neural layer calculation order of the neural network in the distributed training task and the heterogeneous computing power corresponding to the distributed training task; Constructing initial calculation process diagrams corresponding to the heterogeneous computing power according to the neural layer calculation sequence; Add the parameter synchronization process based on CXL to the backward transmission computing nodes in each initial computing process graph to obtain each target computing process graph; Respectively obtain the neural layer computing time and memory access time of each node in the target computing process diagram; wherein each node includes a forward transmission computing node, a backward transmission computing node and a parameter synchronization node; According to the computation time of each neural layer and the memory access time, the total time required for training the distributed training task is predicted.

2. The method for predicting training time in heterogeneous computing power based on CXL according to claim 1, characterized in that: Add a CXL-based parameter synchronization process to the backward transmission calculation nodes in each initial calculation process diagram, including: Add the parameter synchronization node between the backward transmission calculation nodes of the same neural layer in each initial calculation process graph; Obtain the neural layer to which each parameter synchronization node belongs respectively; A parameter synchronization connection is established between each parameter synchronization node and a forward transmission computing node in the corresponding neural layer; wherein the parameter transmission direction of the parameter synchronization connection is from the parameter synchronization node to the forward transmission computing node.

3. The method for predicting training time consumption in heterogeneous computing power based on CXL according to claim 1, characterized in that: The memory access time of each node in the target computing process graph is obtained respectively, including: For each parameter synchronization node, obtain the communication delay of each parameter synchronization node accessing each heterogeneous computing power through CXL; Obtain the neural layer parameter data of each backward transmission computing node in each target computing process graph respectively; Obtain the communication bandwidth of each parameter synchronization node to access each heterogeneous computing power through CXL; For each heterogeneous computing power, the amount of neural layer parameter data is divided by the communication bandwidth, and the obtained quotient is summed with the communication delay to obtain the sub-memory access time. The access time consumption of each sub-memory is summed up and calculated to obtain the memory access time consumption.

4. The method for predicting training time consumption in heterogeneous computing power based on CXL according to claim 1, characterized in that: Obtaining the neural layer computing time and memory access time of each node in the target computing process graph respectively, including: For each forward transmission computing node and each backward transmission computing node, respectively obtain the neural layer computing complexity of each node in each target computing process graph; Obtain the computing power of each heterogeneous computing power separately; Calculating the neural layer computing time of each node according to the computing power and the neural layer computing complexity; Respectively obtain the accessible memory space preset by each heterogeneous computing power for the distributed training task; Obtain the target access latency and target access bandwidth for each heterogeneous computing power to read each accessible memory space; Respectively obtain the neural layer parameter data volume and neural layer input data volume of each node in each target calculation process graph; The memory access time consumption of each node is calculated respectively according to the target access delay, the target access bandwidth, the neural layer parameter data volume and the neural layer input data volume.

5. The method for predicting training time consumption in heterogeneous computing power based on CXL according to claim 4 is characterized in that: Obtain the target access latency and target access bandwidth of each heterogeneous computing power to read each accessible memory space, including: When the accessible memory space is a local memory, obtaining a local access latency and a local access bandwidth of the heterogeneous computing power to read the local memory; The local access delay is determined as the target access delay, and the local access bandwidth is determined as the target access bandwidth.

6. The method for predicting training time consumption in heterogeneous computing power based on CXL according to claim 4, characterized in that: Obtain the target access latency and target access bandwidth of each heterogeneous computing power to read each accessible memory space, including: When the accessible memory space is a CXL memory, obtaining a CXL access latency and a CXL access bandwidth of the heterogeneous computing power reading the CXL memory; The CXL access latency is determined as the target access latency, and the CXL access bandwidth is determined as the target access bandwidth.

7. The method for predicting training time consumption in heterogeneous computing power based on CXL according to claim 6, characterized in that: According to the target access delay, the target access bandwidth, the neural layer parameter data volume and the neural layer input data volume, respectively calculating the memory access time consumption of each node, including: For each forward transmission computing node and each backward transmission computing node, the amount of neural layer input data and the amount of neural layer parameter data are summed up to obtain a total amount of data; The data access time is obtained by dividing the total data volume by the target access bandwidth; The data access time consumption and the target access delay are summed up to obtain the memory access time consumption.

8. The method for predicting training time consumption in heterogeneous computing power based on CXL according to claim 1, characterized in that: According to the computation time of each neural layer and the memory access time, the total time required for training the distributed training task is predicted, including: For each forward transmission computing node and each backward transmission computing node, the larger of the neural layer computing time and the memory access time of each node is determined as the current neural layer processing time corresponding to the node; For each parameter synchronization node, the memory access time of each node is determined as the current neural layer processing time corresponding to the node; The total time required for training the distributed training task is predicted according to the processing time of each current neural layer.

9. The method for predicting training time consumption in heterogeneous computing power based on CXL according to claim 8, characterized in that: Predicting the total time required for training the distributed training task according to the processing time of each current neural layer, including: Determine the initial node in each target calculation process graph as the currently running node; Construct a time-consuming list containing the processing time of the current neural layer corresponding to each currently running node; Select the minimum value of the processing time of each current neural layer from the time-consuming list; Add the minimum value to the current training time; Calculate the difference between the processing time of each current neural layer in the time-consuming list and the minimum value to obtain each time difference; Deleting the zero value in each time difference from the time-consuming list, and updating the time-consuming list according to each target calculation process diagram; Determine whether the current neural layer processing time corresponding to each initial node exists in the updated time-consuming list; If not, repeat the step of selecting the minimum value of the processing time of each current neural layer from the time consumption list; If yes, then obtaining the number of times the current neural layer processing time corresponding to each pre-recorded initial node is added to the time consumption list, and selecting the minimum number of additions from each number of additions; When the minimum number of additions reaches a preset value, the current training time is obtained; wherein the preset value is 1 greater than the preset number of iterations; The current training time is determined as the total time.

10. The method for predicting training time consumption in heterogeneous computing power based on CXL according to claim 9, characterized in that: The time-consuming list is updated according to each target calculation process diagram, including: Determine the node pointed to by the currently running node corresponding to the zero time difference in the target computing process graph as the new currently running node; The current neural layer processing time corresponding to the new currently running node is added to the time consumption list.

11. The method for predicting training time consumption in heterogeneous computing power based on CXL according to claim 10, characterized in that: Adding the current neural layer processing time corresponding to the new currently running node to the time consumption list includes: Determine whether the new currently running node meets the running conditions; If yes, then executing the step of adding the current neural layer processing time corresponding to the new currently running node to the time consumption list; If not, wait for a preset time period and repeat the step of determining whether the new currently running node meets the running conditions.

12. The method for predicting training time consumption in heterogeneous computing power based on CXL according to claim 11, characterized in that: Determine whether the new currently running node meets the running conditions, including: When the new currently running node is a forward transmission computing node and is not an initial node, determine whether the current iteration number corresponding to the new currently running node is the same as the current iteration number of the parameter synchronization node pointing to the new currently running node, and is 1 less than the current iteration number of the forward transmission computing node pointing to the new currently running node; When the new currently running node is the initial node, determine whether the current iteration number corresponding to the new currently running node is the same as the current iteration number of each node pointing to the new currently running node; When the new currently running node is a backward transmission computing node, determining whether the current iteration number corresponding to the new currently running node is less than 1 of the current iteration number of the forward transmission computing node or the backward transmission computing node pointing to the new currently running node; When the new currently running node is a parameter synchronization node, it is determined whether the current iteration number corresponding to the new currently running node is less than 1 of the current iteration number of the backward transmission calculation node pointing to the new currently running node.

13. A training time prediction device in heterogeneous computing power based on CXL, characterized in that: include: Memory for storing computer programs; A processor is used to implement the steps of the method for predicting training time consumption in heterogeneous computing power based on CXL as described in any one of claims 1 to 12 when executing the computer program.

14. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the method for predicting training time consumption in heterogeneous computing power based on CXL as described in any one of claims 1 to 12.

15. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method for predicting training time consumption in heterogeneous computing power based on CXL as described in any one of claims 1 to 12 are implemented.

Images