Heterogeneous computing system and training time prediction method, device, medium, and product thereof

Abstract

The present invention discloses a heterogeneous computing system and its training time prediction method, equipment, medium, and product, which are applied to the field of artificial intelligence technology. Among them, the method includes obtaining user training plan data corresponding to the training task to be deployed, obtaining the maximum resource supply data provided by the heterogeneous computing system to meet the user training plan data; based on the workflow of each neural layer of the network model to be trained in a single iterative training process, predicting the calculation amount and memory reading amount of each neural layer, and determining the node time consumption information of each neural layer in combination with the maximum resource supply data. According to the node time consumption information of each neural layer of the network model to be trained, determine the single iteration time of the heterogeneous computing system to train the network model to be trained. The present invention can solve the problem that the related technology can only predict the time consumption of the actually deployed training tasks and the accuracy is not high, and can accurately predict the single iteration training time of a single computing unit without actually deploying the training tasks.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a heterogeneous computing system and a training time prediction method, device, medium, and product thereof. Background Art

[0002] With the rapid development of artificial intelligence technology, the scale of neural network models is getting larger and larger. Limited by the huge demand for computing resources and storage resources, the computing equipment requirements for performing large-scale neural network model training tasks are also getting higher and higher.

[0003] In order to efficiently complete the training task of large-scale neural network models, related technologies deploy the tasks to be executed on heterogeneous computing systems, and then predict the computing time of the heterogeneous computing systems in a single training iteration based on the tasks and the computing power of the heterogeneous computing systems. However, this method cannot predict the time consumption without actually deploying and executing the training, and the accuracy of the time consumption prediction is not high.

[0004] In view of this, without actually deploying training tasks in heterogeneous computing systems, a high-precision prediction of the training time required for a single iteration of the training tasks to be deployed is achieved.

[0005] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present application, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention

[0006] The present invention provides a heterogeneous computing system and a training time prediction method, an electronic device, a non-volatile storage medium, and a computer program product, which can accurately predict the time required for a single computing unit of a heterogeneous computing system to execute a single iteration of a training task to be deployed without actually deploying the training task.

[0007] In order to solve the above technical problems, the present invention provides the following technical solutions:

[0008] On one hand, the present invention provides a method for predicting the training time consumption of a heterogeneous computing system, comprising:

[0009] Obtain user training plan data corresponding to the training task to be deployed; the user training plan data includes the network model to be trained and the target computing unit that the user needs to execute the training task to be deployed; obtain the maximum resource supply data provided by the heterogeneous computing system to meet the user training plan data; based on the workflow of each neural layer of the network model to be trained during a single iterative training process, predict the calculation amount and memory reading amount of each neural layer respectively, and determine the node time consumption information of each neural layer according to the calculation amount and memory reading amount of each neural layer and the maximum resource supply data; determine the single iterative time consumption of the target computing unit to train the network model to be trained according to the node time consumption information of each neural layer of the network model to be trained.

[0010] In a first exemplary embodiment, the workflow of each neural layer based on the network model to be trained in a single iterative training process includes: obtaining each neural layer of the network model to be trained; generating a training task execution graph according to the workflow of each neural layer in a training iteration process, so as to use the training task execution graph to describe the processing flow of each neural layer on the training sample data and an update process of the model parameters of the network model to be trained.

[0011] In a second exemplary embodiment, the training task execution graph includes a forward propagation execution graph and a backpropagation execution graph. The training task execution graph is generated according to the workflow of each neural layer in a training iteration process, including: in a training iteration process, based on the processing flow of training sample data of each neural layer of the network model to be trained in the forward propagation process, and each neural layer is identified as a forward propagation node to generate a forward propagation execution graph; based on the data processing flow of each neural layer in the network model to be trained in the backpropagation process, and according to whether the neural layer weight is operated, each neural layer is identified as a backpropagation weighted node or a backpropagation unweighted node, and a backpropagation execution graph connected to the forward propagation execution graph is generated; wherein the forward propagation node is a neural layer participating in the forward propagation calculation in a training iteration process, the backpropagation weighted node is a neural layer participating in the backpropagation calculation in a training iteration process and having a weight operation requirement, and the backpropagation unweighted node is a neural layer participating in the backpropagation calculation in a training iteration process and having no weight operation requirement; the weight operation requirement includes a weight update operation requirement and a weight save operation requirement.

[0012] In a third exemplary embodiment, the predicting of the amount of computation and memory reading of each neural layer includes: traversing each neural layer of the training task execution graph from beginning to end in the order from forward propagation to backward propagation; if there is a neural layer at the current position, obtaining the current neural layer, and predicting the amount of computation and memory reading of the current neural layer; if there are multiple parallel neural layers at the current position, randomly obtaining the first neural layer, and after predicting the amount of computation and memory reading of the first neural layer, randomly obtaining the second neural layer, and predicting the amount of computation and memory reading of the second neural layer, and executing the loop until the computation and memory reading of each parallel neural layer are predicted.

[0013] In a fourth exemplary embodiment, the obtaining of the maximum resource supply data provided by the heterogeneous computing system to meet the user training plan data includes: determining the training accuracy, the target memory and the target computing unit required for the user to execute the training task to be deployed by parsing the user training plan data corresponding to the training task to be deployed; obtaining the peak computing power reached by the target computing unit when the heterogeneous computing system meets the training accuracy; obtaining the memory bandwidth from the target memory to the target computing unit, and using the peak computing power and the memory bandwidth as the maximum resource supply data.

[0014] In a fifth exemplary embodiment, the training accuracy, the target memory and the target computing unit required for the user to execute the training task to be deployed are determined by parsing the user training plan data corresponding to the training task to be deployed, including: obtaining the memory identification information of the target memory required for the user to execute the training task to be deployed in the heterogeneous computing system by parsing the user training plan data corresponding to the training task to be deployed; obtaining the computing unit identification information of the target computing unit required for the user to execute the training task to be deployed in the heterogeneous computing system by parsing the user training plan data corresponding to the training task to be deployed.

[0015] In a sixth exemplary embodiment, the respectively predicting the amount of computation and the amount of memory read of each neural layer includes: determining the amount of computation of each neural layer according to the computation stage and the type of computation of each neural layer of the network model to be trained during an iterative training process; determining the amount of memory read of each neural layer according to the computation stage of each neural layer during an iterative training process, the amount of input training sample data and the model parameter information.

[0016] In a seventh exemplary embodiment, the computational amount of each neural layer is determined according to the computational stage and computational type of each neural layer of the network model to be trained during an iterative training process, including: if the current neural layer is a neural layer participating in forward propagation calculations during an iterative training process, the computational amount of the current neural layer is calculated according to the neural layer computational type to which the current neural layer belongs; if the current neural layer is a neural layer participating in back propagation calculations during an iterative training process, the computational amount of the current neural layer is determined according to the computational amount of the neural layer corresponding to the current neural layer in the forward propagation calculation process and a preset adjustment factor.

[0017] In an eighth exemplary embodiment, the memory read amount of each neural layer is determined according to the calculation stage of each neural layer in an iterative training process, the input training sample data volume and model parameter information, including: if the current neural layer is a neural layer participating in forward propagation calculation in a training iteration, the memory read amount of the current neural layer is determined according to the training sample data volume input to the current neural layer, the parameter volume of the current neural layer and the training accuracy data volume; if the current neural layer is a neural layer participating in back-propagation calculation in a training iteration and having weight operation requirements, the memory read amount of the current neural layer is determined according to the input training sample data volume, output data volume, gradient parameter and optimizer state corresponding to the neural layer in the forward propagation calculation process; if the current neural layer is a neural layer participating in back-propagation calculation in a training iteration and not having weight operation requirements, the memory read amount of the current neural layer is determined according to the input training sample data volume corresponding to the neural layer in the forward propagation calculation process.

[0018] In a ninth exemplary embodiment, the node timing information of each neural layer is determined based on the amount of calculation and memory reading of each neural layer and the maximum resource supply data, including: determining the model operation characteristic value based on the amount of calculation and memory reading of each neural layer; wherein the model operation characteristic value is used to represent the relative proportion of the calculation operations and memory access operations of the network model to be trained during the operation process; and determining the node timing information of each neural layer based on the numerical relationship between the model operation characteristic value and the maximum resource supply data.

[0019] In a tenth exemplary embodiment, the node timing information of each neural layer is determined according to the numerical relationship between the model operation characteristic value and the maximum resource supply data, including: determining the maximum resource supply data according to the ratio between the peak computing power of the user's required computing power in the heterogeneous computing system and the memory bandwidth from the target memory to the target computing unit; wherein the peak computing power is the peak computing power of the target computing unit required by the user in the heterogeneous computing system, and the target memory is the memory required by the user; if the maximum resource supply data is greater than or equal to the model operation characteristic value, then the node timing information of the current neural layer is determined according to the memory reading amount of the current neural layer and the memory bandwidth from the target memory to the target computing unit; if the maximum resource supply data is less than the model operation characteristic value, then the node timing information of the current neural layer is determined according to the computing amount of the current neural layer and the peak computing power.

[0020] The present invention also provides an electronic device, comprising a processor, wherein the processor is used to implement the steps of any one of the above-mentioned methods for predicting the training time consumption of a heterogeneous computing system when executing a computer program stored in a memory.

[0021] The present invention also provides a non-volatile storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of any one of the above methods for predicting the training time consumption of a heterogeneous computing system are implemented.

[0022] The present invention also provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps of any one of the above-mentioned methods for predicting the training time consumption of a heterogeneous computing system.

[0023] Finally, the present invention also provides a heterogeneous computing system, including a time consumption prediction processor and multiple computing devices; wherein, at least one computing device includes a first computing unit and a second computing unit with different computing capabilities and types; the first computing unit includes at least a first local video memory and a first extended memory, and the second computing unit includes at least a second local video memory and a second extended memory, and the first local video memory, the first extended memory, the second local video memory, and the second extended memory serve as system local memory resources to be used by the first computing unit and the second computing unit; when the time consumption prediction processor executes the computer program stored in the memory, it implements the steps of the heterogeneous computing system training time consumption prediction method as described in any of the previous items.

[0024] The advantage of the technical solution provided by the present invention is that before the user actually deploys the task to be trained on the heterogeneous computing system and runs the software and hardware of the heterogeneous computing system, the user training plan data corresponding to the training task to be deployed and the hardware support data that the heterogeneous computing system can provide to meet the training task are collected. Taking into account the training task, computing unit, and memory characteristics in the user training plan data, the computational time of an iterative training process for the specified computing unit to execute the training task can be accurately predicted. By jointly evaluating the computing power and memory performance, the accuracy of the single iterative training time of the training task can be effectively improved. Before deploying the training task, the computing performance of the computing unit that executes the training task and the memory performance of the heterogeneous computing system are accurately predicted, and the hardware environment that cannot support model training is discovered in time, which is conducive to accurately selecting the hardware environment for model training and effectively and efficiently completing the model training task.

[0025] In addition, the present invention also provides corresponding heterogeneous computing systems, electronic devices, non-volatile storage media, and computer program products for the method of predicting the training time consumption of heterogeneous computing systems, further making the method more practical, and the heterogeneous computing systems, electronic devices, non-volatile storage media, and computer program products have corresponding advantages.

[0026] The technical features mentioned above, the technical features to be mentioned below, and the technical features shown separately in the drawings can be arbitrarily combined with each other, as long as the combined technical features are not contradictory. All feasible feature combinations are technical contents clearly recorded in this article. Any of the multiple sub-features contained in the same sentence can be applied independently, and does not have to be applied together with other sub-features. It should be understood that the above general description and the detailed description below are only exemplary and cannot limit the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to more clearly illustrate the technical solutions of the present invention or related technologies, the following briefly introduces the drawings required for use in the embodiments or related technical descriptions. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0028] Figure 1 A schematic diagram of a process flow of a method for predicting the time consumption of training a heterogeneous computing system provided by the present invention;

[0029] Figure 2 A schematic diagram of a flow chart of another method for predicting the training time consumption of a heterogeneous computing system provided by the present invention;

[0030] Figure 3A schematic diagram of a training task execution diagram in an exemplary application scenario provided by the present invention;

[0031] Figure 4 A structural framework diagram of an exemplary implementation of the heterogeneous computing system training time prediction device provided by the present invention;

[0032] Figure 5 A structural framework diagram of an exemplary embodiment of an electronic device provided by the present invention;

[0033] Figure 6 A structural framework diagram of an exemplary implementation of a heterogeneous computing system provided by the present invention;

[0034] Figure 7 A schematic diagram of the framework of the heterogeneous computing system provided by the present invention in an exemplary application scenario. DETAILED DESCRIPTION

[0035] In order to enable those skilled in the art to better understand the technical solution of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. Among them, the terms "first", "second", "third", etc. in the specification and the above-mentioned drawings are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "including" and "having" and any variations of the two are intended to cover non-exclusive inclusions. The term "exemplary" means "used as an example, embodiment or illustrative". Any embodiment described here as "exemplary" is not necessarily interpreted as being superior or better than other embodiments.

[0036] The training process of large-scale neural network models requires a lot of computing resources and memory resources. In order to meet the training tasks of large-scale neural network models, processors of different computing performance and types, such as graphics processors, central processing units, field programmable gate arrays, and computing cards, are usually connected to the same data center to form a heterogeneous computing system to perform large-scale neural network model training tasks. In addition, the storage resources of the data center are expanded through memory optimization technologies such as CXL (Compute Express Link, high-speed interconnection) technology, memory pooling technology, and separated memory, thereby alleviating the memory bottleneck problem encountered during training.

[0037] In order to ensure that the computing device that performs the training task can complete the task efficiently, the relevant technology is to predict the time consumption of the training task after the actual deployment of the training task and before the training task is executed. For example, one related technology is to predict the time consumption of the actual deployment in the model inference phase, and this method is applied to the prediction of the training time in the training phase. A related technology can only be applied to the prediction of the training time consumption of the convolutional network, and cannot be well predicted for other network model structures, such as the transformer architecture, and has poor universality. Another related technology can use the characteristics of the model training task to be completed by the computing unit, the dynamic characteristics of the computing unit and / or the dynamic characteristics of the server where the computing unit is located to predict the computing time consumption of the computing unit in a single training iteration, but it uses dynamic characteristics and cannot predict the time consumption without actually deploying and executing the training. It also lacks consideration for memory performance, resulting in low final prediction accuracy. Another related technology can predict the convergence time of the neural network model, but it predicts the convergence speed of the entire neural network model and cannot accurately predict the time consumption of each iteration of each computing unit in the heterogeneous computing system, so it is difficult to apply to dynamic load balancing. Another related technology is to predict the computational time of a single iteration of the entire heterogeneous computing system, but it is not possible to predict the computational time of a single iteration of a single computing unit in the heterogeneous computing system. Another related technology uses the computational time of the previous iteration of the computing unit to predict the computational time of the next iteration of the unit, but if there is a sudden performance change on the server where the computing unit is located, this method cannot achieve accurate prediction. It can be seen that the related technology cannot achieve high-precision prediction of the time it takes for a single computing unit of a heterogeneous computing system to execute a training task to be deployed in one iteration without actually deploying the training task in the heterogeneous computing system.

[0038] In view of this, before the user actually deploys the task to be trained on the heterogeneous computing system and runs the software and hardware of the heterogeneous computing system, the present invention collects the user training plan data corresponding to the training task to be deployed and the hardware support data that the heterogeneous computing system can provide to meet the training task, and takes into account the training task, computing unit, and memory characteristics in the user training plan data. The computing time of an iterative training process of the specified computing unit to perform the training task is accurately predicted, thereby realizing the prediction of the time consumed by a single computing unit of the heterogeneous computing system to perform an iterative process of the training task to be deployed on the basis of not actually deploying the training task in the heterogeneous computing system. After introducing the technical solution of the present invention, various non-limiting embodiments of the present invention are described in detail below. In order to better illustrate the present invention, numerous specific details are given in the specific embodiments below. Those skilled in the art should understand that the present invention can also be implemented without these specific details. In other examples, the methods, means, components and circuits well known to those skilled in the art are not described in detail in order to highlight the main purpose of the present invention.

[0039] First see Figure 1 , Figure 1 A flowchart of a method for predicting the training time consumption of a heterogeneous computing system provided in this embodiment may include the following contents:

[0040] S101: Obtain user training plan data corresponding to the training task to be deployed.

[0041] This step is to obtain the information of the training plan provided by the user from the user side before actually deploying the training task and running the software and hardware of the heterogeneous computing system. The training task to be deployed is the training task that the user is about to deploy in the heterogeneous computing system and execute using the computing resources and memory resources of the heterogeneous computing system. The user training plan data is the relevant information of the training task provided by the user, which at least includes the network model to be trained and the target computing unit that the user needs to execute the training task to be deployed. The network model to be trained is the neural network model structure that the user wants to train. The target computing unit is the hardware device where the computing resources of the heterogeneous computing system are located, that is, the user specifies a computing unit of the heterogeneous computing system to train the network model to be trained. Of course, the user training plan data can also include the training accuracy, model parameter information and the batch-size of the training samples used. The user can flexibly select the data information contained in the user training plan data according to the actual situation.

[0042] S102: Obtain maximum resource supply data provided by the heterogeneous computing system to meet the user's training plan data.

[0043] After the user training plan data is obtained in the previous step, before the actual deployment of training and operation of the software and hardware of the heterogeneous computing system, the necessary information required for evaluating the computing resources and memory resources that meet the user training needs can be obtained from the heterogeneous computing system or other pre-set information source locations based on the user training plan data. Among them, the user training needs are determined by the user training plan data in the previous step, and the maximum resource supply data is the maximum hardware support data that takes into account both memory resources and computing resources. In other words, it refers to the maximum computing resources and memory resources that the heterogeneous computing system can provide for users to achieve their desired training goals.

[0044] S103: Based on the workflow of each neural layer of the network model to be trained during a single iteration training process, the computation amount and memory read amount of each neural layer are predicted respectively, and the node time consumption information of each neural layer is determined according to the computation amount and memory read amount of each neural layer and the maximum resource supply data. According to the node time consumption information of each neural layer of the network model to be trained, the single iteration time consumption of the target computing unit to train the network model to be trained is determined.

[0045] This step predicts the time consumption based on the training-related data collected in the above steps and the hardware resource-related data of the heterogeneous computing system. The workflow refers to the process in which each neural layer of the network model to be trained processes the input training sample data and completes the model parameter update of the network model to be trained during an iterative training process. An iterative training process may include inputting the training sample data in the training sample data set into the network model to be trained according to the batch size in the user training scheme data or the pre-specified training parameters, and completing a forward propagation calculation and a back propagation calculation. Among them, the forward propagation calculation is to calculate the output value of each neural layer for the input training sample data using the current weight and bias value, and is the process in which the network model to be trained calculates the output result based on the input data. Then the error is calculated for each neuron in the output layer. When the back propagation algorithm calculates the error term, each neural layer is multiplied by the derivative of the activation function of this layer, and the gradient value of the loss function to the weight is calculated based on the error. The back propagation calculation is to update the weight and bias according to the calculated gradient value, so as to optimize the model parameters of the network model to be trained by the gradient descent method and reduce the error.

[0046] This embodiment takes into account both accuracy and computational complexity. This step uses the neural layer as the granularity, and predicts the computational operations and memory access operations performed by each neural layer in one iteration based on the workflow of each neural layer in one iteration training. Then, the time consumption information of each neural layer in one iteration training process is predicted in combination with the maximum resource supply data. For ease of distinction, it is defined as node time consumption information. When the node time consumption information of each neural layer is determined by considering the training tasks, heterogeneous computing power, and memory characteristics in the user training plan data, the computational time required for an iterative training process of a specified heterogeneous computing power to execute a training task can be accurately predicted. The final result, i.e., the single iteration training time consumption of the target computing unit, can be fed back to the user end, such as Figure 2 As described above, users can evaluate the pros and cons of their own training tasks to be deployed based on the final results. Of course, the results can also be sent to other designated clients for the demand side to evaluate the training tasks to be deployed. After deploying the training task, the user side or the computing power demander or computing power supplier can provide multiple deployment plans in advance, such as the user training plan data of S101, and use the above steps to obtain the single iteration time of each deployment plan, which is conducive to selecting the optimal deployment plan and reducing the resource consumption and time consumption of the final actual deployment.

[0047] In the technical solution provided in this embodiment, before the user actually deploys the task to be trained on the heterogeneous computing system and runs the software and hardware of the heterogeneous computing system, the user training plan data corresponding to the training task to be deployed and the hardware support data that the heterogeneous computing system can provide to meet the training task are collected. Taking into account the training task, computing unit, and memory characteristics in the user training plan data, the computational time of an iterative training process for the specified computing unit to perform the training task can be accurately predicted. By jointly evaluating the computing power and memory performance, the accuracy of the single iterative training time of the training task can be effectively improved. Before deploying the training task, the computing performance of the computing unit that executes the training task and the memory performance of the heterogeneous computing system are accurately predicted, and the hardware environment that cannot support model training is discovered in time, which is conducive to accurately selecting the hardware environment for model training and effectively and efficiently completing the model training task.

[0048] In the above embodiment, there is no limitation on how to determine the maximum resource supply data. Based on the above embodiment, the present invention further provides an exemplary method for determining the maximum resource supply data, which may include the following contents:

[0049] By parsing the user training plan data corresponding to the training task to be deployed, the training accuracy, the target memory and the target computing unit required by the user to execute the training task to be deployed are determined; the peak computing power reached by the target computing unit when the heterogeneous computing system meets the training accuracy is obtained; the memory bandwidth from the target memory to the target computing unit is obtained, and the peak computing power and memory bandwidth are used as the maximum resource supply data.

[0050] In this embodiment, the user training program data provided by the user is parsed, and the parsing method matches the pre-configured data upload method, that is, the user training program data uploaded by the user can be pre-configured in a specified format, and then the user training program is parsed according to the format, so as to obtain the corresponding parsing results. The parsing results of this embodiment may include training task information, memory deployment information and heterogeneous computing power information. The training task information includes but is not limited to the model structure and the training accuracy of the model. The memory deployment information is the memory of the heterogeneous computing system for the user to perform the training task requirements to be deployed. For the convenience of description, it is defined as the target memory. Since the computing power can only be fully utilized when the memory bandwidth is sufficient, this embodiment also needs to determine the memory bandwidth from the target memory to the target computing unit. The target memory can reflect the memory capacity, and the memory bandwidth can reflect the memory speed. In this embodiment, the maximum computing power that the target computing unit can provide in the heterogeneous computing system, that is, the peak computing power and the memory bandwidth from the target memory to the target computing unit, are used to represent the maximum resource supply data. Among them, the peak computing power information can be obtained through the performance manual of the heterogeneous computing system or related public information, or it can be manually entered into a specified location in advance and retrieved from the location when executing S102, or it can be obtained by searching the heterogeneous computing power execution information in the historical records and using the profiler (model performance analysis tool) provided by the deep learning framework. Similarly, the memory bandwidth from the target memory to the target computing unit can be obtained through a public manual, or it can be extracted and entered into a specified location and called when used, or it can be obtained using a test tool, such as sysbench (multi-threaded performance testing tool), which does not affect the implementation of the present invention.

[0051] From the above, it can be seen that this embodiment can represent the maximum hardware support provided by the heterogeneous computing system by utilizing the memory bandwidth and peak computing power, which is beneficial to improving the prediction accuracy of the time consumption of a single iteration and is beneficial to accurately evaluating the advantages and disadvantages of the deployment plan.

[0052] As a simple and convenient parsing method, it can also simplify user operations. When users upload user training plan data, only corresponding identification information can be filled in for the target memory and target computing unit. Accordingly, by parsing the user training plan data corresponding to the training task to be deployed, the memory identification information of the target memory in the heterogeneous computing system required by the user to execute the training task to be deployed is obtained; by parsing the user training plan data corresponding to the training task to be deployed, the computing unit identification information of the target computing unit in the heterogeneous computing system required by the user to execute the training task to be deployed is obtained.

[0053] In the above embodiment, there is no limitation on how to predict the memory read amount and the calculation amount of each neural layer. The present invention also provides an exemplary determination method of the memory read amount and the calculation amount, which may include the following contents:

[0054] Obtain each neural layer of the network model to be trained; generate a training task execution graph according to the workflow of each neural layer in a training iteration process, so as to use the training task execution graph to describe the processing flow of each neural layer on the training sample data and an update process of the model parameters of the network model to be trained.

[0055] In this embodiment, in order to improve the computational efficiency and computational accuracy of the node time-consuming information, the training task execution graph can be used to clearly and simply represent the workflow of each neural layer of the network model to be trained during a single iterative training process. The training task execution graph can be used to clearly define the data processing order of each neural layer and the input and output of each neural layer during an iterative training process. By traversing the training task execution graph in sequence, the computational operations and memory access operations of each neural layer can be predicted quickly and without omission.

[0056] For example, considering that an iterative training process includes two processes: forward propagation calculation and back propagation calculation, such as Figure 3As shown, the training task execution graph of the present invention may include a forward transmission execution graph and a back transmission execution graph. In a training iteration process, the processing flow of the training sample data in the forward propagation process of each neural layer of the network model to be trained and the update process of the model parameters of the network model to be trained can be used, and each neural layer can be identified as a forward transmission node to generate a forward transmission execution graph; wherein, the forward transmission node is a neural layer participating in the forward propagation calculation in a training iteration process. Based on the data processing flow of each neural layer in the back transmission process of the network model to be trained, and each neural layer is identified as a back transmission weighted node or a back transmission unweighted node according to whether the neural layer weight is operated, a back transmission execution graph connected to the forward transmission execution graph is generated, the back transmission weighted node is a neural layer participating in the back transmission calculation and having a weight operation requirement in a training iteration process, and the back transmission unweighted node is a neural layer participating in the back transmission calculation and having no weight operation requirement in a training iteration process; the weight operation requirement includes a weight update operation requirement and a weight preservation operation requirement. For example, the fully connected layer contains weights that need to be updated, and the activation layer does not have weights that need to be updated. Correspondingly, during the back-propagation calculation process, the fully connected layer is a back-propagation weighted node, and the activation layer is a back-propagation unweighted node.

[0057] After the training task execution graph is generated, each neural layer of the training task execution graph can be traversed from beginning to end in the order from forward propagation to backward propagation. If there is a neural layer at the current position, the current neural layer is obtained, and the computational amount and memory reading amount of the current neural layer are predicted; if there are multiple parallel neural layers at the current position, the first neural layer is randomly obtained, and after predicting the computational amount and memory reading amount of the first neural layer, the second neural layer is randomly obtained, and the computational amount and memory reading amount of the second neural layer are predicted. The loop is executed until the computational amount and memory reading amount of each parallel neural layer are predicted. In other words, when traversing the training task execution graph from beginning to end in the order of forward propagation and backward propagation, if parallel neural layers are included, the order between the parallel neural layers can be randomly selected and does not affect the final calculation result. For example, in Figure 3In the order from forward propagation to reverse propagation, and from the beginning to the end, the forward propagation node a can be read first, and then the calculation amount and memory reading amount of the forward propagation node a are predicted, and then the forward propagation node b is read, and then the calculation amount and memory reading amount of the forward propagation node b are predicted. For the parallel forward propagation nodes c and d, one of the forward propagation nodes can be randomly read, and after predicting the calculation amount and memory reading amount of the forward propagation node, another forward propagation node is read for prediction, and then the forward propagation node e is read, and then the calculation amount and memory reading amount of the forward propagation node e are predicted. After reading all the nodes in the forward propagation process, the same method is used to read and predict each neural layer of the reverse propagation calculation process, that is, the reverse propagation weighted node e, the reverse propagation weighted node c, the reverse propagation weighted node d, the reverse propagation unweighted node b, and the reverse propagation weighted node a.

[0058] Exemplarily, the method for predicting the amount of computation and memory reading of each neural layer in this embodiment may include: determining the amount of computation of each neural layer according to the computation stage and computation type of each neural layer of the network model to be trained during an iterative training process; determining the memory reading amount of each neural layer according to the computation stage of each neural layer during an iterative training process, the amount of input training sample data and model parameter information.

[0059] Among them, the calculation stage is the forward propagation calculation process and the back propagation calculation process. The calculation type corresponds to the neural layer type, that is, it refers to the data operation performed by the neural layer during the training process of the entire network model to be trained. The calculation amount is determined according to the neural layer type. If the current neural layer is the input layer, and the input layer does not perform any complex transformation operations, then its corresponding calculation type is: pass the received original to the next layer, such as the first hidden layer. In the input layer, each neuron usually corresponds to a feature dimension of the input data. In other words, the input layer does not involve calculation operations, and its corresponding calculation amount is 0. If the current neural layer is a hidden layer, and the hidden layer is used for feature extraction and pattern recognition, then its corresponding calculation type is: the neurons in the hidden layer are connected to all neurons in the previous layer through weights, receive signals from the previous layer, and process them through a nonlinear activation function. The calculation amount is determined according to this calculation type. If the current neural layer is the output layer, the output layer is used to output the model prediction results, and the corresponding calculation type is weighted summation through the weight matrix and bias term, and processed through an activation function to produce the final output. Since different neural layers process different tasks, when predicting the amount of computation of a neural layer, the corresponding type of computation can be used, and any computation estimation method of the relevant technology can be used, such as the calculation formula disclosed in the relevant technology or the computation acquisition tool provided by the deep learning framework to determine the amount of memory read. The memory read amount indicates the number of times the network model to be trained needs to read data from the memory during operation, that is, the memory access requirements of the network model to be trained during the training process. Since the computational stages of an iterative training process are different, the memory read operation is also different. Therefore, the computational stage also needs to be considered when determining the memory read amount of each neural layer. Model parameter information includes but is not limited to model parameter quantity, gradient, training accuracy, and optimizer related parameters.

[0060] Exemplarily, the calculation process of the computational amount of each neural layer may include: if the current neural layer is a neural layer participating in forward propagation calculations during a training iteration, the computational amount of the current neural layer is calculated according to the neural layer calculation type to which the current neural layer belongs; if the current neural layer is a neural layer participating in back propagation calculations during a training iteration, the computational amount of the current neural layer is determined according to the computational amount of the neural layer corresponding to the current neural layer in the forward propagation calculation process and a preset adjustment factor.

[0061] In this embodiment, the computational amount of the forward node can be estimated according to the computational type corresponding to the neural layer. Taking the fully connected layer as an example, if the fully connected layer does not consider the bias, the computational amount of the fully connected layer is determined according to the feature dimension of the output data and the computational amount of each neuron. The computational amount of the neuron can be determined according to the input data dimension, such as the input data dimension C. in ×(C in +1), the output data dimension is B×C out, the computational complexity of the fully connected layer is B×C out ×(C in ×(C in +1)). If the bias is taken into account, the computational complexity of the fully connected layer is B×(C in ×C out +C out The computation amount of the back-propagation weighted node can be determined based on the computation amount of the forward propagation node corresponding to the layer. Figure 3 For example, the back propagation weighted node c corresponds to the neural layer in the forward propagation calculation process as the forward propagation node c. After the calculation amount of the forward propagation node c is obtained by the above calculation, the calculation amount of the back propagation weighted node c can be calculated according to the calculation amount of the forward propagation node c. Among them, the preset adjustment factor can be determined according to the training process of the network model to be trained, for example, it can be 2. Correspondingly, the calculation amount of the back propagation weighted node is twice the calculation amount of the forward propagation node corresponding to this layer. Of course, other values ​​can also be taken, which does not affect the implementation of the present invention. The calculation amount of the back propagation unweighted node can also be determined according to the calculation amount of the forward propagation node corresponding to this layer. Figure 3 For example, the back propagation unweighted node b corresponds to the neural layer in the forward propagation calculation process as the forward propagation node b. After the calculation amount of the forward propagation node b is obtained by the above calculation, the calculation amount of the back propagation unweighted node b can be calculated according to the calculation amount of the forward propagation node b. Among them, the preset adjustment factor can be determined according to the training process of the network model to be trained, for example, it can be 2, and accordingly, the calculation amount of the back propagation unweighted node is twice the calculation amount of the forward propagation node corresponding to the layer. Of course, other values ​​can also be taken, which does not affect the implementation of the present invention.

[0062] Exemplarily, the prediction process of the memory read amount of each neural layer may include: if the current neural layer is a neural layer participating in forward propagation calculations during a training iteration, the memory read amount of the current neural layer is determined based on the amount of training sample data input to the current neural layer, the amount of parameters of the current neural layer, and the amount of training accuracy data; if the current neural layer is a neural layer participating in back-propagation calculations during a training iteration and has weight operation requirements, the memory read amount of the current neural layer is determined based on the amount of input training sample data, output data, gradient parameters, and data corresponding to the optimizer state of the current neural layer corresponding to the neural layer in the forward propagation calculation process; if the current neural layer is a neural layer participating in back-propagation calculations during a training iteration and does not have weight operation requirements, the memory read amount of the current neural layer is determined based on the amount of input training sample data corresponding to the neural layer in the forward propagation calculation process.

[0063] In this embodiment, the memory read amount of the forward node is determined according to the input data amount of the layer, the parameter amount of the current neural layer and the training precision data amount. The input data amount is proportional to the batch-size, and the input data amount and the parameter amount are determined by the calculation type corresponding to the neural layer, which can be obtained by the calculation formula disclosed in the relevant technology or the tool provided by the deep learning framework. Exemplarily, the memory read amount = (the input data amount of the layer + the parameter amount of the layer) × the training precision data amount, such as the fp32 precision is the parameter amount multiplied by 4 bytes. The memory read amount of the back propagation weight node is jointly determined by the input data amount, output data amount, gradient amount, parameter data amount and the data amount corresponding to the optimizer state of the forward node corresponding to the layer, wherein the input data amount, output data amount and parameter data amount can be obtained when calculating the memory read amount of the corresponding forward node, the gradient data amount is approximately equal to the parameter data amount, and the optimizer state data amount is determined by the optimizer type used in the current training process. For example, if the AdamW (optimizer name) optimizer is used, the data amount corresponding to the optimizer state is 2×the parameter data amount×the training precision. For the back-propagation unweighted nodes, since the back-propagation process only requires the gradient in the unwinding rule and does not need to update the parameters, there is no need to read the gradient and optimizer state. Therefore, the memory read amount of the back-propagation unweighted node can be approximated to the input data amount of the forward node corresponding to this layer.

[0064] From the above, it can be seen that this embodiment determines the calculation amount and memory reading amount of the neural layer according to the calculation type of the neural layer and the different calculation stages it is in during an iterative training process, which is closer to the calculation operations and memory operations of each neural layer in the actual training process, and is conducive to improving the prediction accuracy of a single iteration time.

[0065] In the above embodiment, there is no limitation on how to predict the node time consumption information. The present invention also provides an exemplary method for determining the node time consumption information, which may include the following contents:

[0066] The model operation characteristic value is determined according to the calculation amount and memory reading amount of each neural layer; the node time consumption information of each neural layer is determined according to the numerical relationship between the model operation characteristic value and the maximum resource supply data.

[0067] In this embodiment, the memory read amount reflects the memory access demand of the network model to be trained, and the ratio of the amount of calculation to the amount of memory read, that is, the model operation characteristic value, is used to represent the relative proportion of the computing operation and the memory access operation of the network model to be trained during operation, which can also reflect the relationship between the computing density and the memory access density of the network model to be trained during operation, and can also reflect the upper limit of the computing intensity of the target computing unit. Exemplarily, a higher ratio of the amount of calculation to the amount of memory read indicates that the computing operation of the network model to be trained is dominant during operation, and the memory access is relatively small, that is, the network model to be trained has a high demand for computing resources, but may rely less on memory. A lower ratio of the amount of calculation to the amount of memory read indicates that the memory access of the model is dominant during operation, and the computing operation is relatively small, which may mean that the memory access efficiency of the network model to be trained is high, but the utilization of computing resources may be low. Determining the ratio of the amount of calculation to the amount of memory read is conducive to selecting suitable hardware. For example, for scenarios requiring high-performance computing, hardware with higher computing power can be selected; for scenarios with intensive memory access, it is necessary to optimize the model design to reduce unnecessary memory access and improve efficiency.

[0068] Among them, the numerical relationship refers to the relationship between the numerical values ​​of the two. If the maximum resource supply data is greater than or equal to the model operation characteristic value, the node time consumption information of the current neural layer is determined according to the memory reading amount of the current neural layer and the memory-related data in the maximum resource supply data; if the maximum resource supply data is less than the model operation characteristic value, the node time consumption information of the current neural layer is determined according to the calculation amount of the current neural layer and the computing power-related data in the maximum resource supply data. Exemplarily, the maximum resource supply data can be determined according to the peak computing power and the memory bandwidth from the target memory to the target computing unit. The peak computing power is the peak computing power of the target computing unit required by the user in the heterogeneous computing system, and the memory is the target memory required by the user. Correspondingly, if the maximum resource supply data is greater than or equal to the model operation characteristic value, the node time consumption information of the current neural layer is determined according to the memory reading amount of the current neural layer and the memory bandwidth from the target memory to the target computing unit; if the maximum resource supply data is less than the model operation characteristic value, the node time consumption information of the current neural layer is determined according to the calculation amount and peak computing power of the current neural layer.

[0069] From the above, it can be seen that this embodiment jointly determines the node time consumption information based on the operating characteristics of the network model to be trained and the hardware support provided by the heterogeneous computing system. The node time consumption information is calculated while considering the training tasks in the deployment plan and the heterogeneous computing power and memory characteristics of the heterogeneous computing system. It is more in line with the calculation time of an iteration of the actual training task, and effectively improves the prediction accuracy of the single iteration time.

[0070] In order to further improve the calculation efficiency and calculation accuracy of the node time-consuming information of each neuron, based on the above embodiment, the present invention also provides an exemplary implementation method, which may include the following contents:

[0071] The resource intensity calculation formula and the model operation characteristic value calculation formula can be stored locally in advance. For each neural layer, after the calculation amount and memory reading amount of the current neural layer are predicted, the model operation characteristic value calculation formula is called to calculate the model operation characteristic value of the current neural layer. The peak computing power and the memory bandwidth from the target memory to the target computing unit are read from the maximum resource supply data, and the resource intensity calculation formula is called to calculate the resource upper limit value of the heterogeneous computing system. If the resource upper limit value is greater than or equal to the model operation characteristic value, the ratio of the memory reading amount of the current neural layer to the memory bandwidth from the target memory to the target computing unit is used as the node time consumption information of the current neural layer; if the resource upper limit value is less than the model operation characteristic value, the ratio of the calculation amount of the current neural layer to the peak computing power is used as the node time consumption information of the current neural layer.

[0072] The storage resource intensity calculation formula can be expressed as: S=Fmax / M b , S is the resource upper limit, Fmax is the peak computing power, expressed in FLOPS (floating point operations per second) or PFLOPS (quadrillion operations per second), M b is the memory bandwidth from memory to computing unit. The model running characteristic value calculation formula can be expressed as S`=F / M, where F is the amount of calculation and M is the amount of memory reading. Correspondingly, when S≥S`, the node time consumption information of the current neural layer is M / M b When S<S`, the node time consumption information of the current neural layer is F / Fmax.

[0073] It should be noted that there is no strict order of execution between the steps in the present invention. As long as they comply with the logical order, these steps can be executed simultaneously or in a certain preset order. Figure 1-Figure 2 This is just a schematic and does not mean that this is the only execution order.

[0074] The present invention also provides a corresponding device for the method for predicting the time consumption of training a heterogeneous computing system, which further makes the method more practical. Among them, the device can be described from the perspective of functional modules and hardware. The following is an introduction to the device for predicting the time consumption of training a heterogeneous computing system provided by the present invention. The device is used to implement the method for predicting the time consumption of training a heterogeneous computing system provided by the present invention. In this embodiment, the device for predicting the time consumption of training a heterogeneous computing system may include or be divided into one or more program modules, which are stored in a storage medium and executed by one or more processors to complete the method for predicting the time consumption of training a heterogeneous computing system disclosed in the first embodiment. The program module referred to in this embodiment refers to a series of computer program instruction segments that can complete specific functions, which is more suitable for describing the execution process of the device for predicting the time consumption of training a heterogeneous computing system in a storage medium than the program itself. The following description will specifically introduce the functions of each program module of this embodiment, and the device for predicting the time consumption of training a heterogeneous computing system described below and the method for predicting the time consumption of training a heterogeneous computing system described above can be referred to each other.

[0075] From the perspective of functional modules, see Figure 4 , Figure 4 The structural diagram of the heterogeneous computing system training time prediction device provided in this embodiment under a specific implementation manner, the device may include:

[0076] The training data collection module 401 is used to obtain user training plan data corresponding to the training task to be deployed; the user training plan data includes the network model to be trained and the target computing unit that the user needs to execute the training task to be deployed.

[0077] The system resource collection module 402 is used to obtain the maximum resource supply data provided by the heterogeneous computing system to meet the user's training plan data.

[0078] The time consumption prediction module 403 is used to predict the calculation amount and memory reading amount of each neural layer based on the workflow of each neural layer of the network model to be trained during a single iterative training process, and determine the node time consumption information of each neural layer according to the calculation amount and memory reading amount of each neural layer and the maximum resource supply data; according to the node time consumption information of each neural layer of the network model to be trained, determine the single iterative time consumption of the target computing unit to train the network model to be trained.

[0079] Exemplarily, in some implementations of the present embodiment, the time-consuming prediction module 403 can also be used to: obtain each neural layer of the network model to be trained; generate a training task execution graph according to the workflow of each neural layer during a training iteration, so as to use the training task execution graph to describe the processing flow of each neural layer on the training sample data and an update process of the model parameters of the network model to be trained.

[0080] As an exemplary implementation method of the above embodiment, the time-consuming prediction module 403 can also be further used for: the training task execution graph includes a forward transmission execution graph and a backward transmission execution graph. In a training iteration process, based on the processing flow of the training sample data of each neural layer of the network model to be trained in the forward propagation process and the update process of the model parameters of the network model to be trained, each neural layer is identified as a forward transmission node to generate a forward transmission execution graph; based on the data processing flow of each neural layer of the network model to be trained in the backward propagation process, and according to whether the neural layer weight is operated, each neural layer is identified as a backward propagation weighted node or a backward propagation unweighted node, and a backward transmission execution graph connected to the forward transmission execution graph is generated; wherein, the forward transmission node is a neural layer participating in the forward propagation calculation in a training iteration process, the backward propagation weighted node is a neural layer participating in the backward propagation calculation in a training iteration process and having a weight operation requirement, and the backward propagation unweighted node is a neural layer participating in the backward propagation calculation in a training iteration process and having no weight operation requirement; the weight operation requirement includes a weight update operation requirement and a weight preservation operation requirement.

[0081] As an exemplary implementation method of the above embodiment, the above-mentioned time-consuming prediction module 403 can be further used to: traverse each neural layer of the training task execution graph from beginning to end in the order from forward propagation to backward propagation; if there is a neural layer at the current position, obtain the current neural layer, and predict the calculation amount and memory reading amount of the current neural layer; if there are multiple parallel neural layers at the current position, randomly obtain the first neural layer, and after predicting the calculation amount and memory reading amount of the first neural layer, randomly obtain the second neural layer, and predict the calculation amount and memory reading amount of the second neural layer, and execute in a loop until the calculation amount and memory reading amount of each parallel neural layer are predicted.

[0082] Exemplarily, in some other implementations of this embodiment, the above-mentioned system resource collection module 402 can also be used to: determine the training accuracy, the target memory and the target computing unit required by the user to execute the training task to be deployed by parsing the user training plan data corresponding to the training task to be deployed; obtain the peak computing power reached by the target computing unit when the heterogeneous computing system meets the training accuracy; obtain the memory bandwidth from the target memory to the target computing unit, and use the peak computing power and memory bandwidth as the maximum resource supply data.

[0083] As an exemplary implementation method of the above embodiment, the above system resource collection module 402 can also be further used for: obtaining the memory identification information of the target memory of the heterogeneous computing system where the user needs to execute the training task to be deployed by parsing the user training plan data corresponding to the training task to be deployed; obtaining the computing unit identification information of the target computing unit in the heterogeneous computing system where the user needs to execute the training task to be deployed by parsing the user training plan data corresponding to the training task to be deployed.

[0084] Exemplarily, in some other implementations of this embodiment, the time consumption prediction module 403 can also be used to: determine the computational amount of each neural layer according to the computational stage and computational type of each neural layer of the network model to be trained during an iterative training process; determine the memory reading amount of each neural layer according to the computational stage of each neural layer during an iterative training process, the amount of input training sample data and model parameter information.

[0085] As an exemplary implementation method of the above embodiment, the above-mentioned time-consuming prediction module 403 can also be further used for: if the current neural layer is a neural layer participating in the forward propagation calculation in a training iteration process, the calculation amount of the current neural layer is calculated according to the neural layer calculation type to which the current neural layer belongs; if the current neural layer is a neural layer participating in the back-propagation calculation in a training iteration process, the calculation amount of the current neural layer is determined according to the calculation amount of the neural layer corresponding to the current neural layer in the forward propagation calculation process and a preset adjustment factor.

[0086] As another exemplary implementation method of the above embodiment, the above-mentioned time-consuming prediction module 403 can also be further used for: if the current neural layer is a neural layer participating in forward propagation calculation during a training iteration, the memory reading amount of the current neural layer is determined according to the amount of training sample data input to the current neural layer, the amount of parameters of the current neural layer and the amount of training accuracy data; if the current neural layer is a neural layer participating in back-propagation calculation during a training iteration and having weight operation requirements, the memory reading amount of the current neural layer is determined according to the amount of input training sample data, output data, gradient parameters and data corresponding to the optimizer state of the current neural layer corresponding to the neural layer in the forward propagation calculation process; if the current neural layer is a neural layer participating in back-propagation calculation during a training iteration and does not have a weight operation requirement, the memory reading amount of the current neural layer is determined according to the amount of input training sample data corresponding to the neural layer in the forward propagation calculation process.

[0087] Exemplarily, in some other implementations of this embodiment, the above-mentioned time consumption prediction module 403 can also be used to: determine the model operation characteristic value according to the calculation amount and memory reading amount of each neural layer; wherein the model operation characteristic value is used to represent the relative proportion of the calculation operations and memory access operations of the network model to be trained during the operation process; determine the node time consumption information of each neural layer according to the numerical relationship between the model operation characteristic value and the maximum resource supply data.

[0088] As another exemplary implementation method of the above embodiment, the above-mentioned time consumption prediction module 403 can also be further used to: determine the maximum resource supply data according to the ratio between the peak computing power of the user's required computing power in the heterogeneous computing system and the memory bandwidth from the target memory to the target computing unit; wherein the peak computing power is the peak computing power of the target computing unit required by the user in the heterogeneous computing system, and the target memory is the memory required by the user; if the maximum resource supply data is greater than or equal to the model operation characteristic value, then the node time consumption information of the current neural layer is determined according to the memory reading amount of the current neural layer and the memory bandwidth from the target memory to the target computing unit; if the maximum resource supply data is less than the model operation characteristic value, then the node time consumption information of the current neural layer is determined according to the computing amount and peak computing power of the current neural layer.

[0089] The heterogeneous computing system training time consumption prediction device mentioned above is described from the perspective of functional modules. Furthermore, the present invention also provides an electronic device, which is described from the perspective of hardware. Figure 5 The following is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention in one implementation manner. Figure 5 As shown, the electronic device includes a memory 50 for storing computer programs; a processor 51 for implementing the steps of the method for predicting the training time consumption of a heterogeneous computing system as mentioned in any of the above embodiments when executing the computer program.

[0090] Among them, the processor 51 may include one or more processing cores, may include a main processor and a coprocessor, and may also be integrated with a graphics processor and an artificial intelligence processor. The memory 50 may include one or more computer non-volatile storage media, and may also include a high-speed random access memory and a non-volatile memory, which may be an internal storage unit and / or an external storage device of an electronic device. It can not only be used to store application software and various types of data installed in the electronic device, for example: the code of the program in the process of executing the training time prediction method of the heterogeneous computing system, but also be used to temporarily store data that has been output or is to be output. In this embodiment, the memory 50 is at least used to store the following computer program 501, wherein the computer program can implement the relevant steps of the heterogeneous computing system training time prediction method disclosed in any of the above embodiments after being loaded and executed by the processor 51. In addition, the resources stored in the memory 50 may also include an operating system 502 and data 503, etc., and the storage method may be temporary storage or permanent storage. Among them, the operating system 502 may include Windows, Unix, Linux, etc. Data 503 may include but is not limited to data corresponding to the training time prediction result of the heterogeneous computing system.

[0091] In some embodiments, the electronic device may further include a display screen 52, an input / output interface 53, a communication interface 54 or a network interface, a power supply 55 and a communication bus 56. Those skilled in the art will appreciate that Figure 5 The structure shown in the figure does not constitute a limitation on the electronic device, and may include more or fewer components than those shown in the figure, for example, it may also include a sensor 57 for realizing various functions.

[0092] It can be understood that if the method for predicting the time consumption of training a heterogeneous computing system in the above embodiment is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a non-volatile storage medium. Based on this understanding, the technical solution of the present invention is essentially or partly contributed to the relevant technology or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium to perform all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes but is not limited to: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), electrically erasable programmable ROM, register, hard disk, multimedia card, card-type memory (such as SD or DX memory, etc.), magnetic memory, removable disk, CD-ROM, disk or optical disk, etc. Various media that can store program codes. Based on this, the present invention also provides a non-volatile storage medium on which a computer program is stored, and when the computer program is executed by a processor, the steps of the method for predicting the time consumption of training a heterogeneous computing system as described in any of the above embodiments are performed.

[0093] It is understandable that if the method for predicting the training time consumption of a heterogeneous computing system in the above-mentioned embodiment is implemented in the form of a software functional unit and sold or used as an independent product, the computer software product may not need to be stored in a physical storage medium. For example, it can be directly transmitted to a computer or other device with information processing capabilities through a wired network or a wireless network to execute all or part of the steps of the methods of each embodiment of the present invention. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the relevant technology or all or part of the technical solution can be embodied in the form of a software product. Based on this, the present invention also provides a computer program product storing a computer program. When the computer program is executed by a processor, the steps of the method for predicting the training time consumption of a heterogeneous computing system are as described in any of the above embodiments.

[0094] The present invention also provides a heterogeneous computing system, see Figure 6 , which may include the following:

[0095] The heterogeneous computing system may include a time-consuming prediction processor 61 and multiple computing devices 62, wherein at least one computing device 62 includes a first computing unit and a second computing unit with different computing capabilities and types; the first computing unit includes at least a first local video memory and a first extended memory, and the second computing unit includes at least a second local video memory and a second extended memory, and the first local video memory, the first extended memory, the second local video memory, and the second extended memory are used as system local memory resources to be used by the first computing unit and the second computing unit. The heterogeneous computing system of this embodiment deploys a group of computing units with multiple types of computing capabilities with different computing performance to a computing device, and the computing device may be, for example, a server. The computing units include but are not limited to GPU (Graphics Processing Unit), FPGA (Field Programmable Gate Array), digital signal processor, and computing cards such as NVIDIA computing acceleration card and Cambrian computing acceleration card. These heterogeneous computing units take advantage of different hardware architectures to provide high-performance heterogeneous computing for the entire system to meet diverse computing needs. The so-called heterogeneous computing is a parallel and distributed computing technology that can best match the parallelism type of computing tasks with the computing type that the machine can effectively support and can make the best use of various computing resources. The types of computing devices included in the heterogeneous computing system and the number of computing units included in each computing device can be flexibly selected according to actual conditions. The memory includes the local video memory of each computing unit, and the extended memory expanded using technologies such as CXL, memory pooling, and separated memory. All extended memories of the heterogeneous computing system and the local memory of the computing units can be called by other computing units and used directly as local memory resources. For example Figure 7 As shown, the first computing unit, the second computing unit, and the third computing unit include respective local memories, namely, a first local video memory, a second local video memory, and a third local video memory, and also include extended memory of a non-local memory device.

[0096] In this embodiment, the time-consuming prediction processor 61 and each computing device 62 can be connected by any communication method, such as wired connection or remote connection. The time-consuming prediction processor 61 can be deployed to a computing device in a heterogeneous computing system or one of the computing units of a computing device, which does not affect the implementation of the present invention. The computing unit or computing device deployed by the time-consuming processor 61 can be the device with the highest computing power performance in the heterogeneous computing system. As an exemplary embodiment, the time-consuming processor 61 can also be deployed to any server or cloud server, and the server is in communication with each computing unit of the heterogeneous computing system. When the time-consuming processor 61 executes the computer program stored in the memory, the steps of the heterogeneous computing system training time-consuming prediction method recorded in any of the above embodiments are implemented.

[0097] The above is a detailed introduction to a heterogeneous computing system and its training time prediction method, electronic device, non-volatile storage medium, and computer program product provided by the present invention. The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referenced to each other. Based on the embodiments of the present invention, for ordinary technicians in this technical field, all other embodiments obtained without creative work are within the scope of protection of the present invention. Without departing from the principles of the present invention, the present invention can also be improved and modified in a number of ways, and these improvements and modifications also fall within the scope of protection of the present invention.

Claims

1. A method for predicting the training time consumption of a heterogeneous computing system, characterized in that: include: Obtain user training plan data corresponding to the training task to be deployed; The user training scheme data includes a network model to be trained and a target computing unit, wherein the target computing unit is a computing unit in a user-specified heterogeneous computing system for training the network model to be trained; Obtaining maximum resource supply data provided by the heterogeneous computing system to meet the user training plan data; Based on the workflow of each neural layer of the network model to be trained in a single iterative training process, the calculation amount and memory reading amount of each neural layer are predicted respectively, and the node time consumption information of each neural layer is determined according to the calculation amount and memory reading amount of each neural layer and the maximum supply data of the resource; The single iteration time consumption of the target computing unit for training the network model to be trained is determined according to the node time consumption information of each neural layer of the network model to be trained.

2. The method for predicting the training time consumption of a heterogeneous computing system according to claim 1, characterized in that: The workflow of each neural layer in a single iteration training process based on the network model to be trained includes: Obtaining each neural layer of the network model to be trained; According to the workflow of each neural layer in a training iteration, a training task execution graph is generated to use the training task execution graph to describe the processing flow of each neural layer on the training sample data and an update process of the model parameters of the network model to be trained.

3. The method for predicting the training time consumption of a heterogeneous computing system according to claim 2, characterized in that: The training task execution graph includes a forward transmission execution graph and a back transmission execution graph. According to the workflow of each neural layer in a training iteration process, the training task execution graph is generated, including: In one training iteration, based on the processing flow of the training sample data by each neural layer of the network model to be trained during the forward propagation process, each neural layer is identified as a forward propagation node, and a forward propagation execution graph is generated; Based on the data processing flow of each neural layer in the back propagation process of the network model to be trained, and according to whether the neural layer weight is operated, each neural layer is identified as a back propagation weighted node or a back propagation unweighted node, and a back propagation execution graph connected to the forward propagation execution graph is generated; Among them, the forward propagation node is a neural layer that participates in the forward propagation calculation during a training iteration, the backward propagation weighted node is a neural layer that participates in the backward propagation calculation during a training iteration and has a weight operation requirement, and the backward propagation unweighted node is a neural layer that participates in the backward propagation calculation during a training iteration and does not have a weight operation requirement; the weight operation requirement includes a weight update operation requirement and a weight preservation operation requirement.

4. The method for predicting the training time consumption of a heterogeneous computing system according to claim 3, characterized in that: Predict the amount of computation and memory reads for each neural layer, including: Traversing each neural layer of the training task execution graph from beginning to end in the order from forward propagation to backward propagation; If there is a neural layer at the current position, obtain the current neural layer and predict the amount of computation and memory read of the current neural layer; If there are multiple parallel neural layers at the current position, the first neural layer is randomly obtained, and after predicting the computational amount and memory reading amount of the first neural layer, the second neural layer is randomly obtained, and the computational amount and memory reading amount of the second neural layer are predicted. The loop is executed until the computational amount and memory reading amount of each parallel neural layer are predicted.

5. The method for predicting the training time consumption of a heterogeneous computing system according to claim 1, characterized in that: Acquiring the maximum resource supply data provided by the heterogeneous computing system to meet the user training solution data, including: Determine the training accuracy, target memory and target computing unit required by the user to execute the training task to be deployed by parsing the user training plan data corresponding to the training task to be deployed; Obtaining the peak computing power achieved by the target computing unit when the heterogeneous computing system meets the training accuracy; The memory bandwidth from the target memory to the target computing unit is obtained, and the peak computing power and the memory bandwidth are used as the maximum resource supply data.

6. The method for predicting the training time consumption of a heterogeneous computing system according to claim 5, characterized in that: By parsing the user training plan data corresponding to the training task to be deployed, the training accuracy, the target memory and the target computing unit required by the user to execute the training task to be deployed are determined, including: By parsing the user training plan data corresponding to the training task to be deployed, the memory identification information of the target memory of the heterogeneous computing system where the user needs to execute the training task to be deployed is obtained; By parsing the user training plan data corresponding to the training task to be deployed, the computing unit identification information of the target computing unit required by the user to execute the training task to be deployed in the heterogeneous computing system is obtained.

7. The method for predicting the training time consumption of a heterogeneous computing system according to any one of claims 1 to 6, characterized in that: Predict the amount of computation and memory read for each neural layer, including: Determine the amount of calculation of each neural layer according to the calculation stage and calculation type of each neural layer of the network model to be trained during an iterative training process; The memory read amount of each neural layer is determined based on the amount of training sample data and model parameter information input to each neural layer.

8. The method for predicting the training time consumption of a heterogeneous computing system according to claim 7, characterized in that: According to the calculation stage and calculation type of each neural layer of the network model to be trained in an iterative training process, the calculation amount of each neural layer is determined, including: If the current neural layer is a neural layer that participates in forward propagation calculations during a training iteration, the calculation amount of the current neural layer is calculated according to the neural layer calculation type to which the current neural layer belongs; If the current neural layer is a neural layer participating in back propagation calculations during a training iteration, the calculation amount of the current neural layer is determined according to the calculation amount of the neural layer corresponding to the current neural layer in the forward propagation calculation process and a preset adjustment factor.

9. The method for predicting the training time consumption of a heterogeneous computing system according to claim 7, characterized in that: According to the amount of training sample data and model parameter information input to each neural layer, the memory read amount of each neural layer is determined, including: If the current neural layer is a neural layer participating in forward propagation calculation in a training iteration process, the memory read amount of the current neural layer is determined according to the amount of training sample data input to the current neural layer, the amount of parameters of the current neural layer and the amount of training accuracy data; If the current neural layer is a neural layer that participates in back-propagation calculations during a training iteration and has weight operation requirements, the memory read amount of the current neural layer is determined according to the amount of input training sample data, output data, gradient parameters, and data corresponding to the optimizer state of the neural layer corresponding to the current neural layer in the forward propagation calculation process; If the current neural layer is a neural layer that participates in back-propagation calculations during a training iteration and does not require weight operations, the memory read amount of the current neural layer is determined based on the amount of input training sample data corresponding to the neural layer in the forward propagation calculation process.

10. The method for predicting the training time consumption of a heterogeneous computing system according to any one of claims 1 to 6, characterized in that: The node time consumption information of each neural layer is determined according to the calculation amount and memory reading amount of each neural layer and the maximum supply data of the resource, including: Determine the model operation characteristic value according to the amount of calculation and memory reading of each neural layer; wherein the model operation characteristic value is used to represent the relative proportion of the calculation operation and memory access operation of the network model to be trained during operation; The node time consumption information of each neural layer is determined according to the numerical relationship between the model operation characteristic value and the resource maximum supply data.

11. The method for predicting the training time consumption of a heterogeneous computing system according to claim 10, characterized in that: Determining the node time consumption information of each neural layer according to the numerical relationship between the model operation characteristic value and the resource maximum supply data, including: Determine the maximum resource supply data according to the peak computing power of the computing power required by the user in the heterogeneous computing system and the ratio between the target memory and the memory bandwidth of the target computing unit; wherein the peak computing power is the peak computing power of the target computing unit required by the user in the heterogeneous computing system, and the target memory is the memory required by the user; If the maximum resource supply data is greater than or equal to the model operation characteristic value, determining the node time consumption information of the current neural layer according to the memory read amount of the current neural layer and the memory bandwidth from the target memory to the target computing unit; If the maximum resource supply data is less than the model operation characteristic value, the node time consumption information of the current neural layer is determined according to the calculation amount of the current neural layer and the peak computing power.

12. An electronic device, characterized in that: It comprises a processor and a memory, wherein the processor is used to implement the steps of the method for predicting the training time consumption of a heterogeneous computing system as claimed in any one of claims 1 to 11 when executing a computer program stored in the memory.

13. A non-volatile storage medium, characterized in that: The non-volatile storage medium stores a computer program, which, when executed by a processor, implements the steps of the method for predicting the training time consumption of a heterogeneous computing system as claimed in any one of claims 1 to 11.

14. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instruction is executed by a processor, the steps of the method for predicting the training time consumption of a heterogeneous computing system described in any one of claims 1 to 11 are implemented.

15. A heterogeneous computing system, characterized in that: including a time-consuming prediction processor and a plurality of computing devices; Wherein, at least one computing device includes a first computing unit and a second computing unit with different computing capabilities and types; the first computing unit includes at least a first local video memory and a first extended memory, and the second computing unit includes at least a second local video memory and a second extended memory, and the first local video memory, the first extended memory, the second local video memory, and the second extended memory are used as system local memory resources to be used by the first computing unit and the second computing unit; When the time consumption prediction processor executes the computer program stored in the memory, the steps of the method for predicting the time consumption of training a heterogeneous computing system as claimed in any one of claims 1 to 11 are implemented.

Images