Resource usage prediction for deep learning models

Abstract

According to implementations of the present disclosure, a scheme for predicting resource usage of a deep learning model is proposed. In the scheme, information related to the deep learning model is obtained. The information includes first information for describing the deep learning model, and second information related to a running environment of a task associated with the deep learning model. Static resource usage of the task is determined based on the first information. A strategy of the task when running in the running environment is determined based on the first information and the second information. Then, resource usage of the task when running in the running environment is predicted based on the strategy and the static resource usage. The scheme can accurately predict the usage of various resources of the deep learning model under a specific running strategy, such as computation power consumption, memory consumption, and execution time, etc. Furthermore, the scheme has a scalable architecture, facilitating support for various different types of deep learning frameworks.

Description

BACKGROUND

[0001] In recent years, deep learning has become very popular in the information technology industry and is changing people's lives. Understanding the accurate resource usage of a deep learning task throughout its entire life cycle is crucial for improving production efficiency. If the resource usage of a deep learning task can be accurately predicted, deep learning developers can pre-select the best model parameters to avoid failures due to insufficient resources and obtain the best model performance under resource constraints. In addition, if the resource usage of a deep learning task can be accurately predicted, the execution plan of the deep learning task can be dynamically adjusted to ensure that the deep learning task is completed under a given service level agreement (SLA) (e.g., to meet a given time or budget requirement). SUMMARY

[0002] According to implementations of the present disclosure, a scheme for predicting resource usage of a deep learning model is proposed. In the scheme, information related to the deep learning model is obtained. The information includes first information for describing the deep learning model, and second information related to a runtime environment of a task associated with the deep learning model. A static resource usage of the task is determined based on the first information. A strategy for running the task in the runtime environment is determined based on the first information and the second information. Then, the resource usage of the task when running in the runtime environment is predicted based on the strategy and the static resource usage. The scheme can accurately predict the usage of various resources of a deep learning model under a specific runtime strategy, such as compute consumption, memory consumption, and execution time, etc. In addition, the scheme has a scalable architecture, which facilitates supporting various different types of deep learning frameworks.

[0003] The summary is provided to introduce a selection of concepts in a simplified form that are further described below in the detailed description. The summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter. BRIEF DESCRIPTION OF DRAWINGS

[0004] Figure 1 A block diagram illustrating a computing device that is capable of implementing multiple implementations of the present disclosure is shown;

[0005] Figure 2 A block diagram illustrating an example system for predicting resource usage of a deep learning model according to implementations of the present disclosure is shown;

[0006] Figure 3A An example program code of a training task of a deep learning model is shown;

[0007] Figure 3BAn example computation graph corresponding to a training task for a deep learning model is shown in accordance with implementations of the present disclosure;

[0008] Figure 3C An example code in a deep neural network code base corresponding to a convolution operator in a forward propagation process is shown; and

[0009] Figure 4 A flowchart of an example method for predicting resource usage of a deep learning model is shown in accordance with implementations of the present disclosure.

[0010] In these drawings, like or similar elements are referred to with like or similar reference numerals. DETAILED DESCRIPTION

[0011] The present disclosure will now be discussed with respect to several example implementations. It should be appreciated that these implementations are discussed only to better illustrate the present disclosure and are not intended to limit the scope of the disclosure.

[0012] As used herein, the term “includes” and its variants are to be read as open-ended terms that mean “includes, but is not limited to.” The term “based on” is to be construed as “based at least in part on.” The terms “a” and “an” are to be construed as “at least one” The term “another” is to be construed as “at least one other.” The terms “first,” “second,” etc. can refer to different or the same objects. Other explicitly and implicitly recited definitions come with the context of the following claims as well.

[0013] As used herein, a “neural network model” (simply “model”) is capable of processing an input and providing a corresponding output, which typically includes an input layer and an output layer and one or more hidden layers between the input layer and the output layer. Models used in deep learning applications (referred to herein as “deep learning models” or “deep neural networks”) typically include many hidden layers, lengthening the depth of the network. The layers of a neural network model are connected in sequence, such that the output of a previous layer is provided as input to a subsequent layer, with the input layer receiving the input to the neural network model and the output of the output layer as the final output of the neural network model. Each layer of a neural network model includes one or more nodes (also referred to as processing nodes or neurons), each of which processes input from the previous layer. In this document, the terms “neural network,” “model,” “network,” and “neural network model” are used interchangeably.

[0014] As mentioned above, in recent years, deep learning has become very popular in the information technology industry and is changing people's lives. It is crucial to understand the accurate resource usage of a deep learning task throughout its entire life cycle to improve productivity. If the resource usage of a deep learning task can be accurately predicted, deep learning developers can pre-select the best model parameters to avoid failures due to insufficient resources and obtain the best model performance under resource constraints. In addition, if the resource usage of a deep learning task can be accurately predicted, the execution plan of the deep learning task can be dynamically adjusted to ensure that the deep learning task is completed under a given SLA (e.g., to meet a given time or budget requirement).

[0015] Some conventional solutions can only predict the execution time of a deep learning task based on the description information of the deep learning model and the specifications of the hardware executing the deep learning task, but cannot predict other resource consumptions of the deep learning task, such as computing power consumption, memory consumption, etc. In addition, some conventional solutions tend to be based on machine learning methods, and the types of description information they are suitable for and the types of machine learning model frameworks they are suitable for are usually limited, and the prediction results they provide tend to lack interpretability, for example, they cannot help deep learning developers find bottlenecks in the model, debug model parameters to provide model performance.

[0016] According to an implementation of the present disclosure, a solution for predicting resource usage of a deep learning model is proposed. In this solution, information related to a deep learning model is obtained. The information includes first information for describing the deep learning model, and second information related to a running environment of a task associated with the deep learning model. A static resource usage of the task is determined based on the first information. A strategy for running the task in the running environment is determined based on the first information and the second information. Then, the resource usage of the task when running in the running environment is predicted based on the strategy and the static resource usage.

[0017] In this way, the solution can accurately predict the usage of various resources of the deep learning model under a specific runtime strategy, such as computing power consumption, memory consumption and execution time. In addition, the solution can support various types of inputs and has an extensible architecture, which is convenient for supporting various types of deep learning frameworks. The prediction results provided by the solution are interpretable and can be connected with other deep learning system components (for example, AutoML systems) or artificial intelligence platforms to improve the production efficiency of deep learning. For example, the prediction results can help deep learning developers find the bottlenecks of the model, thereby improving model performance by debugging model parameters; the prediction results help other deep learning system components (for example, AutoML systems) to trim the search space of the model parameters of the deep learning model in advance to improve the efficiency of model parameter search; the prediction results help optimize the task execution strategy of the artificial intelligence platform, thereby improving resource utilization, etc.

[0018] Various example implementations of this solution are described in detail below in conjunction with the accompanying drawings.

[0019] Figure 1 FIG1 shows a block diagram of a computing device 100 capable of implementing various implementations of the present disclosure. It should be understood that Figure 1 The computing device 100 shown is merely exemplary and should not be construed as limiting the functionality and scope of the implementations described herein. Figure 1 As shown, computing device 100 comprises a computing device in the form of a general-purpose computing device 100. Components of computing device 100 may include, but are not limited to, one or more processors or processing units 110, memory 120, storage device 130, one or more communication units 140, one or more input devices 150, and one or more output devices 160.

[0020] In some implementations, the computing device 100 can be implemented as various user terminals or service terminals. The service terminal can be a server, a large computing device, etc. provided by various service providers. The user terminal is such as any type of mobile terminal, fixed terminal or portable terminal, including a mobile phone, a site, a unit, a device, a multimedia computer, a multimedia tablet, an Internet node, a communicator, a desktop computer, a laptop computer, a notebook computer, a netbook computer, a tablet computer, a personal communication system (PCS) device, a personal navigation device, a personal digital assistant (PDA), an audio / video player, a digital camera / camcorder, a positioning device, a television receiver, a radio broadcast receiver, an e-book device, a gaming device, or any combination thereof, including accessories and peripherals of these devices, or any combination thereof. It is also foreseeable that the computing device 100 can support any type of interface for the user (such as a "wearable" circuit, etc.).

[0021] Processing unit 110 may be a real or virtual processor and is capable of performing various processes according to a program stored in memory 120. In a multi-processor system, multiple processing units execute computer-executable instructions in parallel to increase the parallel processing capabilities of computing device 100. Processing unit 110 may also be referred to as a central processing unit (CPU), a microprocessor, a controller, or a microcontroller.

[0022] The computing device 100 typically includes a plurality of computer storage media. Such media can be any available media that is accessible to the computing device 100, including but not limited to volatile and non-volatile media, removable and non-removable media. The memory 120 can be volatile memory (e.g., registers, cache, random access memory (RAM)), non-volatile memory (e.g., read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof.

[0023] Storage device 130 may be a removable or non-removable medium and may include machine-readable media such as memory, a flash drive, a disk, or any other medium that can be used to store information and / or data and can be accessed within computing device 100. Computing device 100 may further include additional removable / non-removable, volatile / non-volatile storage media. Although not described in Figure 1 As shown in FIG, a magnetic disk drive for reading or writing from a removable, nonvolatile magnetic disk and an optical disk drive for reading or writing from a removable, nonvolatile optical disk can be provided. In these cases, each drive can be connected to a bus (not shown) by one or more data media interfaces.

[0024] The communication unit 140 enables communication with other computing devices via a communication medium. Additionally, the functionality of the components of the computing device 100 can be implemented as a single computing cluster or multiple computing machines that can communicate via a communication connection. Thus, the computing device 100 can operate in a networked environment using logical connections to one or more other servers, personal computers (PCs), or another general network node.

[0025] Input device 150 can be one or more various input devices such as a mouse, keyboard, trackball, voice input device, etc. Output device 160 can be one or more output devices such as a display, a speaker, a printer, etc. Computing device 100 can also include, as necessary, other input or output devices such as a disk drive or the like. Computing device 100 also includes a communication interface 140 taking the form of a modem, a network interface card, etc., for communicating with one or more other devices, such as storage devices, display devices, etc. and one or more devices that enable a user to interact with computing device 100. The communication can be via input / output (I / O) interfaces 170.

[0026] In some implementations, some or all of the components of computing device 100 can be provided in the form of a cloud computing architecture, in addition to being integrated on a single device. In a cloud computing architecture, the components can be arranged remotely and can work together to implement the functionality described in this disclosure. In some implementations, cloud computing provides computation, software, data access, and storage services that do not require end-user knowledge of the physical location or configuration of the system that delivers the services. In various implementations, cloud computing uses appropriate protocols over a wide area network, such as the Internet, to deliver services. For example, cloud computing providers deliver applications over a wide area network, and they can be accessed through a web browser or any other computing component. The software or components of a cloud computing architecture, and the corresponding data, can be stored on servers at remote locations. Computing resources in a cloud computing environment can be consolidated at a remote data center location or they can be dispersed. Cloud computing infrastructure can provide services through a shared data center, even though they appear as a single point of access for users. Thus, the components and functionality described herein can be provided from a service provider at a remote location using a cloud computing architecture. Alternatively, they can be provided from a conventional server, or they can be installed directly or otherwise on a client device.

[0027] Computing device 100 can be used to implement the prediction of resource usage of a deep learning model in various implementations of the present disclosure. Memory 120 can include a resource prediction module 122 having one or more program instructions that can be accessed and run by processing unit 110 to implement the functionality of the various implementations described herein.

[0028] In predicting resource usage of a deep learning model, the computing device 100 can receive information related to the deep learning model to be trained through the input device 150. The information can include information 170 (also referred to herein as “description information” or “first information”) describing the deep learning model and information 180 (also referred to herein as “execution information” or “second information”) related to a runtime environment of a task associated with the deep learning model. The “task” described herein is also referred to as “deep learning task”, which can include a task of training the deep learning model (simply referred to as “training task”) or a task of inference using the trained deep learning model (simply referred to as “inference task”). In some implementations, the description information 170 can include, for example, configuration parameters (e.g., hyperparameters) of the deep learning model. Additionally or alternatively, the description information 170 can include, for example, a model file (e.g., a binary model file) of the deep learning model and / or program code of the task (training task or inference task) associated with the deep learning model. The execution information 180 can include, for example, a framework type (e.g., TensorFlow or PyTorch) of the deep learning model, specifications and number of at least one computing device (e.g., central processing unit / graphics processing unit, network, storage device, etc.) in the runtime environment for executing the task, and / or an execution strategy of the task on the at least one computing device. The execution strategy can indicate, for example, whether the task is executed on a single machine or multiple machines, in a serial manner or in a parallel manner, by a single graphics processing unit or multiple graphics processing units, whether the task is executed in a distributed manner, and the like.

[0029] The description information 170 and the execution information 180 can be input to the resource prediction module 122 in the memory 120. The resource prediction module 122 can predict, based on the description information 170 and the execution information 180, resource usage 190 (also referred to as “prediction result 190”) of the task associated with the deep learning model at runtime. The prediction result 190 can indicate, for example, compute power consumption, memory consumption (e.g., consumption of main memory in a central processing unit or memory consumption of a graphics processing unit), and / or I / O resource consumption of the task when running in a given runtime environment with a given execution strategy. In addition, the prediction result 190 can also indicate, for example, other resource consumptions of the task at runtime, such as execution time, power consumption, budget to be consumed on a specific AI platform, and the like. The other resource consumptions can be determined based on, for example, compute power consumption and / or memory consumption.

[0030] Figure 2 A block diagram of an example system 200 for predicting resource usage of a deep learning model according to implementations of the present disclosure is shown. The system 200 can be implemented in the computing device 100, for example, at the resource prediction module 122. As shown in FIG. 2, the system 200 can include a description information input module 210, an execution information input module 220, a resource prediction module 230, and a prediction result output module 240. Figure 1 The description information input module 210 can receive the description information 170 from the input device 150. The execution information input module 220 can receive the execution information 180 from the input device 150. The resource prediction module 230 can predict, based on the description information 170 and the execution information 180, the resource usage 190 of the task associated with the deep learning model at runtime. The prediction result output module 240 can output the prediction result 190 to the output device 160.Figure 2 As shown, system 200 can include a parsing module 210 and a prediction module 220.

[0031] In some implementations, parsing module 210 can be configured to parse the description information 170 of a deep learning model to generate a computation graph 201 corresponding to the deep learning model. As previously described, the description information 170 can include, for example, a model file (e.g., a binary model file) of the deep learning model, program code used to train the deep learning model, and / or configuration parameters (e.g., hyperparameters) of the deep learning model, etc. In some implementations, parsing module 210 can generate computation graph 201 based on a program analysis method. The generated computation graph 201 can include a plurality of nodes, where the plurality of nodes correspond to a plurality of operators in the deep learning model (e.g., each node corresponds to one operator, and edges connecting the plurality of nodes can indicate dependency relationships between the plurality of operators. The generation of computation graph will be illustrated below with respect to Figure 3A and 3B .

[0032] Figure 3A An example program code 300 of a training task of a deep learning model is shown. The example program code 300 can be one example of the description information 170 as shown in Figure 2 The program code 300, for example, uses the PyTorch framework and constructs a convolutional neural network composed of convolution operators (e.g., self.conv() as shown in the program code 300), pooling operators (e.g., self.pool() as shown in the program code 300), and fully connected operators (e.g., self.fc() as shown in the program code 300). PyTorch is a popular deep learning framework that adopts a hybrid programming model in which Python code describes the high-level model structure of a deep learning program. Although the generation of computation graph is illustrated with respect to the PyTorch framework, it should be understood that this is merely for illustrative purposes and is not intended to limit the scope of the present disclosure. Implementations of the present disclosure are also applicable to other similar frameworks such as TensorFlow, MXNet, etc.

[0033] Figure 3B An example computation graph 305 of a model training task corresponding to the example program code 300 as shown in Figure 3A The example computation graph 305 can be one example of the computation graph 201 as shown in Figure 2

[0034] In some implementations, the execution of a deep neural network can be represented as a directed acyclic computation graph where each node u (i) ​are calls to an operator f to be executed on a graphics processing unit (GPU) (i) from node u (i) whose output points to the input of node u (j) whose output points to the input of node u (i) whose output points to the input of node u (j) indicate execution dependencies.

[0035] As shown in example computation graph 305, deep learning model training program 300 is a loop program that includes three stages: a forward propagation stage 310, corresponding to lines 21-22 of code as shown in Figure 3A , a backward propagation stage 320, corresponding to line 23 of code as shown in Figure 3A , and a gradient descent stage 330, corresponding to line 24 of code as shown in Figure 3A . As shown in Figure 3B , forward propagation stage 310 may, for example, include three nodes 311, 312, and 313, which correspond to respective operators in the deep learning model. For example, node 311 corresponds to a convolution operator (as shown in line 11 of code in Figure 3A ), node 312 corresponds to a pooling operator (as shown in line 12 of code in Figure 3A ), and node 313 corresponds to a fully connected operator (as shown in line 14 of code in Figure 3A ). Backward propagation stage 320 may, for example, include three nodes 321, 322, and 323, which correspond to the operators in forward propagation stage 310 in reverse. For example, node 321 corresponds to a convolution operator, node 322 corresponds to a pooling operator, and node 323 corresponds to a fully connected operator. Gradient descent stage 330 may, for example, include a node 331, which corresponds to a gradient descent operator (as shown in line 24 of code in Figure 3A ). The directed edges between the different nodes represent the data flow between the different operators.

[0036] Referring back to Figure 2In some implementations, the computation graph 201, the cost models (also referred to as "resource prediction models") 202 corresponding to the multiple operators in the computation graph 201, and the execution information 180 may be provided to the prediction module 220 for predicting the resource usage of the deep learning task at runtime (i.e., the prediction result 190). As previously described, the execution information 180 may include, for example, the framework type of the deep learning model (e.g., TensorFlow or PyTorch, etc.), the specifications and number of at least one computing device (e.g., central processing unit / graphics processing unit, network, storage device, etc.) used to execute the deep learning task in the runtime environment, and / or the execution policy of the deep learning task on the at least one computing device (e.g., the execution policy may indicate whether the task is executed on a single machine or multiple machines, in serial or parallel mode, by a single graphics processing unit or multiple graphics processing units, or in a distributed manner, etc.). The prediction result 190 may, for example, indicate the computing power consumption, memory consumption (e.g., main memory consumption in a central processing unit or memory consumption in a graphics processing unit), and / or I / O resource consumption of the deep learning task when it is executed in the execution environment indicated by the execution information 180 using a given execution strategy. Furthermore, the prediction result 190 may also indicate other resource consumptions of the deep learning task when it is executed in a specific execution environment, such as execution time, power consumption, budget to be consumed, etc. Other resource consumptions may be determined based on computing power consumption and / or memory consumption, for example.

[0037] like Figure 2 As shown, in some implementations, the prediction module 220 may include, for example, a static prediction unit 221, a simulation prediction unit 222, and a machine learning prediction unit 223. Figure 2 The functional units included in the prediction module 220 are merely exemplary and are not intended to limit the scope of the present disclosure. In a specific implementation, the prediction module 220 may include additional functional units or omit the functional units. Figure 2 The functional units shown in .

[0038] In some implementations, the static prediction unit 221 can predict the corresponding static resource usage of multiple operators in the computation graph based on the computation graph 201 and the cost model 202. In some implementations, the static resource usage of an operator can include the operator's memory consumption (e.g., the size of the output buffer, weight buffer, temporary workspace, etc.), the operator's computing power consumption (e.g., the number of floating-point operations, etc.), the operator's I / O resource consumption, etc. The static prediction unit 221 can determine the static resource usage of the deep learning task based on the corresponding static resource usage of the multiple operators. In some implementations, the simulation prediction unit 222 can determine the strategy for the deep learning task when it is executed in the specific execution environment indicated by the execution information 180, such as the resource allocation strategy and the execution strategy, based on the computation graph 201 and the execution information 180. The simulation prediction unit 222 can further perform a simulation analysis of resource consumption based on the determined strategy, thereby adjusting the determined static resource usage of the learning task based on the simulation results to obtain the resource usage 190 of the deep learning task during execution. Additionally or alternatively, the machine learning prediction unit 223 can utilize a trained machine learning model (or deep learning model) to generate parameters for optimizing the predicted resource usage 190 of the deep learning task at runtime, and optimize and calibrate the predicted resource usage 190 of the deep learning task at runtime based on the parameters to improve the accuracy of the resource usage prediction.

[0039] In some implementations, a cost model for predicting resource consumption can be pre-established for each of the multiple operators in the computation graph 201. In some implementations, the cost model can be established based on the computation graph and can be universal for different deep learning frameworks (such as TensorFlow, PyTorch, MXNet, etc.). Alternatively, in some implementations, corresponding cost models can be constructed for some operators based on different deep learning frameworks.

[0040] In some implementations, for a given operator (e.g., a specific operator), the same or different cost models can be established for different resources consumed by the operator (e.g., computing power, main memory / GPU memory, network I / O, etc.) to estimate the corresponding resource consumption caused by the operator. The following describes the construction of the cost model using memory resources as an example.

[0041] As Figure 3A Take the code 300 shown as an example, Figure 3A The Python code 300 shown hides the details of tensor allocation and deallocation. Figure 3BThe operators corresponding to each node in the illustrated computation graph 305, when executed on a GPU, can correspond to code in a respective deep neural network code library (e.g., cuDNN). Figure 3C An example 315 of cuDNN code corresponding to a convolution operator in a forward propagation process is shown. For example, Figure 3A The 6thline of code in Figure 3C will be executed as shown in the example code 315. In some implementations, the specific tensor allocation requirements for an operator can be determined from the cuDNN code, and a cost model corresponding to the operator can be constructed based on the cuDNN application programming interface and the implementation of the operator specific to the particular framework.

[0042] For example, for a convolution operator, by analyzing the example cuDNN code 315 as shown in Figure 3C , the details of the tensor types and specific allocations and deallocations in the forward and backward propagation processes of the convolution operator can be determined, and thus a corresponding cost model for the convolution operator can be determined. For example, as shown in the 4th-15thlines of code in Figure 3C , the required buffers (input, filter, output, workspace, etc.) need to be allocated first through the respective APIs. Since cuDNN is a closed code library, it can be configured to support only a few convolution algorithms. These algorithms use different workspace sizes depending on the tradeoff between speed and space. Thus, the workspace size can be obtained through the respective APIs or estimated according to the known algorithms to construct the corresponding cost model. Then, as shown in the 20th-26thlines of code in Figure 3C , the convolution operation will be performed. Once the convolution operation is performed, if the allocated buffers are not dependent on other operators, they can be deallocated through the respective APIs (as shown in the 29thline of code in Figure 3C ). In this way, by analyzing the example code 315, a cost model for the memory consumption of the convolution operator can be constructed for predicting the memory consumption of the convolution operator. Similarly, cost models for other types of resource consumption can also be constructed in advance for the convolution operator.

[0043] In some implementations, a unified cost model can be established for estimating resource consumption such as memory consumption, compute consumption (e.g., the amount of floating point operations consumed), I / O related resource consumption (e.g., I / O time, throughput, etc.), and other resource consumption (e.g., execution time, budget, power consumption, etc. based on compute and memory consumption). The construction of the cost model will be explained in detail below with examples of GPU memory and floating point operations (FLOPs).

[0044] As described above, the execution of a deep neural network can be represented as a directed acyclic computation graph where each node u (i) is a call to an operator f (i) to be executed on a graphics processing unit (GPU), a directed edge {(u (i) , u (j) )} with the output of node u (i) pointing to the input of node u (j) indicates an execution dependency.

[0045] In some implementations, the total memory consumption M of a deep neural network can be determined by accumulating the memory consumption of all operators, as shown in the following equation (1).

[0046] M = (M cg + M r )(1 + a) (1)

[0047] where M cg represents the sum of memory consumption of all operators determined based on the computation graph, M r represents the runtime memory consumption, and a is a pre-set ratio. For example, a can be set to 0 in most cases. When a machine learning model is used to optimize the predicted resource usage, it can be set to the ratio of the potential memory consumption predicted by the machine learning model to the total memory consumption.

[0048] In estimating M cg , the memory consumed by each operator u, M(u), can be added up:

[0049]

[0050] The runtime memory consumption M r may include GPU memory reserved by the specific hardware platform (such as for kernel launch or cuDNN processing) and allocator fragmentation M f , as shown in the following equation (3).

[0051] M r = M c + M f (3)

[0052] For each operator u, the memory consumed by it, M(u), can include the sum of one or more of the following: a weight buffer W(u), an output buffer O(u), and a temporary buffer E(u), as shown in the following equation (4).

[0053] M(u) = W(u) + O(u) + E(u) (4)

[0054] where the weight buffer W(u) can include a buffer Wm (u) and a buffer W for caching the gradient of the weight of the operator g (u), as shown in the following equation (5).

[0055] W(u) = W m (u) + W g (u) (5)

[0056] The output buffer O(u) can include a buffer O for caching the forward propagation output of the operator o (u) and a buffer O for caching the derivative of the output of the operator in the backward propagation phase g (u), as shown in the following equation (6).

[0057] O(u) = O o (u) + O g (u) (6)

[0058] It should be noted that the statistics of the above buffers should avoid duplication since the input of an operator is the output of the previous operator. The temporary buffer E(u) can include the following parts: a temporary workspace E w (u) (such as cuDNN convolution operator workspace) and temporary buffers E v (u) consumed by other operators in the deep neural network code library, as shown in the following equation (7).

[0059] E(u) = E w (u) + E v (u) (7)

[0060] The above memory consumption estimation is a complete set of memory consumption estimation for operators, and the memory consumption of different operators can be a subset of the complete set.

[0061] In some implementations, the total computing power consumption F cg (e.g., total floating point operation) of the deep neural network can be determined by accumulating the computing power consumption of all operators, as shown in the following equation (8).

[0062]

[0063] where the computing power consumption of each operator u is denoted as F(u).

[0064] Taking the convolution operator as an example, assuming it is implemented as a sliding window, its floating-point operation cost can be determined based on the scale of the matrix operations involved. Here, it is assumed that the size of the set of input feature maps of the convolution operator is N×C×H×W, where N represents the number of input feature maps, C represents the number of channels, and H×W represents the shape of each feature map (e.g., the number of rows and columns of the corresponding matrix). Here, it is assumed that the size of the set of convolution filters used by the convolution operator is K×C×R×S, where K represents the number of convolution filters, C represents the number of channels, and R×S represents the shape of each convolution filter (e.g., the number of rows and columns of the corresponding matrix). Here, it is assumed that the output of the convolution operator is N×K feature maps, where each feature map has a shape of P×Q (e.g., it can be determined based on the shape of the input feature maps and filters combined with additional stride and padding parameters). Assuming the floating-point operation cost of a floating-point multiplication-addition operation is 2, the total floating-point operation cost of the convolution operator in the forward propagation phase is 2×K×C×R×S×N×P×Q. Accordingly, the floating-point operation amount of other operators can be determined in a similar manner.

[0065] The above describes the construction of the cost model according to the implementation of the present disclosure using memory consumption and computing power consumption as examples. It should be understood that code models for other types of resource consumption can be constructed in a similar manner, which will not be described in detail here.

[0066] The cost model above only considers static estimates of resource usage. Taking memory consumption as an example, when deep learning tasks are running, the framework (e.g., TensorFlow, PyTorch, or MXNet, etc.) will perform runtime memory management and optimization, which will affect the estimation of memory consumption. For example, Figure 3C In the example shown, the size of the workspace corresponding to lines 13-15 depends on one of several strategies for balancing space and time. Figure 3C When line 29 of the code is called will be affected by the computation graph and the liveness of the tensor. The framework's built-in allocator will round and align the Figure 4 The amount of space allocated for lines 4-15 of code in [ 15 ]. In addition, the framework may also include hidden memory overhead that is difficult to infer from the program. These factors together determine the memory allocation strategy of the deep learning task at runtime. In addition, the specific execution environment of the deep learning task, such as the specifications and number of computing devices used to execute the deep learning task (e.g., central processing unit / graphics processing unit, network, storage device, etc.) and the execution strategy of the deep learning task on the computing device, will also affect the estimation of memory consumption.

[0067] In some implementations, to make the prediction of resource usage of a deep learning model more accurate, the simulation prediction unit 222 can determine a resource allocation strategy and an execution strategy of the deep learning task at runtime based on the computation graph 201 and the execution information 180. The resource allocation strategy may, for example, include a memory allocation strategy as described above and / or other resource allocation strategies. The execution strategy may, for example, indicate whether the deep learning task is executed on a single machine or multiple machines, executed in a serial manner or a parallel manner, executed by a single graphics processing unit or multiple graphics processing units, or executed in a distributed manner. The simulation prediction unit 222 can adjust the static resource usage determined by the static prediction unit 221 based on at least one of the determined resource allocation strategy and the execution strategy to obtain the resource usage of the deep learning task when running in a specific runtime environment.

[0068] The static resource prediction and the simulation-based resource prediction described above can be adapted to most deep learning frameworks. However, certain frameworks can involve complex resource allocation strategies, resulting in fluctuations in the accuracy of resource prediction. In some implementations, to further optimize the predicted resource usage, a trained machine learning model can be utilized to predict potential resource consumption of a deep learning task when running in a specific runtime environment. Such potential resource consumption is, for example, difficult to be predicted by analyzing static code and runtime environment, and machine learning is a good method to predict such resource consumption.

[0069] In some implementations, a machine learning-based estimation model can be established, which may, for example, be established based on the following manner. First, part of the features and labels extracted in the previous program analysis process can be used as training data for the machine learning model. The extracted features can include resource consumption-related features, computation graph-related features, and execution environment-related features, and the like. The selection of specific features can be determined according to specific implementations, and the scope of the present disclosure is not limited in this regard. In some implementations, a machine learning or deep learning model can be used for model training. In some implementations, the loss function L used for model training may, for example, be determined as the mean squared error (MSE) between the predicted ratio Y p and the target ratio (or any other target ratio that can optimize the prediction result), where Y e represents the predicted resource usage (such as memory, execution time, floating point operation amount, etc.), and Y r represents the actual resource usage. With the trained machine learning model (or deep learning model), the ratio of potential resource consumption to total resource consumption For example, when predicting memory consumption, the predicted memory usage can be optimized by plugging in a into the above equation (1). Similarly, the predicted memory usage for other types of resources can be optimized using the trained machine learning model, thereby improving the accuracy of the resource usage prediction.

[0070] It should be appreciated that the implementations of the present disclosure only utilize the machine learning method to optimize and calibrate the resource usage of the deep learning task predicted by the program analysis method, rather than only utilizing the machine learning method to make the prediction of the resource usage. In this way, the output prediction result 190 has interpretability, for example, can help the deep learning developer to find the bottleneck of the deep learning, thereby providing the model performance by debugging the model parameters. In addition, the output prediction result 190 can be interfaced with other deep learning system components (e.g., AutoML system) or artificial intelligence platforms to improve the production efficiency of deep learning. For example, the prediction result helps to prune the search space of the model parameters of the deep learning model in advance, to improve the model parameter search efficiency; the prediction result helps to optimize the task scheduling of the artificial intelligence platform, thereby improving the work efficiency, and the like.

[0071] ​ A flowchart illustrating a method 400 for predicting resource usage of a deep learning model according to some implementations of the present disclosure is shown. The method 400 can be implemented by the computing device 100, for example, can be implemented at the resource prediction module 122 implemented in the memory 120 of the computing device 100. At block 410, the computing device 100 obtains information related to a deep learning model, the information including first information describing the deep learning model, and second information related to a runtime environment of a task associated with the deep learning model. At block 420, the computing device 100 determines a static resource usage of the task based on the first information. At block 430, the computing device 100 determines a strategy for the task running in the runtime environment based on the first information and the second information. At block 440, the computing device 100 predicts a resource usage of the task running in the runtime environment based on the strategy and the static resource usage.

[0072] In some implementations, the first information includes configuration parameters of the deep learning model.

[0073] In some implementations, the first information includes at least one of: a model file of the deep learning model; and program code of the task.

[0074] In some implementations, the second information includes at least one of: a framework type of the deep learning model; a specification and a number of computing devices in the running environment for executing the task; and an execution strategy of the task on the computing devices.

[0075] In some implementations, the resource usage includes at least one of: a computation power consumption; a memory consumption; an I / O resource consumption; an execution time; and a power consumption.

[0076] In some implementations, the resource usage includes other resource consumptions determined based on at least one of the computation power consumption and the memory consumption.

[0077] In some implementations, determining the static resource usage includes: generating, based on the first information, a computation graph corresponding to the deep learning model, the computation graph including a plurality of nodes corresponding to a plurality of operators in the deep learning model, and edges connecting the plurality of nodes indicating dependency relationships between the plurality of operators; predicting, based on the computation graph and respective resource prediction models of the plurality of operators, respective static resource usages of the plurality of operators; and determining, based on the respective static resource usages of the plurality of operators, the static resource usage of the task.

[0078] In some implementations, the strategy includes at least one of: a resource allocation strategy of the deep learning model; and an execution strategy of the task in the running environment.

[0079] In some implementations, predicting the resource usage of the task when running in the running environment includes adjusting the static resource usage based on at least one of the resource allocation strategy and the execution strategy to obtain the resource usage of the task when running in the running environment.

[0080] In some implementations, the method 400 further includes: generating, with the trained machine learning model, parameters for optimizing the predicted resource usage; and optimizing, based on the parameters, the predicted resource usage.

[0081] Based on the above description, it can be seen that the scheme for predicting resource usage of a deep learning model implemented according to the present disclosure can accurately predict the usage of various resources of a deep learning model under a specific runtime strategy, such as computing power consumption, memory consumption, and execution time, etc. In addition, the scheme can support various types of input and has a scalable architecture, facilitating support for various different types of deep learning frameworks. The prediction results provided by the scheme have interpretability and can be interfaced with other deep learning system components (e.g., AutoML systems) or artificial intelligence platforms to improve the production efficiency of deep learning. For example, the prediction results can help deep learning developers find bottlenecks of a model, thereby providing model performance by debugging model parameters; the prediction results help other deep learning system components (e.g., AutoML systems) to prune the search space of model parameters of a deep learning model in advance, thereby improving the efficiency of model parameter search; the prediction results help to optimize the task execution strategy of an artificial intelligence platform, thereby improving resource utilization, etc.

[0082] Some example implementations of the present disclosure are listed below.

[0083] In an aspect, the present disclosure provides a computer-implemented method. The method comprises: obtaining information related to a deep learning model, the information comprising first information for describing the deep learning model, and second information related to a runtime environment of a task associated with the deep learning model; determining a static resource usage of the task based on the first information; determining a strategy of the task when running in the runtime environment based on the first information and the second information; and predicting a resource usage of the task when running in the runtime environment based on the strategy and the static resource usage.

[0084] In some implementations, the first information comprises configuration parameters of the deep learning model.

[0085] In some implementations, the first information comprises at least one of: a model file of the deep learning model; and program code of the task.

[0086] In some implementations, the second information comprises at least one of: a framework type of the deep learning model; specifications and number of computing devices in the runtime environment for executing the task; and an execution strategy of the task on the computing devices.

[0087] In some implementations, the resource usage comprises at least one of: computing power consumption; memory consumption; I / O resource consumption; execution time; and power consumption.

[0088] In some implementations, the resource usage includes other resource consumptions determined based on at least one of the compute consumption and the memory consumption.

[0089] In some implementations, determining the static resource usage includes: based on the first information, generating a computation graph corresponding to the deep learning model, the computation graph including a plurality of nodes corresponding to a plurality of operators in the deep learning model, and edges connecting the plurality of nodes indicating dependency relationships between the plurality of operators; based on the computation graph and respective resource prediction models of the plurality of operators, predicting respective static resource usages of the plurality of operators; and based on the respective static resource usages of the plurality of operators, determining the static resource usage of the task.

[0090] In some implementations, the policy includes at least one of: a resource allocation policy of the deep learning model; and an execution policy of the task in the running environment.

[0091] In some implementations, predicting the resource usage of the task when running in the running environment includes adjusting the static resource usage based on at least one of the resource allocation policy and the execution policy to obtain the resource usage of the task when running in the running environment.

[0092] In some implementations, the method further includes: generating, with a trained machine learning model, parameters for optimizing the predicted resource usage; and optimizing the predicted resource usage based on the parameters.

[0093] In another aspect, the present disclosure provides an electronic device. The electronic device includes: a processing unit; and a memory coupled to the processing unit and containing instructions stored thereon that, when executed by the processing unit, cause the device to perform acts including: obtaining information related to a deep learning model, the information including first information describing the deep learning model and second information related to a running environment of a task associated with the deep learning model; determining, based on the first information, a static resource usage of the task; determining, based on the first information and the second information, a policy of the task when running in the running environment; and predicting, based on the policy and the static resource usage, a resource usage of the task when running in the running environment.

[0094] In some implementations, the first information includes configuration parameters of the deep learning model.

[0095] In some implementations, the first information includes at least one of: a model file of the deep learning model; and program code of the task.

[0096] In some implementations, the second information includes at least one of: a framework type of the deep learning model; a specification and a number of a computing device in the running environment for executing the task; and an execution strategy of the task on the computing device.

[0097] In some implementations, the resource usage includes at least one of: a computing power consumption; a memory consumption; an I / O resource consumption; an execution time; and a power consumption.

[0098] In some implementations, the resource usage includes other resource consumption determined based on at least one of the computing power consumption and the memory consumption.

[0099] In some implementations, determining the static resource usage includes: generating, based on the first information, a computation graph corresponding to the deep learning model, the computation graph including a plurality of nodes corresponding to a plurality of operators in the deep learning model, and edges connecting the plurality of nodes indicating dependency relationships between the plurality of operators; predicting, based on the computation graph and respective resource prediction models of the plurality of operators, respective static resource usages of the plurality of operators; and determining, based on the respective static resource usages of the plurality of operators, the static resource usage of the task.

[0100] In some implementations, the strategy includes at least one of: a resource allocation strategy of the deep learning model; and an execution strategy of the task in the running environment.

[0101] In some implementations, predicting the resource usage of the task when running in the running environment includes adjusting the static resource usage based on at least one of the resource allocation strategy and the execution strategy to obtain the resource usage of the task when running in the running environment.

[0102] In some implementations, the actions further include: generating, with a trained machine learning model, parameters for optimizing the predicted resource usage; and optimizing, based on the parameters, the predicted resource usage.

[0103] In yet another aspect, the present disclosure provides a computer program product tangibly stored in a non-transitory computer storage medium and comprising machine executable instructions that, when executed by a device, cause the device to perform the method of the above aspect.

[0104] In yet another aspect, the present disclosure provides a computer readable medium having stored thereon machine executable instructions, the machine executable instructions, when executed by a device, cause the device to perform the method of the above aspect.

[0105] The functionality described herein above can be performed, at least in part, by one or more hardware logic components. For example, and without limitation, illustrative types of hardware logic components that can be used include Field-programmable Gate Arrays (FPGAs), Application-specific Integrated Circuits (ASICs), Application-specific Standard Products (ASSPs), System-on-a-chip systems (SOCs), Complex Programmable Logic Devices (CPLDs), etc.

[0106] Program code for carrying out methods of the present disclosure can be written in any combination of one or more programming languages. The program code can be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the program code, when executed by the processor or controller, causes the machine to perform the functions / operations specified in the flowcharts and / or block diagrams. The program code can execute entirely on a machine, partly on a machine, as a stand-alone software package, partly on a machine and partly on a remote machine or entirely on a remote machine or server.

[0107] In the context of the present disclosure, a machine-readable medium can be a tangible medium that can contain or store program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable storage medium can include, without limitation, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of the machine-readable storage medium will include one or more lines of electrical connections, portable computer disks, hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or Flash memory), optical fibers, portable compact disc read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination of the foregoing.

[0108] Moreover, while operations are depicted in a particular order, this should not be understood as requiring such an order nor that all illustrated operations be performed to achieve desirable results. In certain circumstances, multitasking and parallel processing can be advantageous. Likewise, while several specific implementation details are contained in the above discussion, these should not be construed as limitations on the scope of the disclosure, but rather as descriptions of particular implementations. Certain features that are described in the context of separate implementations can also combine in other implementations. Conversely, various features that are described in the context of a single implementation can also be implemented separately or in any suitable subcombination.

[0109] Although the subject matter has been described in language specific to structural features and / or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are disclosed as example forms of implementing the claims.

Claims

1. A computer-implemented method comprising: Acquire information related to a deep learning model, the information including first information for describing the deep learning model and second information related to an operating environment of a task associated with the deep learning model; Determining a static resource usage of the task based on the first information, wherein determining the static resource usage includes: Based on the first information, generating a computation graph corresponding to the deep learning model, the computation graph including a plurality of nodes, the plurality of nodes corresponding to a plurality of operators in the deep learning model, and edges connecting the plurality of nodes indicating dependencies between the plurality of operators; Predicting corresponding static resource usage of the multiple operators based on the computation graph and corresponding resource prediction models of the multiple operators; and Determining the static resource usage of the task based on corresponding static resource usage of the multiple operators; Determining a strategy for the task when it is executed in the execution environment based on the first information and the second information; and The resource usage of the task when running in the running environment is predicted based on the policy and the static resource usage.

2. The method according to claim 1, wherein the first information includes configuration parameters of the deep learning model.

3. The method according to claim 1, wherein the first information includes at least one of the following: a model file of the deep learning model; and a program code of the task.

4. The method according to claim 1, wherein the second information includes at least one of the following: The framework type of the deep learning model; Specifications and number of computing devices in the execution environment used to perform the tasks; and An execution policy for the task on the computing device.

5. The method according to claim 1, wherein the resource usage includes at least one of the following: computing power consumption; memory consumption; I / O resource consumption; execution time; and power consumption. 6 . The method according to claim 5 , wherein the resource usage includes other resource consumption determined based on at least one of the computing power consumption and the memory consumption.

7. The method of claim 1 , wherein the strategy comprises at least one of the following: Resource allocation strategy for the deep learning model; and The execution strategy of the task in the execution environment.

8. The method according to claim 7, wherein predicting the resource usage of the task when it is executed in the execution environment comprises: The static resource usage is adjusted based on at least one of the resource allocation policy and the execution policy to obtain the resource usage when the task is running in the running environment.

9. The method according to claim 1, further comprising: generating, using the trained machine learning model, parameters for optimizing the predicted resource usage; as well as Based on the parameters, the predicted resource usage is optimized.

10. An electronic device comprising: processing unit; as well as a memory coupled to the processing unit and containing instructions stored thereon, the instructions, when executed by the processing unit, causing the apparatus to perform actions, the actions comprising: Acquire information related to a deep learning model, the information including first information for describing the deep learning model and second information related to an operating environment of a task associated with the deep learning model; Determining a static resource usage of the task based on the first information, wherein determining the static resource usage includes: Based on the first information, generating a computation graph corresponding to the deep learning model, the computation graph including a plurality of nodes, the plurality of nodes corresponding to a plurality of operators in the deep learning model, and edges connecting the plurality of nodes indicating dependencies between the plurality of operators; Predicting corresponding static resource usage of the multiple operators based on the computation graph and corresponding resource prediction models of the multiple operators; and Determining the static resource usage of the task based on corresponding static resource usage of the multiple operators; Determining a strategy for the task when it is executed in the execution environment based on the first information and the second information; and The resource usage of the task when running in the running environment is predicted based on the policy and the static resource usage. The apparatus according to claim 10 , wherein the first information comprises configuration parameters of the deep learning model.

12. The device according to claim 10, wherein the first information includes at least one of the following: a model file of the deep learning model; and a program code of the task.

13. The apparatus according to claim 10, wherein the second information comprises at least one of the following: The framework type of the deep learning model; Specifications and number of computing devices in the execution environment used to perform the tasks; and An execution policy for the task on the computing device.

14. The device of claim 10, wherein the resource usage comprises at least one of: computing power consumption; memory consumption; I / O resource consumption; execution time; and power consumption. 15 . The device according to claim 14 , wherein the resource usage comprises other resource consumption determined based on at least one of the computing power consumption and the memory consumption.

16. The apparatus of claim 10, wherein the policy comprises at least one of: Resource allocation strategy for the deep learning model; and The execution strategy of the task in the execution environment.

17. The device according to claim 16, wherein predicting the resource usage of the task when it is executed in the execution environment comprises: The static resource usage is adjusted based on at least one of the resource allocation policy and the execution policy to obtain the resource usage when the task is running in the running environment.

18. A computer program product tangibly stored in a non-transitory computer storage medium and comprising machine-executable instructions that, when executed by a device, cause the device to perform actions comprising: Acquire information related to a deep learning model, the information including first information for describing the deep learning model and second information related to an operating environment of a task associated with the deep learning model; Determining a static resource usage of the task based on the first information, wherein determining the static resource usage includes: Based on the first information, generating a computation graph corresponding to the deep learning model, the computation graph including a plurality of nodes, the plurality of nodes corresponding to a plurality of operators in the deep learning model, and edges connecting the plurality of nodes indicating dependencies between the plurality of operators; Predicting corresponding static resource usage of the multiple operators based on the computation graph and corresponding resource prediction models of the multiple operators; and Determining the static resource usage of the task based on corresponding static resource usage of the multiple operators; Determining a strategy for the task when it is executed in the execution environment based on the first information and the second information; and The resource usage of the task when running in the running environment is predicted based on the policy and the static resource usage.

Images