Dynamic Task Allocation of Neural Networks

Abstract

The present disclosure relates to dynamic task allocation of neural networks. The subject technology determines an operation to be performed at a node of a neural network model. The subject technology assigns an annotation to indicate whether the operation is better performed on a CPU or a GPU based at least in part on the hardware capabilities of a target platform. The subject technology determines whether the neural network model includes a second layer. In response to determining that the neural network model includes a second layer, for each node of the second layer of the neural network model, the subject technology determines a second operation to be performed at the node. In addition, the subject technology assigns a second annotation to indicate whether the second operation is better performed on a CPU or a GPU based at least in part on the hardware capabilities of the target platform.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application is a divisional application of the invention patent application with international application number PCT / US2018 / 029201, international application date April 24, 2018, date of entry into the Chinese national phase on November 27, 2019, Chinese national application number 201880035228.0, and invention name “Dynamic Task Allocation of Neural Networks”.

[0003] This application claims the benefit of U.S. Provisional Patent Application Serial No. 62 / 514,779, filed on June 3, 2017, and entitled “DYNAMIC TASK ALLOCATION FOR NEURALNETWORKS,” which is hereby incorporated by reference in its entirety and made a part of this U.S. Utility Patent Application for all purposes. Technical Field

[0004] The present invention generally relates to task allocation, including dynamic task allocation for neural networks. Background Art

[0005] Software engineers and scientists have been using computer hardware for machine learning to make improvements in different industry applications, including image classification, video analysis, speech recognition, and natural language processing. Notably, deep learning neural networks are being used more frequently to create systems that can perform different computing tasks based on large amounts of data. Summary of the invention

[0006] The present disclosure provides a system.

[0007] According to one aspect, a system is provided, comprising: a processor; a graphics processing unit; a memory device containing instructions, which, when executed by the processor, cause the processor to: at runtime, for each node of a first layer of a neural network model: determine an operation performed at the node; and determine whether to override an annotation of the operation performed at the node, the annotation having been assigned based at least in part on a cost of transferring the operation between the processor and the graphics processing unit.

[0008] The present invention can achieve beneficial technical effects. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Certain features of the subject technology are set forth in the appended claims. However, for purposes of explanation, several embodiments of the subject technology are set forth in the following drawings.

[0010] Figure 1An exemplary network environment for performing dynamic task allocation of a neural network according to one or more implementations is shown.

[0011] Figure 2 An exemplary software architecture for performing dynamic task allocation of a neural network in accordance with one or more implementations is shown.

[0012] Figure 3A-3C Exemplary neural network models in different examples of annotations according to one or more specific implementations are shown.

[0013] Figure 4 A flowchart of an exemplary process for annotating a neural network model according to one or more implementations is shown.

[0014] Figure 5 An exemplary process for executing a neural network model and determining whether to override a particular annotation of a node according to one or more specific implementations is shown.

[0015] Figure 6 An electronic system is shown that can be used to implement one or more implementations of the subject technology. DETAILED DESCRIPTION

[0016] The specific embodiments shown below are intended to be descriptions of various configurations of the subject technology and are not intended to represent the only configuration that the subject technology can be practiced. The accompanying drawings are incorporated herein and constitute a part of the specific embodiments. The specific embodiments include specific details intended to provide a thorough understanding of the subject technology. However, the subject technology is not limited to the specific details set forth herein, but can be practiced using one or more other specific implementations. In one or more specific implementations, structures and components are shown in block diagram form to avoid blurring the concept of the subject technology.

[0017] Machine learning has been around for many years, but has seen a huge surge in popularity in recent years due to the availability of large amounts of training data and advances in more powerful and efficient computing hardware. A common approach is to utilize graphics processing units (GPUs) to train deep neural networks, and also to post-train and execute deep neural networks on new input data. However, when deploying a given deep neural network for execution on a target platform, it may be more efficient to run certain operations on the GPU (if available) or, in some cases, on a node-by-node basis on the CPU (if available), depending on the available hardware. Existing solutions for executing deep neural networks on a given platform lack the ability to make such determinations prior to execution, and also lack the ability to dynamically allocate execution of operations on the CPU or GPU during runtime based on available resources and / or other factors at runtime.

[0018] Particular implementations of the subject technology minimize the total runtime associated with a neural network model across any number of different device types by applying, for each layer of the neural network model, annotations at each node of the layer to enable dynamic allocation (e.g., at runtime) of whether to execute node operations on a CPU or a GPU. The annotations provide an indication of whether a GPU or a CPU is best suited to execute the node operations based on, for example, the type of operation being executed. A device executing the neural network model can use the annotations to determine whether to use a GPU or a CPU for each node of each layer (e.g., based at least on the device's hardware capabilities and / or current resource utilization). Annotating the nodes of the neural network model in this manner enables the annotated neural network model to be compiled once and used efficiently across any number of different types of devices, including future devices with different hardware capabilities.

[0019] Figure 1 An exemplary network environment 100 for performing dynamic task allocation of a neural network according to one or more implementations is shown. However, not all of the depicted components may be used in all implementations, and one or more implementations may include additional or different components than those shown in the figures. Changes in the arrangement and types of these components may be made without departing from the spirit or scope of the claims set forth herein. Additional components, different components, or fewer components may be provided.

[0020] The network environment 100 includes an electronic device 110, an electronic device 115, and a server 120. The network 106 may communicatively couple (directly or indirectly) the electronic device 110 and / or the server 120, the electronic device 115 and / or the server 120, and / or the electronic device 110 and / or the electronic device 115. In one or more specific implementations, the network 106 may be an interconnected network that may include the Internet or devices communicatively coupled to the Internet. For purposes of explanation, the network environment 106 is described in detail in detail in the following. Figure 1 1. Network environment 100 is shown as including electronic device 110, electronic device 115, and server 120; however, network environment 100 may include any number of electronic devices and any number of servers.

[0021] The electronic device 110 may be, for example, a desktop computer; a portable computing device such as a laptop computer, a smart phone, a peripheral device (e.g., a digital camera, a headset), a tablet device, a wearable device (e.g., a watch, a wristband, etc.); or any other suitable device including, for example, one or more wireless interfaces, such as a WLAN radio device, a cellular radio device, a Bluetooth radio device, a Zigbee radio device, a near field communication (NFC) radio device, and / or other radio devices. Figure 1In the embodiment, the electronic device 110 is depicted as a desktop computer by way of example. The electronic device 110 may be and / or may include the following with respect to Figure 6 All or part of the electronic system.

[0022] In one or more specific implementations, electronic device 110 may provide a system for dynamically allocating tasks performed by a neural network model as described herein. Specifically, the subject system may include a neural network annotator component that works with a neural network compiler so that during compilation of a neural network model, nodes of the neural network model are annotated to indicate a specific processor (e.g., a CPU or GPU) for completing an operation associated with a given node in a minimum amount of total runtime. In an example, using the compiled code, the subject system may create a software package for deployment on a target device under the guidance of server 120. When executing the compiled neural network model, the target device may utilize the annotation as an indication of performing a given operation on a CPU or GPU (e.g., completing the operation with a minimum amount of total runtime).

[0023] The electronic device 115 may include a touch screen and may be, for example, a portable computing device, such as a laptop computer including a touch screen, a smart phone including a touch screen, a peripheral device including a touch screen (e.g., a digital camera, headphones), a tablet device including a touch screen, a wearable device including a touch screen (e.g., a watch, a wristband, etc.), any other suitable device including, for example, a touch screen, or any electronic device with a touchpad. In one or more specific implementations, the electronic device 115 may not include a touch screen, but may support touchscreen-like gestures, such as in a virtual reality environment or an augmented reality environment. In one or more specific implementations, the electronic device 115 may include a touchpad. Figure 1 In the embodiment, the electronic device 115 is depicted as a tablet device having a touch screen by way of example. In one or more specific implementations, the electronic device 115 can be and / or can include all or part of the electronic device described below in conjunction with the electronic system, which is described below in conjunction with Figure 6 described.

[0024] In one or more specific implementations, server 120 deploys the compiled code included in the software package to the target device for execution. In an example, electronic device 115 may be a target device for receiving a software package with compiled neural network code and executing the compiled code in a runtime environment of electronic device 115. Electronic device 115 (or any electronic device as a target device) includes a framework that can access annotations included in the compiled code of the neural network and then make decisions for each operation (e.g., running the operation on a CPU, GPU, dedicated processor, etc.) based on the accessed annotations. A framework may refer to a software environment that provides specific functionality as part of a larger software platform to facilitate software application development.

[0025] Figure 2 An exemplary software architecture 200 for optimizing a neural network according to one or more specific implementations is shown. For the purpose of illustration, the software architecture 200 is described as consisting of Figure 1 The software architecture 200 is provided by the electronic device 110, such as by the processor and / or memory of the electronic device 110; however, the software architecture 200 can be implemented by any other electronic device. However, not all of the components depicted can be used in all specific implementations, and one or more specific implementations may include additional or different components than those shown in the figure. Changes in the arrangement and type of these components can be made without departing from the spirit or scope of the claims listed herein. Additional components, different components, or fewer components may be provided.

[0026] As shown, the software architecture 200 includes a neural network task distributor 205, which includes a neural network annotator 210 and a neural network compiler 215. The memory 240 includes a neural network source code, which generates a neural network executable file 242 after being compiled by the neural network compiler 215, and the neural network executable file can be deployed to different target platforms for execution. Although the neural network annotator 210 is shown as separate from the neural network compiler 215 for illustrative purposes, in at least one specific implementation, the neural network annotator 210 can be a part of the neural network compiler 215, so that the neural network compiler 215 performs annotations on the neural network source code 244 during compilation in the same manner as the neural network annotator 210 described below.

[0027] During compile time, for each node in a neural network model layer, the neural network annotator 210 may determine the operation performed at the node and annotate the node to perform the operation on a CPU or GPU (or some other processor or hardware) of a given electronic device that is a target platform for executing the neural network model. In an example, the neural network annotator 210 may initially determine whether a given operation is supported only by the CPU or only by the GPU, and annotate the particular node accordingly. The neural network annotator 210 may also consider the total runtime to complete the operation (e.g., for the case where the operation can be run on the CPU or GPU) for determining the annotation for the particular node. To determine the total runtime of the operation, the neural network annotator 210 determines a sum based on the cost of transferring the operation to another processor (if any) and the cost of completing the operation on a particular processor. Specifically, the cost (e.g., in terms of time delay) of transferring the operation from the CPU to running on the GPU or vice versa is determined. The cost of an operation on the CPU and / or GPU is also determined based on the number of cycles to complete the operation, where a more expensive operation requires more cycles to complete than a less expensive operation. The neural network annotator 210 then determines the total runtime by adding the cost of the transfer operation and the number of cycles used to complete the operation.

[0028] In an example, the neural network annotator 210 may compare the total runtime values ​​for completing an operation on a CPU or GPU, and then annotate the node with the lower total runtime value. For another example, if the cost of transferring the operation to the GPU plus the cost of running the operation on the GPU results in a total runtime greater than keeping the operation on the CPU, the neural network annotator 210 does not annotate the node to complete the operation on the GPU. In one or more specific implementations, the neural network annotator 210 determines the annotation without considering the cost of transferring the operation to another processor.

[0029] In addition, the neural network annotator 210 may determine whether a given operation is optimized for a CPU or a GPU. In this example, the neural network annotator 210 will look at the architecture of the target device to determine whether the device will better run the operation on the CPU or GPU. The neural network annotator 210 may also determine the estimated computational cost of executing the operation on the CPU and / or GPU. For example, where the operation is supported by both a CPU and a GPU, the neural network annotator 210 may determine which processor completes the operation in a faster time (e.g., by looking at the architecture of the device to determine whether the device will better run the operation on the CPU or on the GPU).

[0030] The storage device 240 also includes a registry of operations 248, which includes information for each supported operation that can be run on a given platform and / or each processor (e.g., a CPU, a GPU, or a dedicated or custom processor, such as an AI optimized processor). In an example, an annotation for each node in the neural network model is assigned in part by using the registry of operations 248 and determining whether a given operation is supported by a particular CPU or GPU of a given platform. For example, the neural network annotator 210 accesses the registry of operations 248 and determines whether the operation of the node is not supported by the CPU but is supported by the GPU (or vice versa) based on the information provided in the registry of operations 248. The neural network annotator 210 can annotate the node accordingly to indicate the supported hardware (e.g., CPU or GPU) for performing the operation.

[0031] In one or more specific implementations, the registry 248 of operations may be implemented as one or more lookup tables, including: 1) one or more lookup tables of required types and parameters for each supported operation for each device; and 2) one or more lookup tables of performance measures of operations for each device or hardware processor. In an example, a lookup table is a data structure that enables quick access to values ​​that replace runtime calculations that may be required to determine such values ​​(e.g., the above-mentioned performance measures and required types and parameters). In an instance, the neural network annotator 210 may utilize the lookup table to determine whether a CPU or GPU has better performance when running a particular operation. In addition to performance measures, the neural network annotator 210 may also check the types and / or one or more parameters required for a given operation. If there is a type mismatch or missing parameters that do not allow a particular operation to be completed on the CPU or GPU, the neural network annotator 210 may annotate the particular node accordingly to use another processor.

[0032] The neural network compiler 215 takes the neural network source code 244 and compiles it into a neural network executable file 242. As described above, during the compilation process, the neural network annotator 210 assigns an annotation to each node to indicate whether the operation is best performed on a CPU or a GPU based at least in part on the hardware capabilities of the target device and considerations described elsewhere herein. In an example, the neural network source code 244 may represent a set of operations to be performed by a corresponding node of a given neural network model in code form. Therefore, the neural network annotator 210 may annotate within the neural network source code 244 instructions for executing a particular operation of a node on a particular hardware such as a CPU or GPU or a dedicated processor. Then, the neural network compiler 215 uses the annotation to compile the neural network source code 244. Therefore, in one specific implementation, the annotation is included as part of the compiled neural network model code. In at least one specific implementation, the neural network annotator 210 may also store the associated annotation data 246 separately in the storage device 240.

[0033] Figure 3A-3C Exemplary neural network models in different examples of annotations according to one or more specific implementations are shown. Figures 3A-3C By reference Figure 2 200, particularly with respect to the components of the neural network task allocator 205. Figures 3A-3C For the purpose of explanation, a neural network model may refer to a feedforward deep neural network implemented as an approximate function f. In this regard, the model is referred to as a feedforward model because the information flows through the function evaluated by the input x, flows through one or more intermediate operations used to define f, and finally reaches the output y. Feedforward deep neural networks are called networks because they can be represented by connecting different operations together. A model of a feedforward deep neural network can be represented as a graph that represents how operations are connected together from an input layer through one or more hidden layers and finally connected to an output layer. In an example, each node in such a graph represents an operation to be performed. However, it should be understood that the specific implementations described herein contemplate other types of neural networks. For example, a recursive neural network such as a long short-term memory (LSTM) neural network can be provided for annotation, or a convolutional neural network (CNN) can be provided for annotation.

[0034] like Figure 3AAs shown in , the exemplary neural network model 300 includes an input layer having an input layer node 310, a plurality of hidden layers, and an output layer having an output layer node 350, wherein the plurality of hidden layers include hidden layer nodes 315 and 320 in a first hidden layer, and hidden layer nodes 330 and 340 in a second hidden layer. In this example, each node in the neural network model 300 corresponds to an operation performed by the neural network model 300 based on received input data. Initially, input data 305 is received by the input layer node 310. The input node may perform an operation on the received input data 305 and then send the output to the hidden layer nodes 315 and 320 for further processing. Similarly, each of the hidden layer nodes 315 and 320 may perform an operation on the received data and then forward the corresponding output to the hidden layer nodes 330 and 340. Likewise, the hidden layer nodes 330 and 340 will perform an operation on the received data and forward the corresponding output to the output layer node 350. The output layer node 350 may perform an operation and then provide the result as output data 360. In this manner, data is propagated through the neural network model 300.

[0035] Although the example neural network model 300 includes a single node in the input layer and a single node in the output layer for the purpose of explanation, it should be understood that any number of nodes may be included in the input layer and / or the output layer. In addition, any number of nodes may be included in a particular hidden layer, and any number of hidden layers may be included in the neural network model 300.

[0036] The neural network annotator 210 can annotate each node in the neural network model 300. Figure 3B As shown in , the nodes of the neural network model 300 are assigned an annotation "G" for GPU or an annotation "C" for CPU. However, it should be understood that any other type of annotation may be utilized, including additional annotations for additional processors. In addition, the annotation may include additional information (e.g., metadata describing why a particular node is annotated for running on a GPU or CPU).

[0037] In one or more specific implementations, the neural network annotator 210 may annotate each node starting from the output layer and traversing toward the input layer, annotating one node at a time in this manner. To annotate the nodes, the neural network annotator 210 takes into account the hardware and architecture of the target device (e.g., electronic device 115) used to execute the compiled version of the neural network model 300. The annotation may be based in part on using a registry of operations and determining whether the operation is supported by a particular CPU or GPU of a given platform, such as Figure 2More specifically, to determine the annotation, the neural network annotator 210 determines whether the operation is supported on the CPU or GPU, determines whether the operation is more optimized to run on the CPU or GPU, determines an estimated cost of transferring (e.g., copying the operation) the operation from the CPU to the GPU or from the GPU to the CPU, and / or determines an estimated computational cost of performing the operation on the CPU and / or GPU.

[0038] As shown, the neural network annotator 210 has annotated the output layer node 350 with "G" to indicate that the operation is to be performed on the GPU of the target device. The neural network annotator 210 may then annotate each node of the hidden layer including the hidden layer nodes 330 and 340. In this example, the neural network annotator 210 has annotated these nodes with "C" based on a separate determination to indicate that their respective operations are to be performed on the CPU of the target device. The neural network annotator 210 then moves toward the input layer to the next hidden layer and annotates the nodes therein. In this example, the neural network annotator 210 has annotated the hidden layer nodes 315 and 320 with "G" based on a separate determination to indicate that their respective operations are to be performed on the GPU of the target device. Finally, the neural network annotator 210 annotates the input layer node 310 with "G" to indicate that its corresponding operation is to be performed on the GPU of the target device.

[0039] It should be understood that while for purposes of illustration, hidden layer nodes 315 and 320 have the same type of annotation, and hidden layer nodes 330 and 340 have the same type of annotation, each of these hidden layer nodes may have a different type of annotation.

[0040] The neural network compiler 215 uses the above-mentioned annotations to compile the source code corresponding to the neural network model 300. In this way, the annotations are included as part of the compiled neural network model code.

[0041] Although each node of neural network model 300 has been annotated, in one or more embodiments, while executing neural network model 300, a target device may overwrite annotations on specific nodes during runtime to indicate that operations are to be performed on different hardware. Figure 3C shows where different nodes have been covered during runtime from Figure 3B 300 of its original annotated neural network models. Figure 3C The discussion will be with reference to electronic device 115, which is the target device running the compiled executable version of neural network model 300.

[0042] In one or more specific implementations, the electronic device 115 may include a framework that can access the annotations provided for each operation in the nodes of the neural network model 300. The framework on the electronic device 115 loads the compiled annotated neural network model 300 and accesses each node of the annotated neural network model 300. The framework of the electronic device 115 refers to a system interface that provides a software architecture and an abstraction layer to facilitate communication with the underlying hardware of the device. The framework may include a dynamic shared library and the resources required to support the library (such as header files, images, and auxiliary applications).

[0043] During runtime, the framework of the electronic device 115 starts at the output layer of the neural network model 300 and traverses backward to the input layer, analyzing the annotations at each node and determining whether to retain the annotations or overwrite them based at least on the current utilization of the CPU or GPU and / or the registry of operations.

[0044] exist Figure 3C , starting from the output layer of the neural network model 300, the framework of the electronic device 115 determines that the output layer node 350 includes an annotation of "G", which indicates that the operation is to be performed on the GPU of the electronic device 115. During runtime, based on the current computing resources of the electronic device 115, the framework of the electronic device 115 determines that the operation in the output layer node 350 should be run using the CPU instead, and thus overwrite the annotation. In an example, overwriting the annotation may include replacing the existing annotation with a different annotation, or creating a temporary copy of a set of annotations corresponding to one or more new annotations. The framework of the electronic device 115 continues to the next layer, and based on a separate determination, determines that the hidden layer nodes 330 and 340 should be run using the GPU instead and overwrite the previous annotations. The framework of the electronic device 115 continues to the next layer, and based on a separate determination, determines that the hidden layer nodes 315 and 320 should be run using the CPU instead and overwrite the corresponding annotations. The framework of the electronic device 115 accesses the input layer node 310 and determines that the annotation indicates that the operation is to be run on the GPU, and determines that the annotation will be adhered and not overwritten.

[0045] It should be understood that Figure 3C For purposes of illustration of the example, hidden layer nodes 315 and 320 have the same type of annotation, and hidden layer nodes 330 and 340 have the same type of annotation, each of these hidden layer nodes may have a different type of annotation.

[0046] Overlaying the corresponding annotations as described above can be determined using different methods. The registry of operations can be used to determine whether the current hardware of the electronic device supports the operation at a given node. For example, the annotation may call for running the operation on the CPU, but the new GPU in the current hardware may support the operation and run the operation in a manner that is less expensive than the CPU (for example, another lookup table may be used in this example to compare estimated performance data). Therefore, if a node is annotated to run on the CPU, the node can be instructed to run on the new GPU instead.

[0047] When an annotation calls for a transfer to another processor, the framework of the electronic device 115 determines the cost of transferring from the CPU to the GPU, or vice versa, and if the total cost based on the transfer and running on the other processor is greater than remaining operating on the current processor, the framework of the electronic device 115 may overwrite the annotation of the node.

[0048] The framework of the electronic device 115 may also determine the utilization of the CPU and GPU at runtime to determine whether to follow the annotation. The framework of the electronic device 115 also considers the quality of service (QoS) associated with the rest of the device, so that if the annotation calls for transfer to another processor, the annotation may be overridden when the utilization of the other processor is higher or above a predefined utilization threshold.

[0049] In addition, the framework of the electronic device 115 may also determine the priority of the application that will run the neural network model. For example, a photo application launched by a user may be prioritized over other background applications (e.g., daemons) so that when the photo application requests the same resources (e.g., CPU or GPU), these background applications will lose resources. In the example, if the background applications are currently utilizing more CPU and the photo application has a higher priority than those applications, the annotation may be overwritten from the GPU name to the CPU name, so that it can be ensured that the photo application utilizes more resources from the CPU than the background applications.

[0050] Additionally, the framework of the electronic device 115 may determine the priority of each processor (eg, a GPU may have a higher priority than a CPU and / or another processor) and determine to override the annotation if another processor has a higher priority.

[0051] Additionally, the framework of the electronic device 115 may review the current power consumption and remaining battery power of the device and determine an override annotation based on whether running an operation on a particular processor will severely impact power usage or remaining battery power.

[0052] The framework of the electronic device 115 may also utilize predictive information (e.g., from a service running on the electronic device 115 that provides historical usage information of an application based on previous user activity) to predict battery life, availability when idle, power capacity, etc., in order to determine whether to override annotations when system performance will be severely impacted based on predicted behavior of use of the application.

[0053] The above examples may be used in conjunction with each other to determine whether to overwrite a given annotation.While several examples are discussed above, it should be appreciated that other information and heuristics may be considered when determining whether to overwrite a given annotation at a node.

[0054] Figure 4 A flowchart of an exemplary process 400 for annotating a neural network model according to one or more specific implementations is shown. Figure 2 The components of the software architecture of the process 400 are described, and the process can be performed by Figure 1 The process 400 is performed by one or more processors of the electronic device 110. However, the process 400 is not limited to the electronic device 115, and one or more blocks (or operations) of the process 400 may be performed by one or more other components of other suitable devices. Further for the purpose of explanation, the blocks of the process 400 are described herein as occurring sequentially or linearly. However, multiple blocks of the process 400 may occur in parallel. In addition, the blocks of the process 400 do not have to be performed in the order shown, and / or one or more blocks of the process 400 do not have to be performed and / or may be replaced by other operations.

[0055] The neural network annotator 210 selects a layer of the neural network model (402). In an example, the selected initial layer is the output layer of the neural network model. The neural network annotator 210 selects nodes of a layer of the neural network model (404).

[0056] The neural network annotator 210 determines the operations corresponding to the nodes of the layer (406) of the neural network model. The neural network annotator 210 annotates the nodes of the layer (408) of the neural network model. The annotation can be performed by using a registry of operations and determining whether the operation is supported by a particular CPU or GPU of a given platform, such as Figure 2 Specifically, to determine the annotation, the neural network annotator 210 determines whether the operation is supported on the CPU or GPU, determines whether the operation is more optimized to run on the CPU or GPU, determines an estimated cost of transferring (e.g., copying the operation) the operation from the CPU to the GPU or from the GPU to the CPU (depending on the annotations of previously annotated nodes), and / or determines an estimated computational cost of performing the operation on the CPU and / or GPU.

[0057] The neural network annotator 210 determines whether there are more nodes in the layer (410) of the neural network model. If so, the neural network annotator 210 selects another node (404) and repeats the subsequent operations. If there are no more nodes in the layer, the neural network annotator 210 determines whether there are more layers in the neural network model. If there are more layers (412), the neural network annotator 210 selects another layer (402) and repeats the subsequent operations. Alternatively, if there are no more layers (412), the process 400 can be completed.

[0058] Figure 5 An exemplary process 500 for executing a neural network model and determining whether to cover a particular annotation of a node according to one or more specific implementations is shown. Figure 2 The process 500 is described by the components of the software architecture of Figure 1 The process 500 is performed by one or more processors of the electronic device 115. However, the process 500 is not limited to the electronic device 115, and one or more blocks (or operations) of the process 500 may be performed by one or more other components of other suitable devices. Further for the purpose of explanation, the blocks of the process 500 are described herein as occurring sequentially or linearly. However, multiple blocks of the process 500 may occur in parallel. In addition, the blocks of the process 500 do not have to be performed in the order shown, and / or one or more blocks of the process 500 do not have to be performed and / or may be replaced by other operations.

[0059] The framework of the electronic device 115 selects a layer of the neural network model associated with the compiled and annotated neural network model (502). In an example, the selected initial layer is the output layer of the neural network model. The framework of the electronic device 115 selects a node of a layer (504) of the neural network model.

[0060] The framework of the electronic device 115 determines whether to overwrite the operation corresponding to the node of the layer (506) of the neural network model. The framework of the electronic device 115 overwrites (508) the annotation of the node of the layer of the neural network model. Figure 3C Different methods of overlaying the corresponding annotation are discussed. Alternatively, the framework of the electronic device 115 determines not to overlay the annotation (506).

[0061] The framework of the electronic device 115 determines whether there are more nodes in the layer (510) of the neural network model. If so, the framework of the electronic device 115 selects another node (504) and repeats the subsequent operations. If there are no more nodes in the layer, the framework of the electronic device 115 determines whether there are more layers in the neural network model. If there are more layers (512), the framework of the electronic device 115 selects another layer (502) and repeats the subsequent operations. Alternatively, if there are no more layers (512), the process 500 can be completed.

[0062] Figure 6 An electronic system 600 is shown that can be used to implement one or more implementations of the subject technology. The electronic system 600 can be Figure 1 The electronic device 110, the electronic device 115, and / or the server 120 are shown and / or may be a part thereof. The electronic system 600 may include various types of computer-readable media and interfaces for various other types of computer-readable media. The electronic system 600 includes a bus 608, one or more processing units 612, a system memory 604 (and / or cache), a ROM 610, a permanent storage device 602, an input device interface 614, an output device interface 606, and one or more network interfaces 616, or a subset and variant thereof.

[0063] The bus 608 generally represents all system buses, peripheral busses, and chipset buses that communicatively connect the many internal devices of the electronic system 600. In one or more implementations, the bus 608 communicatively connects the one or more processing units 612 with the ROM 610, the system memory 604, and the permanent storage device 602. The one or more processing units 612 retrieve instructions to be executed and data to be processed from these various memory units in order to perform the processes disclosed in the present subject matter. In different implementations, the one or more processing units 612 can be a single processor or a multi-core processor.

[0064] ROM 610 stores static data and instructions required by one or more processing units 612 and other modules of electronic system 600. On the other hand, permanent storage device 602 can be a read-write memory device. Permanent storage device 602 can be a non-volatile memory unit that stores instructions and data even when electronic system 600 is turned off. In one or more specific implementations, a mass storage device (such as a magnetic or optical disk and its corresponding disk drive) can be used as permanent storage device 602.

[0065] In one or more implementations, a removable storage device (such as a floppy disk, a flash drive and its corresponding disk drive) may be used as the permanent storage device 602. Like the permanent storage device 602, the system memory 604 may be a read-write memory device. However, unlike the permanent storage device 602, the system memory 604 may be a volatile read-write memory, such as a random access memory. The system memory 604 may store any of the instructions and data that one or more processing units 612 may need at runtime. In one or more implementations, the processes disclosed in the subject matter are stored in the system memory 604, the permanent storage device 602, and / or the ROM 610. The one or more processing units 612 retrieve instructions to be executed and data to be processed from these various memory units in order to perform the processes of one or more implementations.

[0066] The bus 608 is also connected to an input device interface 614 and an output device interface 606. The input device interface 614 enables a user to convey information and select commands to the electronic system 600. Input devices that can be used with the input device interface 614 may include, for example, an alphanumeric keyboard and a pointing device (also referred to as a "cursor control device"). The output device interface 606 may, for example, enable the display of images generated by the electronic system 600. Output devices that can be used with the output device interface 606 may include, for example, a printer and a display device, such as a liquid crystal display (LCD), a light emitting diode (LED) display, an organic light emitting diode (OLED) display, a flexible display, a flat panel display, a solid-state display, a projector, or any other device for outputting information. One or more specific implementations may include a device that acts as both an input device and an output device, such as a touch screen. In these specific implementations, the feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user may be received in any form, including acoustic, voice, or tactile input.

[0067] Finally, if Figure 6 As shown, bus 608 also couples electronic system 600 to one or more networks and / or to one or more network nodes, such as via one or more network interfaces 616. Figure 1 . In this manner, electronic system 600 may be part of a computer network, such as a LAN, a wide area network ("WAN"), or an intranet, or may be part of a network of networks, such as the Internet. Any or all components of electronic system 600 may be used with the subject disclosure.

[0068] Implementations within the scope of the present disclosure may be implemented in part or in whole using a tangible computer-readable storage medium (or multiple tangible computer-readable storage media of one or more types) programmed with one or more instructions. Tangible computer-readable storage media may also be non-transitory in nature.

[0069] Computer-readable storage media can be any storage media that can be read, written, or otherwise accessed by a general or special computing device, including any processing electronics and / or processing circuitry capable of executing instructions. For example, without limitation, computer-readable media can include any volatile semiconductor memory, such as RAM, DRAM, SRAM, T-RAM, Z-RAM, and TTRAM. Computer-readable media can also include any non-volatile semiconductor memory, such as ROM, PROM, EPROM, EEPROM, NVRAM, flash memory, nvSRAM, FeRAM, FeTRAM, MRAM, PRAM, CBRAM, SONOS, RRAM, NRAM, racetrack memory, FJG, and Millipede memory.

[0070] In addition, the computer-readable storage medium may include any non-semiconductor memory, such as optical disk storage, magnetic disk storage, magnetic tape, other magnetic storage devices, or any other medium capable of storing one or more instructions. In one or more specific implementations, the tangible computer-readable storage medium may be directly coupled to the computing device, while in other specific implementations, the tangible computer-readable storage medium may be indirectly coupled to the computing device, for example, via one or more wired connections, one or more wireless connections, or any combination thereof.

[0071] Instructions can be directly executable, or can be used to develop executable instructions. For example, instructions can be implemented as executable or non-executable machine code, or can be implemented as high-level language instructions that can be compiled to produce executable or non-executable machine code. In addition, instructions can also be implemented as data, or can include data. Computer executable instructions can also be organized in any format, including routines, subroutines, programs, data structures, objects, modules, applications, applets, functions, etc. As recognized by those skilled in the art, the details including but not limited to the number, structure, sequence and organization of instructions can be significantly different without changing the underlying logic, function, processing and output.

[0072] Although the above discussion mainly involves microprocessors or multi-core processors that execute software, one or more implementations are performed by one or more integrated circuits such as ASICs or FPGAs. In one or more implementations, such integrated circuits execute instructions stored on the circuits themselves.

[0073] Those skilled in the art will recognize that the various illustrative frames, modules, elements, parts, methods and algorithms described herein can be implemented as electronic hardware, computer software or a combination of the two. In order to illustrate this interchangeability of hardware and software, various illustrative frames, modules, elements, parts, methods and algorithms have been generally described above according to functionality. Whether such functionality is implemented as hardware or software depends on the specific application and the design constraints imposed on the entire system. The technician can implement the described functionality in different ways for each specific application. Various parts and frames can be arranged differently (e.g., arranged in different orders, or divided in different ways) without departing from the scope of the subject technology.

[0074] It should be understood that the specific order or the hierarchical structure of the frame in the process disclosed by the present invention are the illustration of exemplary methods. Based on the design preferred requirements, it should be understood that the specific order or the hierarchical structure of the frame in the process can be rearranged or all the frames shown are executed. Any frame in these frames can be executed simultaneously. In one or more specific implementations, multitasking and parallel processing may be advantageous. In addition, the division of each system component in the above-mentioned specific implementation should not be understood as requiring such division in all specific implementations, and it should be understood that program components and systems can be generally integrated together in a single software product or encapsulated in multiple software products.

[0075] As used in this specification and any claims of this patent application, the terms "base station", "receiver", "computer", "server", "processor" and "memory" all refer to electronic devices or other technical devices. These terms exclude people or groups of people. For the purpose of this specification, the term "display" or "displaying" means displaying on an electronic device.

[0076] As used herein, the phrase "at least one of" following a list of items, any of which is separated by the terms "and" or "or", modifies the list as a whole, rather than each member (i.e., each item) of the list. The phrase "at least one of" does not require selection of at least one of each item listed; rather, the phrase allows for a meaning that includes at least one of any one item and / or at least one of any combination of items and / or at least one of each item. For example, the phrases "at least one of A, B, and C" or "at least one of A, B, or C" each refer to only A, only B, or only C; any combination of A, B, and C; and / or at least one of each of A, B, and C.

[0077] The predicate words "configured to," "operable to," and "programmed to" do not imply any specific tangible or intangible modification of a subject matter but are intended to be used interchangeably. In one or more specific implementations, a processor configured to monitor and control an operation or component may also mean that the processor is programmed to monitor and control the operation or that the processor is operable to monitor and control the operation. Similarly, a processor configured to execute code may be interpreted as a processor programmed to execute code or operable to execute code.

[0078] Phrases such as aspect, this aspect, on the other hand, some aspects, one or more aspects, a specific implementation, this specific implementation, another specific implementation, some specific implementations, one or more specific implementations, an embodiment, this embodiment, another embodiment, some embodiments, one or more embodiments, configuration, this configuration, other configurations, some configurations, one or more configurations, subject technology, disclosure, the present disclosure, other variations thereof, etc. are for convenience and do not mean that the disclosure involving such one or more phrases is essential to the subject technology, nor does it mean that such disclosure applies to all configurations of the subject technology. Disclosures involving such one or more phrases may apply to all configurations or one or more configurations. Disclosures involving such one or more phrases may provide one or more examples. Phrases such as aspect or some aspects may refer to one or more aspects and vice versa, and this applies similarly to the other aforementioned phrases.

[0079] The word "exemplary" is used herein to mean "serving as an example, instance, or illustration." Any implementation described herein as "exemplary" or as an "example" is not necessarily to be construed as preferred or advantageous over other implementations. Furthermore, to the extent that the terms "including," "having," and the like are used in the specification or claims, such terms are intended to be inclusive, in a manner similar to the way the term "comprising" is interpreted when used as a transitional word in a claim.

[0080] All structural and functional equivalents to the elements of the various aspects described throughout this disclosure that are known or later become known to those of ordinary skill in the art are expressly incorporated herein by reference and are intended to be covered by the claims. In addition, nothing disclosed herein is intended to be made available to the public regardless of whether the disclosure is explicitly stated in the claims. Pursuant to 35 U.S.C. §112, paragraph 6, no claim element shall be construed unless the element is expressly stated using the phrase “means for” or, in the case of a method claim, the element is stated using the phrase “step for”.

[0081] The previous description is provided to enable those skilled in the art to practice the various aspects described herein. Various modifications of these aspects are apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects. Therefore, the claims are not intended to be limited to the aspects shown herein, but are intended to make the full scope consistent with the language claims, wherein reference to elements in singular values ​​is not intended to mean "only one", but refers to "one or more", unless specifically noted. Unless otherwise specifically stated, the term "some" refers to one or more. Male pronouns (e.g., his) include female and neutral (e.g., her and its), and vice versa. Titles and subtitles (if any) are used only for convenience and do not limit the present disclosure.

Claims

1. An electronic system comprising: processor; Graphics processing unit; a memory device containing instructions that, when executed by the processor, cause the processor to: At runtime, for each node in the first layer of the neural network model: determining an operation to be performed at the node; and Determining whether to overwrite an annotation of the operation performed at the node based at least in part on a current condition of the electronic system, the annotation having been assigned based at least in part on a cost of transmitting the operation between the processor and the graphics processing unit, or a number of cycles for completing the operation on at least one of the processor or the graphics processing unit.

2. The electronic system of claim 1 , wherein the memory device contains further instructions that, when executed by the processor, further cause the processor to: determining whether the neural network model includes a second layer, the second layer being different from the first layer; and In response to determining that the neural network model includes the second layer, for each node of the second layer of the neural network model: determining a second operation to be performed at the node; and A determination is made whether to overwrite an annotation of the second operation performed at the node.

3. The electronic system of claim 1 , wherein the annotation for determining whether to override the operation performed at the node further comprises: A determination is made as to whether the processor or the graphics processing unit supports the operation. 4 . The electronic system of claim 3 , wherein determining whether the processor or the graphics processing unit supports the operation is based on a register of operations. 5 . The electronic system of claim 1 , wherein the annotation used to determine whether to override the operation performed at the node is based on a cost of transferring the operation from the processor to the graphics processing unit. 6 . The electronic system of claim 1 , wherein the annotation used to determine whether to override the operation performed at the node is based on current utilization of the processor and the graphics processing unit.

7. The electronic system of claim 1, wherein the annotation used to determine whether to override the operation performed at the node is based on a priority of an application that is running the neural network model. 8 . The electronic system of claim 1 , wherein the annotation used to determine whether to override the operation performed at the node is based on a priority of the processor and a priority of the graphics processing unit.

9. The electronic system of claim 1, wherein the annotation used to determine whether to override the operation performed at the node is based on current power usage or remaining battery power of the electronic system.

10. The electronic system of claim 1, wherein the first layer comprises an output layer.

11. The electronic system of claim 1, wherein the same annotation is used for all nodes of the first layer.

12. A method for determining whether to cover an annotation, comprising: At runtime, for each node in the first layer of the neural network model on the electronic system: determining an operation to be performed at the node; as well as Determining whether to overwrite an annotation of the operation performed at the node based at least in part on a current condition of the electronic system, the annotation having been assigned based at least in part on a cost of transmitting the operation between a processor and a graphics processing unit, or a number of cycles for completing the operation on at least one of the processor or the graphics processing unit.

13. The method according to claim 12, further comprising: determining whether the neural network model includes a second layer, the second layer being different from the first layer; as well as In response to determining that the neural network model includes the second layer, for each node of the second layer of the neural network model: determining a second operation to be performed at the node; and A determination is made whether to overwrite an annotation of the second operation performed at the node.

14. The method of claim 12, wherein the annotation for determining whether to override the operation performed at the node further comprises: A determination is made as to whether the processor or the graphics processing unit supports the operation. 15 . The method of claim 14 , wherein determining whether the processor or the graphics processing unit supports the operation is based on a register of operations.

16. The method of claim 12, wherein the annotation used to determine whether to override the operation performed at the node is based on a cost of transferring the operation from the processor to the graphics processing unit.

17. The method of claim 12, wherein the annotation used to determine whether to override the operation performed at the node is based on current utilization of the processor and the graphics processing unit.

18. The method of claim 12, wherein the annotation used to determine whether to override the operation performed at the node is based on a priority of an application that is running the neural network model.

19. The method of claim 12, wherein the annotation used to determine whether to override the operation performed at the node is based on a priority of the processor and a priority of the graphics processing unit.

20. The method of claim 12, wherein the first layer comprises an output layer.

21. The method of claim 12, wherein the same annotation is used for all nodes of the first layer.

22. A non-transitory computer readable medium comprising instructions that, when executed by a computing device, cause the computing device to perform operations comprising: At runtime, for each node in the first layer of the neural network model: determining an operation to be performed at the node; and Determining whether to overwrite an annotation for the operation performed at the node based at least in part on a current state of the computing device, the annotation having been assigned based at least in part on a cost of transmitting the operation between a processor and a graphics processing unit, or a number of cycles for completing the operation on at least one of the processor or the graphics processing unit.

23. The non-transitory computer readable medium of claim 22, wherein the same annotation is used for all nodes of the first layer.

24. A method for compiling a neural network model, comprising: Get the neural network source code for the neural network model; compiling the neural network source code for execution by a target device having the first processor and the second processor at least in part by annotating nodes of the neural network model to indicate a recommended processor of the first processor or the second processor for executing the node at runtime; providing a result of said compiling as a neural network executable file including annotations of said nodes for execution at a target device, The annotation of the node includes: obtaining a previous annotation of a previous node in the neural network model, the previous annotation indicating the first processor for executing the node at runtime; and determining, based on the prior annotation, a cost of transferring the node's operations to the second processor; and The nodes are annotated based at least in part on the cost of transmission.

25. The method of claim 24, wherein the first processor comprises a central processing unit and the second processor comprises a graphics processing unit or a dedicated processor separate from the central processing unit.

26. The method according to claim 24, annotating the node comprises: determining a cost of performing the operation with the first processor; determining a cost of performing the operation with the second processor; comparing the cost of performing the operation with the first processor to a combination of the cost of performing the operation with the second processor and the cost of transferring the operation to the second processor; as well as The node is annotated based at least in part on the comparison.

27. The method of claim 26, wherein annotating the node based at least in part on the comparison comprises: determining, based on the comparison, that the cost of performing the operation with the first processor is less than a combination of the cost of performing the operation with the second processor and the cost of transmitting the operation to the second processor; as well as The node is annotated to indicate the first processor as the recommended processor among the first processor or the second processor for executing the node at runtime.

28. The method of claim 26, wherein annotating the node based at least in part on the comparison comprises: determining, based on the comparison, that the cost of performing the operation with the first processor is greater than a combination of the cost of performing the operation with the second processor and the cost of transferring the operation to the second processor; as well as The node is annotated to indicate the second processor as the recommended processor among the first processor or the second processor for executing the node at runtime.

29. The method of claim 26, wherein determining the cost of performing the operation with the first processor comprises: A number of cycles to complete the operation using the first processor is determined.

30. The method of claim 29, wherein determining the cost of performing the operation with the second processor comprises: A number of cycles to complete the operation using the second processor is determined.

31. The method of claim 24, wherein determining the cost of transferring the operation of the node to the second processor comprises: A time is determined for transferring the operation of the node to the second processor.

32. The method of claim 24, wherein the compiling further comprises: Based at least in part on the annotation of the node, annotating a subsequent node of the neural network model to indicate a recommended processor in the first processor or the second processor for executing the subsequent node at runtime.

33. A method according to claim 32, wherein the node is a node of a first layer of the neural network model, and the subsequent node is a node of a second layer of the neural network model, and the second layer of the neural network model is different from the first layer of the neural network model.

34. The method of any one of claims 24 to 33, wherein annotating the node comprises: Based on an operation registration table for the target device, it is determined whether an operation of the node is supported by the first processor, the second processor, or both the first processor and the second processor.

35. The method of claim 34, wherein annotating the node comprises: Determining, based on the registry for operations on the target device, that the operation of the node is not allowed on the second processor; as well as The node is annotated to indicate the first processor as the recommended processor among the first processor or the second processor for executing the node at runtime.

36. The method of claim 34, wherein annotating the node comprises: determining, based on the registry of operations for the target device, that the operation of the node is optimized for the second processor; as well as The node is annotated to indicate the second processor as the recommended processor among the first processor or the second processor for executing the node at runtime.

37. A method for assigning an annotation, comprising: For each node in the first layer of the neural network model: determining an operation to be performed at the node; assigning an annotation to indicate whether the operation is better performed on a CPU or a GPU based at least in part on hardware capabilities of a target platform, wherein the annotation is determined based at least in part on using a registry of operations and determining whether the operation is supported by the CPU or the GPU of the target platform or determining a cost of transferring the operation from the CPU or the GPU of the target platform; determining whether the neural network model includes a second layer, the second layer being different from the first layer; as well as In response to determining that the neural network model includes the second layer, for each node of the second layer of the neural network model: determining a second operation to be performed at the node; and A second annotation is assigned to indicate whether the second operation is better performed on the CPU or on the GPU.

38. An electronic system comprising: processor; Graphics processing unit; A memory device containing instructions which, when executed by the processor, cause the processor to perform a method according to any one of claims 24 to 37.

39. A non-transitory computer readable medium comprising instructions which, when executed by a computing device, cause the computing device to perform the method of any one of claims 24 to 37.

Images