Hardware consciousness generation for machine learning models

Abstract

This specification relates to the automatic generation of machine learning models, such as neural networks. One exemplary method includes obtaining a first machine learning model having one or more first inference operations. The exemplary method also includes identifying a number of second inference operations supported by an inference hardware architecture. The exemplary method also includes generating a second machine learning model by modifying the first machine learning model to include individual second inference operations supported by an inference hardware architecture. The exemplary method also includes selecting a final machine learning model from the second machine learning models based on one or more metrics.

Description

Background Art

[0001] Background Traditionally, machine learning models have been manually generated by experts who would define the model and then use automated techniques for model training. As machine learning models have grown, various attempts have been made to automate the process of generating learning models. Machine learning models generated using automated techniques can be very accurate, but such models may not fully utilize some of the efficiencies provided by modern computing hardware.

Summary of the Invention

Means for Solving the Problems

[0002] Summary This "Summary of the Invention" is provided to introduce, in simplified form, a selected one of the concepts further described in the "Detailed Description of the Invention" below. This "Summary of the Invention" is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter.

[0003] This specification generally relates to techniques for automatically generating machine learning models. One example includes methods or techniques that can be performed on a computing device. The method or technique can include obtaining a first machine learning model having one or more first inference operations. The method or technique can also include identifying a plurality of second inference operations supported by an inference hardware architecture. The method or technique can also include generating a second machine learning model by modifying the first machine learning model to include individual second inference operations supported by the inference hardware architecture. The method or technique can also include selecting a final machine learning model from the second machine learning model based on one or more metrics.

[0004] Another example includes a system comprising a hardware processing unit and storage resources. The storage resources may store computer-readable instructions that, when executed by the hardware processing unit, cause the hardware processing unit to search a machine learning model search space having multiple inference operations supported by an inference hardware architecture. The search may include emulation of the inference architecture hardware. The computer-readable instructions may also cause the hardware processing unit to output a final machine learning model selected from the machine learning model search space.

[0005] Another example includes a system having a hardware processing unit and storage resources. The hardware processing unit may be configured to perform multiple aided inference operations. When a computer-readable instruction is executed by the hardware processing unit, it causes the hardware processing unit to determine the device context of the computing device and to select a specific machine learning model from several machine learning models available to the computing device. The multiple machine learning models may have various aided inference operations. The computer-readable instruction may also cause the hardware processing unit to execute a specific machine learning model to perform a specific task.

[0006] The above enumeration is intended to provide a quick reference to assist the reader, but is not intended to define the scope of the concepts described herein.

[0007] Brief explanation of the drawing The "Modes for Carrying Out the Invention" will be described with reference to the accompanying drawings. In the accompanying drawings, the leftmost digit of the reference numeral identifies the drawing in which the reference numeral first appears. The use of similar reference numerals in various instances in this specification and drawings may refer to the same or identical items. [Brief explanation of the drawing]

[0008] [Figure 1] This paper presents exemplary machine learning model evolutionary exploration procedures using several implementations of this concept. [Figure 2A] This section presents examples of modifications that may be made to a machine learning model during the search process, based on several implementations of this concept. [Figure 2B] This section presents examples of modifications that may be made to a machine learning model during the search process, based on several implementations of this concept. [Figure 2C] This section presents examples of modifications that may be made to a machine learning model during the search process, based on several implementations of this concept. [Figure 2D] This section presents examples of modifications that may be made to a machine learning model during the search process, based on several implementations of this concept. [Figure 3] This document presents exemplary model generation workflows for generating machine learning models using several implementations of this concept. [Figure 4] This shows scatter plots relating to the continuous iteration of machine learning model search procedures for several implementations of this concept. [Figure 5] This shows scatter plots relating to the continuous iteration of machine learning model search procedures for several implementations of this concept. [Figure 6] This shows scatter plots relating to the continuous iteration of machine learning model search procedures for several implementations of this concept. [Figure 7] This section presents illustrative systems based on several implementations of this concept. [Figure 8] This section presents illustrative graphic user interfaces based on several implementations of this concept. [Figure 9] This flowchart illustrates several implementations of this concept, illustrating exemplary methods for hardware-aware generation in machine learning models. [Figure 10] This flowchart illustrates several implementations of this concept, demonstrating exemplary methods for dynamic runtime selection of machine learning models. [Modes for carrying out the invention]

[0009] Detailed explanation overview As noted earlier, one way to generate a machine learning model is for a human to manually define the model's structure. In this case, the model can be trained by a computer on several training datasets to obtain a trained model, which can then be validated using a validation dataset. Subsequently, modified forms of the model's structure can be manually generated (e.g., by adding or removing operations or connections between operations). The modified machine learning model can then be trained again to obtain an additional trained model, which can then be compared to each other to select the final model and corresponding structure that works well for a given task. However, this method requires expert involvement to generate the initial model structure and modified forms, and to select the final model structure.

[0010] Another approach involves automating the process of generating machine learning models using a computer. For example, the computer could generate various candidate machine learning models and then select the final model from among them. While this method can generate highly precise models for a wide range of tasks, the resulting models may not be very efficient. For instance, if the model is automatically generated prioritizing accuracy over efficiency, the resulting model may tend to run relatively slowly (e.g., with long latency) and exhibit high memory usage or high power consumption.

[0011] One way to generate efficient machine learning models is to run them on inference hardware, such as a neural processing unit (which provides dedicated hardware support for inference operations). For example, a neural processing unit may provide dedicated instructions and circuit configurations for operations such as convolution, vector operations, matrix operations, pooling operations, and activation function operations. However, conventional automation techniques for generating machine learning models tend to generate models using inference operations that are not directly supported by the available inference hardware architecture. For example, a given inference hardware architecture may provide a discrete set of convolution or matrix operations with specified input and output sizes, and machine learning models generated using conventional automation techniques may include convolution or matrix operations with various data sizes that are not directly supported by that inference hardware architecture.

[0012] This disclosed implementation provides a technique for automatically generating machine learning models in a manner that takes into account the availability of specific inference operations provided by a particular inference hardware architecture. As a result, this disclosed implementation can automatically generate models that are far more efficient than those generated by using traditional techniques that do not take into account the availability of hardware-assisted inference operations during model generation. Furthermore, models generated using the techniques described herein can still provide accuracy equivalent to that of models generated using traditional techniques.

[0013] Background of Machine Learning There are various types of machine learning frameworks that can be trained to perform a given task, such as estimating the quality of a signal or enhancing a signal. Support vector machines, decision trees, and neural networks are just a few examples of machine learning frameworks that have been used in a wide variety of applications, including image processing and natural language processing. Some machine learning frameworks, such as neural networks, use layers of operations or "nodes" connected by one or more edges.

[0014] In a neural network, nodes are connected to each other via one or more edges. A neural network may include an input layer, an output layer, and one or more intermediate layers. Individual nodes within each layer can perform special operations on their inputs, such as convolution operations, vector operations, matrix operations, pooling operations, or activation function operations. Each operation can provide an output to a subsequent layer or, in some cases, to a previous layer. The inputs to a given node can be multiplied by the corresponding weight values of the edges between the input and the node. Additionally, a node can have individual bias values that are also used to generate an output. Various training procedures can be applied to learn the edge weights and / or the bias values. Neural networks and other machine learning models can be considered to operate in two stages: training and inference. In the training stage, the model is used to make predictions when given training data, and the model parameters are updated based on whether those predictions are correct. In the inference stage, the trained model is employed to process input data (often without further modification of the model parameters) to perform a specific task. Training can include algorithms such as batch gradient descent that calculate the error gradient to update the model parameters. In contrast, inference typically does not involve such calculations. Since the inference process does not necessarily involve training calculations, it is particularly efficient in performing inference operations, and thus it is possible to build dedicated inference hardware that does not necessarily need to fully support model training.

[0015] For example, an inference hardware architecture may be implemented as a systolic array. In such an array, input data may be split and distributed to groups of parallel nodes each performing the same operation on a subset of the input data, and then the results of those processes may be passed to the next group of nodes within the array. For example, individual hardware nodes may be able to perform multiplication and accumulation with respect to a defined data size very quickly (e.g., by using a dedicated circuit configuration for performing large convolution operations or matrix multiplication operations in relatively few processing cycles, by using relatively few memory transfer operations). For example, large matrices or vectors may be loaded or output to memory in a single operation.

[0016] In contrast, performing large convolutions or matrix multiplications on a conventional CPU tends to involve executing a long stream of sequential operations. Individual instructions may retrieve parts of a matrix or vector from memory, process them individually, and then subsequent operations may combine intermediate results into a final output. As a result, complex convolution or matrix operations tend to be much less efficient on a general-purpose CPU than when implemented on dedicated inference hardware. A CPU implementation of a large convolution or matrix operation may have much longer latency, higher power consumption, and / or greater memory usage than the same operation implemented by using hardware-assisted inference operations. However, models that do not include specific inference operations (e.g., data size) supported in inference hardware may not fully utilize these efficiencies.

[0017] Neural architecture search procedure There are various techniques for exploring a machine learning model search space to identify a final model. For example, as further discussed below, the disclosed techniques can be employed to perform neural architecture search by using evolutionary techniques, reinforcement learning techniques, Bayesian optimization techniques, hill-climbing techniques, one-shot techniques, and the like. The following discussion uses an evolutionary technique as a specific example of how the disclosed concepts can be employed for the automatic generation of a machine learning model. However, as further discussed below, the disclosed concepts can be readily incorporated into other techniques for generating a machine learning model.

[0018] FIG. 1 shows an exemplary machine learning model evolutionary search procedure 100. First, a parent model 110 is modified to include a trained child model 120 and trained. As further explained below, in some cases, the parent model is modified to include inference operations supported by a target inference hardware architecture. Additionally, in some cases, the parent model is modified by removing other inference operations not supported by the target inference hardware architecture. The parent model can also be modified by adding or removing connections between individual inference operations.

[0019] Next, the trained child model 120 is pruned to obtain the pruned child model 130. For example, the trained child models may be pruned to remove individual child models that do not perform as well as the others with respect to one or more metrics. As will be discussed further below, metrics may relate to loss or accuracy, latency, power consumption, memory usage, etc. After pruning, the remaining trained child models are designated as the next-generation parent model 140. Further iterations of model search may be performed by further training and pruning the child models until a stopping condition is reached (the point at which the final model can be selected from the available child models). In some cases, model search may proceed until all unsupported inference operations are removed, and all remaining models contain only inference operations supported by the target hardware architecture. In other cases, model search may allow unsupported inference operations to persist, in which case the unsupported inference operations may be performed by using mathematical equivalence operations that can conventionally be performed on a CPU or supported by the target inference architecture.

[0020] Exemplary model modification Figures 2A, 2B, 2C, and 2D illustrate exemplary modifications that may be made to transform a parent model into a child model. Figure 2A shows a hardware-assisted inference operation 200 including three specific inference operations (convolution X, convolution Y, and convolution Z). For example, each inference operation may be a convolution operation with specific input / output tensor sizes and / or kernel sizes supported by the target inference hardware architecture. In other words, a given processing unit implementing the target inference hardware architecture has a dedicated circuit configuration for performing convolution operations with those tensor and / or kernel sizes (perhaps by using a single machine code instruction, e.g., an opcode). Each dedicated circuit configuration for performing a given convolution operation may have parallel hardware nodes that can perform part of the convolution operation on part of the input data to generate part of the output data. The output data may be synthesized and further processed by using another circuit configuration provided by the processing unit implementing the target hardware architecture.

[0021] Seed model 202 may be a model originally developed without considering the target inference architecture. For example, seed model 202 may be a model developed manually or by using automated techniques and is known to work well for a specific task (such as a specific image processing operation (e.g., background segmentation, object recognition)). Seed model 202 includes three types of convolutional operations A, B, and C that are not of the kind supported by the target hardware architecture. In other words, convolutional operations A, B, and C may have different tensor and / or kernel sizes than those available from hardware-assisted operations. Multiple iterations of the machine learning model search procedure may be performed by starting with seed model 202 as the parent model, as will be further described below. In each iteration, one or more operations or connections between operations may be added or removed until a final model is generated, which may include operations that perform similar functionality to the seed model but are supported by the target inference hardware architecture. In the first model search iteration, the convolution A operation 203 of the seed model 202 may be replaced to generate child models 204, 208, and 212. Child model 204 may be generated by replacing the convolution A operation with a convolution X operation 206, child model 208 may be generated by replacing the convolution A operation with a convolution Y operation 210, and child model 212 may be generated by replacing the convolution A operation with a convolution Z operation 214. As described above, each child model may be trained, and one or more of the child models may be selected as the next generation parent model of the model.

[0022] For illustrative purposes, we assume that child model 208 is selected as the next-generation parent model (re-designated as parent model 216 in Figure 2B). This parent model can be modified by replacing the convolution B operation 218 to generate child models 220 and 226. Child model 220 can be generated by replacing the convolution B operation with convolution X operations 222 and convolution Z operations 224. Child model 226 can be generated by replacing the convolution B operation with convolution Y operations 228 and convolution Y operations 230. As described above, each child model can be trained, and one or more of the child models can be selected as the next-generation parent model.

[0023] For illustrative purposes, we assume that child model 226 is selected as the next generation parent model (re-designated as parent model 232 in Figure 2C). This parent model can be modified by replacing the convolution C operation 234 to generate child models 236 and 242. Child model 236 can be generated by replacing the convolution C operation with the convolution X operation 238 and the ReLU operation 240. Child model 242 can be generated by replacing the convolution C operation with the convolution Z operation 244.

[0024] Now, let's assume that child model 242 is selected as the next generation parent model (re-designated as parent model 246 in Figure 2D). This parent model can be modified by replacing convolution A operation 248 to generate child models 250 and 254. Child model 250 can be generated by replacing convolution A operation with convolution Z operation 252. Child model 254 can be generated by replacing convolution A operation with convolution Y operation 256.

[0025] At this point, the stopping condition may be reached, and the final model is selected from the models generated so far. For example, child model 250 may be selected as the final model, as shown in bold text in Figure 2D. The final model may be output for execution on inference hardware to perform a specific task (which the model was trained to perform). Thus, referring back to Figure 2A, seed model 202 has been transformed into a final model that can perform the same task as the seed model, but uses inference operations supported by the target inference hardware architecture. Therefore, the final model may provide similar functionality to the seed model while leveraging the efficiency provided by the target inference hardware architecture.

[0026] Exemplary Model Generation Workflow Figure 3 shows an exemplary model generation workflow 300 that may be employed to explore a machine learning model space with computations supported by the inference hardware architecture. Hardware definition storage 302 stores one or more hardware definitions that identify several hardware-supported inference operations, such as the convolution operations X, Y, and Z shown above in Figures 2A-D. Parent model storage 304 stores parent models that may be replaced over time by new parent models, as will be further described below. In some cases, parent model storage is initialized by using one or more seed models that are selected, for example, based on their performance (e.g., accuracy) in a particular task. Subsequent generations of parent models may be used to be incorporated into parent model storage over time.

[0027] For each generation, one or more parent models 306 may be retrieved from parent model storage and input to child model generation 308. Child model generation may use inference operations available in hardware-defined storage 302 to generate child models 310. The child models may be trained in 312 using, for example, supervised learning, unsupervised learning, transfer learning, etc., to generate trained child models 314. The trained child models may be run in 316 to obtain metrics 318. For example, metrics may characterize the accuracy or loss of the trained child models, the latency of the trained child models (e.g., execution time), the power consumption or memory usage of the trained child models, etc.

[0028] Metrics can be used to evaluate trained child models in order to identify selected child models 322. For example, in some cases, child models are selected based on trade-offs between two or more metrics, such as selecting child models with relatively high accuracy and relatively low power consumption. A similar approach can also be used to select the final model 324 when a stopping condition is reached.

[0029] Child model generation 308 may involve replacing or adding operations and / or connections between operations, as shown above with respect to Figures 2A-2D. For example, child model generation may include random or deterministic methods for selecting operations to add to the parent model, operations to remove from the child model, and / or connections to add or remove between individual operations. In some implementations, child model generation is sufficiently constrained by operations supported by the target inference hardware architecture, for example, only supported operations are added to a given child model. In other implementations, the model space may be explored by considering both supported and unsupported operations, and in some cases by preferring the selection of supported operations over unsupported operations using a weighting scheme.

[0030] refinement In some cases, the techniques described above can be performed by simulating the target hardware using general-purpose hardware, without running the model on hardware that implements the target inference hardware architecture, and without emulating the target inference hardware architecture. Simulating the model may involve approximating the functionality of a given target hardware architecture without directly performing the intrinsic inference operations supported by the target hardware architecture. By using simulation, general-purpose hardware can be used to simulate and perform operations that are mathematically or logically equivalent to those supported by the target inference hardware architecture, so it is still possible to predict the accuracy of a given model.

[0031] For example, performing a convolution or matrix operation that can be executed on a general-purpose CPU using a single operation by an NPU in just a few processing cycles may require hundreds or thousands of calculation and processing cycles. However, since these operations are mathematically or logically equivalent, or at least approximately equivalent, the accuracy or loss of a given model can still be estimated on the CPU. Therefore, a general-purpose CPU can be used to transform an initial seed model with operations not supported by the target inference hardware architecture into a final model fully supported by the target inference hardware architecture. Even assuming that simulation on the CPU cannot estimate the performance of the final model in terms of latency, power consumption, or resource utilization without emulation, the final model is still likely to present a significant improvement over the seed model, given that it can leverage the efficiencies provided by the target inference hardware architecture.

[0032] In another implementation, hardware emulation of individual operations supported by the target inference hardware architecture can further guide the search. In hardware emulation, the CPU can perform operations that directly correspond to the inference operations of a given model. In other words, the CPU can iterate over the target hardware architecture by mapping each inference operation in a given model to a set of CPU instructions designated to emulate that inference operation. By using emulation, performance information about each model can be inferred. For example, the total latency, power consumption, and / or resource utilization of a given model can be estimated based on the emulation. This allows for multi-objective searches where child models can be selected as next-generation parents based not only on accuracy or loss but also on the performance of each model.

[0033] In yet another implementation, hardware emulation is adapted to expose several metrics of each individual inference operation in the model. Thus, for example, two different convolution operations may have different latency, different memory footprints, different power consumption, etc. By exposing per-operation metrics during model generation and evaluation, model exploration can be guided towards a preferred selection of operations that are not only precise but also efficient. For example, when generating child models, a weighting scheme may be employed that prefers to add efficient operations to the parent model. The weighting may be proportional to the performance of a given inference operation with respect to a given metric. Individual inference operations may be randomly selected according to the weighting scheme so that relatively more efficient inference operations are more likely to be selected, while the model space can still be adequately explored.

[0034] In addition, it should be noted that a single seed model can be transformed into multiple final models to suit various purposes in various device contexts. For example, as will be further discussed below, one final model may be generated by using a search that considers both accuracy and power consumption metrics, and another final model may be generated by using a different search that considers accuracy and memory usage metrics. The first final model may be selected and executed when the current device context indicates that available power is constrained (e.g., the device is not plugged in and / or the battery level is below a threshold), and the second final model may be executed when the current device context indicates that available memory is constrained (e.g., memory usage exceeding a specified threshold %).

[0035] In yet another implementation, models can be generated for computing systems with multiple processing units. For example, consider a scenario where a single computing device conventionally has both a CPU and an NPU. By considering the bandwidth between the CPU and the NPU, the model can be generated by considering not only which operations should be employed but also whether those operations will be executed on the CPU or the NPU. Thus, for example, the final model may have a first path for operations designated to be executed on the CPU and a second path for operations designated to be executed on the NPU.

[0036] Evaluating and designating child models as parents As previously noted, several child models are selected during evaluation 320 and added to parent model storage 304 for use as parent models in subsequent generations. One method for determining which child models to add to parent model storage involves using one or more metrics to predict which child models are likely to produce offspring that will present improvements over previously discovered models in subsequent iterations. Generally, metrics may consider factors such as the loss or precision of a given child model, the latency of a given child model, the power consumption of a given child model, and the computing resource consumption (e.g., memory consumption) of a given child model. Child models exhibiting features such as relatively low loss or high precision, short latency, low power consumption, and / or low computing resource consumption may be favored for selection as parent models in the next generation.

[0037] In this specification, one specific method for selecting child models for a parent pool is shown with respect to Figure 4. This figure shows an illustrative scatter plot 400 of various trained models. For each child model that has completed training, the cost of that child model may be calculated and plotted on the x-axis 402, where the cost may be defined based on latency, power consumption, computing resource consumption, etc. In some cases, the cost may be normalized to a number between 0 and 1, as shown in Figure 4. In addition, the loss of the child model may be calculated and plotted on axis 404. Once all models have been plotted for a given iteration, the lower convex hull 406 may be calculated from the plotted values.

[0038] The lower convex hull 406 can be used as a mechanism for determining whether a given child model is added to the parent model pool. For example, a child model of the lower convex hull can be added to the parent model pool with a probability defined by using the following specific algorithm. If m1 and m2 are two adjacent models on the lower convex hull (having c1 and c2 (c1 < c2)), the probability weighting of m2 can be set proportionally to c2 - c1. The most precise model (having no follower models on the curve) can be selected for inclusion within the parent model pool with a probability of 0.5. In FIG. 4, the most precise model is model 408 because it has the lowest loss.

[0039] Generally, the lower convex hull is a subset of the Pareto frontier, and thus another approach is to select child models on the Pareto frontier for inclusion within the parent pool. Either approach can provide good performance for selecting the child models to be added to the parent model pool. One way to view the lower convex hull and / or the Pareto frontier is as follows. A given model on the lower convex hull or the Pareto frontier cannot be improved with respect to one metric by moving to another model on the lower convex hull / Pareto frontier without degrading other metrics.

[0040] Note that the same model can have various verification errors due to the randomness in forming the probability gradients. As a result, the lower convex hull or the Pareto frontier can be relaxed by a multiplicative bandwidth. Thus, a child model whose verification error is (1 + γ) times the lower convex hull verification error at the same computational cost can be considered to be on the lower convex hull and can be selected as a parent. Some implementations can set γ = 0.025. This approach allows some child models that are close to but not exactly located on the lower convex hull to still be designated as parent models.

[0041] Other techniques may also be used to enable the selection of a child model as a parent model if it is located within a predetermined neighborhood of the lower convex hull. For example, some implementations may define a threshold distance from the lower convex hull, allowing child models within that threshold distance to be selected as parent models. This is just one of many techniques that can be used to select a subset of one or more child models as a parent model based on one or more metrics.

[0042] Figure 4 shows the trained model as a black dot. For explanatory purposes, we assume that Figure 4 represents the state of the scatter plot 400 after N iterations. One or more child models on or near the lower convex hull 406 may be selected as the parent model for subsequent iterations N+1, where additional operations are added from further child models as discussed above.

[0043] Figure 5 shows a scatter plot 400 of the subsequent state after N+1 iterations. The child models trained during N+1 iterations are shown in Figure 5 by using squares. A new lower convex hull 502 can be computed. The previous lower convex hull 406 is shown as a dotted line to indicate the downward movement of the lower convex hull in N+1 iterations.

[0044] Again, one or more child models within or near the lower convex hull 502 may be selected for subsequent iterations N+2. The child models trained during iterations N+2 are shown as triangles in Figure 6. A new lower convex hull 602 may be computed, and the previous lower convex hulls 406 and 502 are shown by dotted lines to indicate their positions relative to the lower convex hull 602.

[0045] One way to examine the methods shown in Figures 4-6 is through a greedy approach to find a cost-effective predictor. Note that this is a multi-objective approach that considers both loss / accuracy and model performance in terms of latency, power consumption, or resource utilization. Alternative implementations may involve using different and / or additional metrics, such as multidimensional plots of three or more metrics, or objective functions defined with respect to one or more metrics.

[0046] The methods described above typically grow networks by using randomization techniques. However, instead of purely random techniques, which may be computationally unfeasible, this method is guided by a preference for selecting a known good model as a basis for further modifications. As previously noted, training a model from scratch can be highly computationally intensive. For example, a training dataset may contain millions of training data items, and a given model may need to be trained over several training epochs before convergence. A training epoch may involve one forward operation and one backward operation through the entire model for each data item in the training dataset.

[0047] The method described above offers several advantages over conventional methods for automated model generation. Note that not all child models will necessarily be used as parent models for subsequent iterations. Rather, by using a subset of child models that arise along the lower convex hull as new parent models, the disclosed implementation starts each new iteration with a child model structure that inherits the parent model structure of a known good model. This allows subsequent iterations to proceed without training models that occupy a significant portion of the search space away from the lower convex hull, and thus can save a tremendous amount of training time. In addition, by using cost as well as accuracy as a criterion for selecting which child model should be used as the new parent model, the disclosed implementation avoids generating new models that tend to have long latency or consume significant power or computing resources.

[0048] It should be noted that conventional techniques for the automatic generation of machine learning models tend to produce models that do not fully utilize the latest inference hardware. In contrast, the techniques described herein not only consider the availability of specific inference operations when generating models, but also consider how those specific inference operations tend to affect the characteristics of the resulting model (such as latency, power consumption, and resource utilization).

[0049] Exemplary system This implementation can be carried out on various devices and in various scenarios. Figure 7 shows an exemplary system 700 in which this implementation, as further discussed below, may be employed.

[0050] As shown in Figure 7, the system 700 includes client devices 710, servers 720, servers 730, and client devices 740 connected by one or more networks 750. It should be noted that client devices can be embodied as both mobile devices such as smartphones or tablets and fixed devices such as desktops and server devices. Similarly, servers can be implemented using various types of computing devices. In some cases, any of the devices (especially servers) shown in Figure 7 can be implemented in data centers, server farms, etc.

[0051] Some components of the device shown in Figure 7 may be referred to herein by reference numbers in parentheses. For the purposes of the following description, the interjection (1) indicates the occurrence of a given component on client device 710, (2) indicates the occurrence of a given component on server 720, (3) indicates the occurrence on server 730, and (4) indicates the occurrence on client device 740. Unless a specific instance of a given component is identified, this specification generally refers to a component without an interjection.

[0052] Generally, devices 710, 720, 730, and / or 740 may have processing resources 701 and storage resources 702, respectively, which are discussed in more detail below. These devices may also have various modules that function by using the processing and storage resources to perform the techniques discussed herein. Storage resources may include both persistent storage resources, such as magnetic drives or solid drives, and volatile storage, such as one or more random access memory devices. In some cases, modules are provided as executable instructions stored on persistent storage devices, loaded into random access memory devices, and read from random access memory by processing resources for execution.

[0053] The client device 710 may include a configuration module 711 that can interact with the model generation module 721 on the server 720. Generally speaking, the configuration module may provide several configuration parameters to the model generation module. The model generation module uses these configuration parameters to perform model generation as discussed herein. In particular, the model generation module may perform a model generation workflow 300 based on the configuration parameters.

[0054] The model generation module 721 can output the final model to the server 730 and / or client device 740. The server 730 and client device 740 may have instances of the model selection module 703 and the model execution module 704, respectively. For example, the model selection module may select from multiple models generated by the server 720 according to context such as resource constraints. The model execution module 704 may execute the selected model.

[0055] Exemplary Graphic Interface As noted above, the configuration module 711 on the client device 710 may provide initial configuration parameters to the model generation module 721. The model generation module 721 may perform the model generation workflow 300 according to the configuration parameters provided by the configuration module. Figure 8 shows an exemplary configuration graphic user interface ("GUI") 800 that may be presented on the client device 710 for the user to define these configuration parameters.

[0056] The seed model element 801 allows the user to specify what type of seed model or model should be used to begin exploring the machine learning model space. Figure 8 shows a user who has selected the default parent model. For example, the model generation module 721 may provide a default neural network structure for use as a general seed model. Another option may include randomly generated models, in which case the module generation module selects a random model structure for use as a seed model. Another option is for the user to navigate to an existing model known to provide relatively good performance for a particular task. In this case, the configuration module 711 may upload the specified model to the model generation module for use as a seed model.

[0057] The computation element 802 allows the user to specify which types of operations are considered by the model generation module 721. For example, the model generation module may offer various options for groups of operations supported by various inference hardware architectures. In Figure 8, the user selected NPU model D42, which may have dedicated circuit configurations for performing specific operations on a particular inference hardware architecture. For example, the NPU may have circuit configurations for performing convolution operations with specific tensor and kernel sizes in a single operation, circuit configurations for performing vector or matrix operations with specific input / output sizes in a single operation, circuit configurations for performing specific pooling operations or activation function operations, etc. The budget input element 803 allows the user to specify a computer budget for model generation. For example, the user may specify a budget of 10,000 GPU days, and the model generation module 721 may use this budget as a stop condition. Alternative implementations may use other metrics, such as the number of operations, the number of virtual machines, or the amount of time, as the computer budget.

[0058] Metric 1 element 804 allows the user to define a first metric for evaluating the model, and Metric 2 element 805 allows the user to define a second metric. In Figure 8, these metrics are shown as power consumption and loss, respectively. However, the user may want to define other metrics such as latency, power consumption, or resource utilization.

[0059] It should be noted that the configuration parameters shown in Figure 8 are illustrative only, and therefore various other implementation forms are intended. For example, in some cases, the user may define connectivity parameters. As an example, the user may define that an inserted operation may receive input from a specified number of preceding layers or a variable (e.g., random) number of preceding layers. As another example, the user may define whether skip connections are allowed, for example, here a layer may not directly provide input to a subsequent layer, but instead may skip a subsequent layer directly and connect to another subsequent layer. The user may also define a dense network architecture in which each layer is connected to all preceding layers in the model.

[0060] Furthermore, it should be noted that some implementations may provide one or more GUIs to show the progress of model exploration. For example, some implementations may generate a GUI showing scatter plots 400 that change over various iterations of model growth in a manner similar to that shown in Figures 4-6. Other implementations may display graphic representations of individual models as they are generated.

[0061] Methods for generating hardware consciousness in machine learning models Figure 9 shows exemplary methods 900 in several implementation forms of this concept. Method 900 can be implemented on many different types of devices, for example, by one or more cloud servers, by client devices such as laptops, tablets or smartphones, or by a combination of one or more servers, client devices, etc.

[0062] Method 900 begins in block 902, where a first machine learning model is obtained. The first machine learning model may have one or more first inference operations.

[0063] Method 900 is followed by block 904 in which a second inference operation is identified. The second inference operation may be supported by an inference hardware architecture. In some cases, the inference hardware architecture may not support some or all of the first inference operations of the first machine learning model obtained in block 902.

[0064] Method 900 follows block 906, which is generated by modifying the first machine learning model so that the second machine learning model includes individual second inference operations supported by an inference hardware architecture. Block 906 may also include training the second machine learning model from scratch or by using transfer learning and / or warm-start techniques.

[0065] Method 900 is followed by block 908 in which the final machine learning model is selected from the second machine learning model based on one or more metrics. For example, the metrics may relate to the loss or accuracy of the second machine learning model, the latency of the second machine learning model, the power consumption of the second child machine learning model, or the memory usage of the second child machine learning model.

[0066] Method 900 is followed by block 910, which outputs the final model. For example, block 910 may include sending the final model to a different device, registering the final model for use in a specific application, making the final model available via a web service, etc.

[0067] Method 900 is followed by block 912 in which the task is performed by the final model. For example, block 912 may include providing input data to the final model, running the final model on the input data to obtain one or more results, and outputting the results. The results may be output to a specific application via an API, output by writing to persistent storage, transmitted to a remote application over a network, or output via an input / output device such as a display or speaker.

[0068] Dynamic runtime selection method for machine learning models Figure 10 illustrates an exemplary method 1000 in several implementation forms of this concept. Method 1000 can be implemented on many different types of devices, for example, by one or more cloud servers, by client devices such as laptops, tablets, or smartphones, or by a combination of one or more servers and client devices, and so on.

[0069] Method 1000 begins in block 1002, where the device context is determined. For example, the device context may relate to the resource availability of the device, such as memory, CPU, network or storage utilization, or whether the device is plugged in.

[0070] Method 1000 is followed by block 1004 in which a specific machine learning model is selected from a plurality of machine learning models available to the computing device. For example, some or all of the available machine learning models may be installed locally on the device or may be available for download by the device. Individual models may be optimized or adapted for various device contexts (e.g., low power consumption, low memory usage, low latency, low network or processor usage).

[0071] Method 1000 is followed by block 1006, in which a specific machine learning model is executed to perform a task. For example, block 1006 may include providing input data to the specific model and running the specific model on the input data to obtain one or more results.

[0072] Method 1000 is followed by block 1008, which outputs the results. The results may be output to a specific application via an API, written to persistent storage, transmitted to a remote application over a network, or output via an input / output device such as a display or speaker.

[0073] Alternative implementation forms The concepts described herein were conveyed above by using evolutionary search procedures to show how the machine learning model space can be explored, while taking into account the availability of hardware-assisted inference operations. However, the specific techniques described above can be readily extended to various other methods for the automatic generation of machine learning models.

[0074] For example, consider a method that employs reinforcement learning to discover new model architectures. In some implementations, an inquiry strategy may be provided that encourages the search for model architectures with hardware-assisted inference operations. Furthermore, a reward function may be defined that considers not only the overall model performance for one or more metrics, but also the per-operation metrics obtained through hardware emulation of these operations.

[0075] As another example, consider a method that employs Bayesian optimization to explore new machine learning models. In some implementations, an acquisition function may be defined that takes into account the availability and / or performance of hardware-assisted inference operations when determining which model to explore. A similar approach may be employed for one-shot model generation, for example, by defining a supernetwork with candidate hardware-assisted inference operations that are trained together and then sorted through the supernetwork to select a specific path as the final model.

[0076] Technical effects As noted above, modem inference hardware architectures provide specific hardware instructions that perform operations that tend to occur in neural networks, such as convolutions or matrix operations. For example, an inference hardware architecture may provide instructions for convolution operations with specific input / output tensors and / or kernel size vectors, or matrix operations, activation functions, etc., with specific input and output tensor size pooling operations. When a machine learning model is developed using convolutions or matrix operations supported by a given inference hardware architecture, the machine learning model can be executed very efficiently on a processing unit that supports that architecture.

[0077] However, as noted above, conventional methods for automatically generating machine learning models tend to be unaware of the availability of hardware-assisted inference operations when developing new models. By searching for inference operations supported by inference hardware architectures and modifying existing models to include those supported inference operations, new models that achieve comparable accuracy to the original models with significantly better performance can be identified. For example, when run on a processing unit that implements an inference hardware architecture, the new model may have lower latency, lower power consumption, lower memory usage, and so on.

[0078] Furthermore, this disclosed implementation enables the generation of new machine learning models based on multiple metrics. Therefore, multiple new models can be generated, tailored to several device contexts, such as the availability of power or computing resources. As a result, the device can dynamically adjust which model is executed in different contexts.

[0079] Furthermore, this disclosed implementation enables the generation of machine learning models in a manner that considers the arrangement of individual inference operations on various processing units. In some cases, the search considers the arrangement of several inference operations on processing units that do not support the target inference hardware architecture and on other processing units that do support the target inference hardware architecture. As a result, models can be generated that leverage the advantages of various types of processing units while considering the bandwidth between processing units for the transmission of intermediate results during processing.

[0080] Exemplary application The technologies discussed herein can be used in a variety of applications, but are not limited to them. Nevertheless, the following are some specific examples for illustrative purposes.

[0081] As a first example, suppose an entity wants to provide an application that performs background segmentation during video calls. This entity may have an existing model that it currently uses for this purpose and that can run on client device 740. However, suppose the entity discovers that the background segmentation model exhibits high power consumption and long latency, resulting in excessive battery consumption and occasional video jitter during video calls.

[0082] An entity can upload an existing model-to-model generation module 721 onto server 720 and configure various initial parameters as discussed above by using configuration module 711. The model generation module can then modify the existing model by performing two different searches: one using a first set of metrics relating to accuracy / loss and power consumption, and the other using a second set of metrics relating to accuracy / loss and latency. Thus, the model generation module can output two final models: a first final model optimized for short latency (perhaps at the expense of higher power consumption), and a second final model optimized for low power consumption (perhaps at the expense of some additional latency).

[0083] At runtime, the model selection module 703(4) on the client device 740 may evaluate the device context (such as whether the device is plugged in and / or the current battery level) to determine which model to use. If the device is plugged in and / or the current battery level is above a threshold (e.g., 80%), the first model may be selected and executed, thus providing a low-latency user experience through seamless background segmentation. If the device is not plugged in and / or the current battery level is below a threshold, the second model may be selected and executed, thus saving battery power while potentially degrading the video experience somewhat compared to the first model.

[0084] Now, let's consider a second example where an entity wants to provide an object recognition service for user-uploaded images. The entity may have an existing model on server 730 that works well for object recognition. However, the entity may discover that server 730 starts to run out of memory during busy periods, and that at other times electricity costs are very high, making it expensive to run the object recognition service.

[0085] As described above, the entity uploads the model to the model generation model 721 on server 720, configures the search, and may obtain two new final models: a first model with relatively low memory usage and a second model with relatively low power consumption. If memory is constrained on server 730, the server may use the first model; if electricity costs are high, the server may use the second model.

[0086] definition For the purposes of this specification, the term “inference hardware architecture” refers to a set of operations provided by one or more processing units adapted for machine learning inference processing. For example, inference operations may be performed in a dedicated circuit configuration on a processing unit configured to use a specific data size (e.g., input size, output size, kernel size, etc.). The term “inference operations” refers to operations performed by a machine learning model to perform a task. For example, inference operations may be performed by applying learned parameters obtained by training a machine learning model.

[0087] The term "trained parameters" refers to parameters such as edge weights and bias values ​​that are learned by training a machine learning model, such as a neural network. The term "operation" refers to a function that can be performed by one or more nodes. The term "model structure" refers to the overall architecture of a model, including the number of layers or nodes, the connectivity of the layers, and / or the type of operations performed by each layer. The term "neural network structure" refers to the model structure of a neural network. The term "trained model" refers to a model structure that has the trained parameters of the model structure. Note that two trained models may share the same model structure but may have different trained parameters (for example, if the two models are trained on different training data or if there is an intrinsic stochastic process in the training process).

[0088] The term "parent model" refers to a model that is later modified to obtain a "child model." A "seed model" (for example, an existing model selected as a starting point for exploring a machine learning model search space) is one type of parent model. The term "final model" is used herein solely to indicate that a given model is designated for actual use in an application. In some cases, a final model output by a first exploration of a machine learning model search space may be later adopted as a seed model to initiate a second exploration, resulting in a second final model.

[0089] Device implementation form As noted above with respect to Figure 7, system 700 includes several devices, including client device 710, server 720, server 730, and client device 740. Also as noted, not all device implementation configurations can be shown, and other device implementation configurations should be apparent to those skilled in the art from the above and below descriptions.

[0090] As used herein, the terms “device,” “computer,” “computing device,” “client device,” or “server device” may mean any type of device having a certain amount of hardware processing power and / or hardware storage / memory capacity. Processing power may be provided by one or more hardware processors (e.g., hardware processing units / cores) capable of executing data in the form of computer-readable instructions to provide functionality. Computer-readable instructions and / or data may be stored on storage such as storage / memory and / or data storage. As used herein, the term “system” may mean a single device, a group of devices, and so on.

[0091] Storage resources may be located inside or outside the respective devices to which they relate. Storage resources may include, among many others, any one or more of volatile or non-volatile memory, hard drives, flash storage devices, and / or optical storage devices (e.g., CDs, DVDs, etc.). As used herein, the term “computer-readable medium” may include signals. In contrast, the term “computer-readable storage medium” excludes signals. Computer-readable storage medium includes “computer-readable storage devices.” Examples of computer-readable storage devices include, among many others, volatile storage media such as RAM, and non-volatile storage media such as hard drives, optical discs, and flash memory.

[0092] In some cases, the device consists of a general-purpose hardware processor and storage resources. In other cases, the device may include a system-on-a-chip (SOC) type design. In an SOC design implementation, the functionality provided by the device may be integrated on a single SOC or multiple combined SOCs. One or more associated processors may be configured to coordinate shared resources such as memory and storage and / or one or more dedicated resources such as hardware blocks configured to perform certain specific functionalities. Thus, as used herein, the terms “processor,” “hardware processor,” or “hardware processing unit” may also refer to a central processing unit (CPU), graphics processing unit (GPU), controller, microcontroller, processor core, or other types of processing devices suitable for implementations in both traditional computing architectures and SOC designs.

[0093] Alternatively or in addition, the functionality described herein may be performed at least in part by one or more hardware logic components. For example, exemplary types of hardware logic components that may be used include, without limitation, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-chip systems (SOCs), complex-programmable logic devices (CPLDs), and so on.

[0094] In some configurations, any of the modules / code discussed herein may be implemented in software, hardware, and / or firmware. In any case, the modules / code may be provided during the manufacture of the device or by an intermediary who creates the device for sale to the end user. In other instances, the end user may install these modules / code later, for example, by downloading the executable code and installing the executable code on the corresponding device.

[0095] It should also be noted that devices can generally have input and / or output functionality. For example, computing devices can have various input mechanisms such as keyboards, mice, touchpads, voice recognition, and gesture recognition (e.g., using depth cameras such as stereoscopic or time-of-flight camera systems, infrared camera systems, RGB camera systems, or accelerometers / gyroscopes, facial recognition, etc.). Devices can also have various output mechanisms such as printers and monitors.

[0096] It should also be noted that the devices described herein may function standalone or in a cooperative manner to implement the aforementioned technologies. For example, the methods and functionalities described herein may be performed by a single computing device and / or distributed across multiple computing devices communicating on a network 750. Network 750 may include, but is not limited to, one or more local area networks (LANs), wide area networks (WANs), the Internet, and so on.

[0097] Various examples have been described above. Additional examples are described below. One example includes a method performed on a computing device, which includes obtaining a first machine learning model having one or more first inference operations, identifying multiple second inference operations supported by an inference hardware architecture, generating a second machine learning model by modifying the first machine learning model to include each of the second inference operations supported by the inference hardware architecture, and selecting a final machine learning model from the second machine learning model based on one or more metrics.

[0098] Another example may include any of the above and / or below examples in which one or more metrics relate to the loss or accuracy of the second machine learning model.

[0099] Another example may include any of the above and / or below examples in which one or more metrics relate to the latency, power consumption, or memory usage of a second machine learning model.

[0100] Another example may include any of the examples above and / or below, further comprising the method simulating the execution of a second machine learning model on a central processing unit to determine one or more metrics.

[0101] Another example may include any of the examples above and / or below, wherein the method further includes determining the frontiers of a second machine learning model for multiple metrics and selecting a final machine learning model from the frontiers.

[0102] Another example may include any of the examples above and / or below, further comprising two or more iterations of selecting a subset of a second machine learning model for further modification and generating a further second machine learning model from the selected subset.

[0103] Another example may include any of the examples above and / or below, in which generating individual second machine learning models involves removing individual first inference operations from the first machine learning model that are not supported by the inference hardware architecture.

[0104] Another example may include any of the examples above and / or below, in which the method further involves running a second machine learning model by using hardware emulation of individual second inference operations.

[0105] Another example may include any of the examples above and / or below, wherein the method further includes obtaining each per-operation metric via hardware emulation and using each per-operation metric to select individual second machine learning models as parent models for further modifications or to select the final machine learning model.

[0106] Another example includes a hardware processing unit and a storage resource for storing computer-readable instructions, which, when executed by the hardware processing unit, cause the hardware processing unit to: perform a search of a machine learning model search space having multiple inference operations supported by an inference hardware architecture, the search including emulation of the inference architecture hardware, and output a final machine learning model selected from the machine learning model search space.

[0107] Another example may include any of the above and / or below examples, where the inference operation involves a convolution operation, a vector operation, or a matrix operation with specified input and output data sizes.

[0108] Another example may include any of the examples above and / or below, in which the search is performed by starting with a seed model selected based on its performance for a particular task.

[0109] Another example may include any of the above and / or below examples, where the seed model includes certain inference operations that are not supported by the inference hardware architecture.

[0110] Another example may include any of the above and / or below examples in which the final machine learning model does not include specific inference operations.

[0111] Another example may include any of the examples above and / or below, where the search involves training multiple machine learning models with various inference operations supported by an inference hardware architecture.

[0112] Another example may include any of the above and / or below examples, where the search considers the placement of individual inference operations on a first processing unit that does not support the inference hardware architecture and a second processing unit that supports the inference hardware architecture, and the final machine learning model instructs that some inference operations are performed on the first processing unit and other inference operations are performed on the second processing unit.

[0113] Another example includes a computing device comprising a hardware processing unit configured to perform multiple aided inference operations and storage resources for storing computer-readable instructions: when the computer-readable instructions are executed by the hardware processing unit, the hardware processing unit is caused to: determine the device context of the computing device, at least based on the device context; select a particular machine learning model from several machine learning models available to the computing device, the several machine learning models having various aided inference operations; and execute the particular machine learning model to perform a particular task.

[0114] Another example may include any of the above and / or below examples where the device context relates to the availability of power or memory on a computing device.

[0115] Another example may include any of the above and / or below examples. When a computer-readable instruction is executed by a hardware processing unit, it causes the hardware processing unit to: in a first instance where the availability of memory on the computing device is constrained, select a first machine learning model generated based on a first metric relating to at least memory usage as a specific machine learning model to perform a particular task; in a second instance where the availability of power to the computing device is constrained, select a second machine learning model generated based on a second metric relating to at least power consumption as a specific machine learning model to perform a particular task.

[0116] conclusion While this subject matter is described in language specific to structural features and / or methodological actions, it should be understood that the subject matter defined in the attached claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are disclosed as exemplary forms that implement the claims, and other features and actions that would be recognized by those skilled in the art are intended to fall within the scope of the claims.

Claims

1. A method performed on a computing device, wherein the method is: Obtaining a first machine learning model having one or more first inference operations; Identifying multiple second inference operations supported by the inference hardware architecture; Generating a second machine learning model by modifying the first machine learning model to include individual second inference operations supported by the aforementioned inference hardware architecture; and Selecting a final machine learning model from the second machine learning model based on one or more metrics. Methods that include...

2. The method according to claim 1, wherein the one or more metrics relate to the loss or accuracy of the second machine learning model.

3. The method according to claim 1, wherein the one or more metrics relate to the latency, power consumption, or memory usage of the second machine learning model.

4. The method according to claim 1, further comprising simulating the execution of the second machine learning model on a central processing unit to determine the one or more metrics.

5. Determining the frontier of the aforementioned second machine learning model with respect to multiple metrics; and Selecting the final machine learning model from the aforementioned frontier The method according to claim 1, further comprising:

6. The process involves two or more iterations of selecting a subset of the second machine learning model for further modification, and generating a further second machine learning model from the selected subset. The method according to claim 1, further comprising:

7. The method according to claim 1, wherein generating individual second machine learning models includes removing individual first inference operations from the first machine learning model that are not supported by the inference hardware architecture.

8. The second machine learning model is executed by using hardware emulation of each of the second inference operations. The method according to claim 1, further comprising:

9. Obtaining each metric for each operation via the aforementioned hardware emulation; and Use the respective per-operation metrics to select individual second machine learning models as parent models for further modification or to select the final machine learning model. The method according to claim 8, further comprising:

10. Outputting multiple final machine learning models selected according to various metrics. The method according to claim 1, further comprising:

11. Hardware processing unit and; Storage resources for storing computer-readable instructions and A system including, where the computer-readable instruction, when executed by the hardware processing unit, is sent to the hardware processing unit: The present invention relates to a search for a machine learning model having multiple inference operations supported by an inference hardware architecture, wherein the search includes emulation of the inference architecture hardware; The final machine learning model selected from the aforementioned machine learning model search space is output. A system that enables this to happen.

12. The system according to claim 11, wherein the inference operation includes a convolution operation, a vector operation, or a matrix operation having specified input and output data sizes.

13. The system according to claim 11, wherein the search is performed by starting with a seed model selected based on performance for a specific task, the seed model includes specific inference operations not supported by the inference hardware architecture, and the final machine learning model does not include the specific inference operations.

14. The system according to claim 11, wherein the search includes training a plurality of machine learning models having various inference operations supported by the inference hardware architecture.

15. The system according to claim 11, wherein the search takes into account the arrangement of individual inference operations on a first processing unit that does not support the inference hardware architecture and a second processing unit that supports the inference hardware architecture, and the final machine learning model indicates that some inference operations are performed on the first processing unit and other inference operations are performed on the second processing unit.

Images