Efficient hardware accelerator configuration exploration

Abstract

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for training a proxy neural network configured to determine a predicted performance metric of a hardware accelerator having a target hardware configuration on a target application. In addition to, or instead of, hardware simulation, a trained instance of the proxy neural network can be used during a search process to determine a hardware configuration of an application-specific hardware accelerator (i.e., a hardware accelerator on which one or more neural networks can be deployed to perform one or more target machine learning tasks).

Description

[0001] Cross-references to related applications

[0002] This application claims priority to U.S. Provisional Application No. 63 / 216,925, filed June 30, 2021. The disclosure of the prior application is considered part of the disclosure of this application and is incorporated herein by reference. Background Technology

[0003] This manual relates to determining the configuration of the hardware accelerator.

[0004] Hardware accelerators are computing devices with specialized hardware configured to perform specialized computations, such as graphics processing units (“GPUs”), field-programmable gate arrays (“FGPAs”), and application-specific integrated circuits (“ASICs”), including tensor processing units (“TPUs”). Summary of the Invention

[0005] This specification describes a system of computer programs implemented as one or more computers at one or more locations, which determines the hardware configuration of an application-specific hardware accelerator on which one or more machine learning models can be deployed to perform one or more machine learning tasks.

[0006] Hardware accelerators are computing devices that include specialized hardware for performing certain types of operations, such as matrix multiplication, and are more efficient than non-specialized or "general-purpose" computing devices. Different hardware accelerators can have different hardware characteristics, such as different computing power, memory, bandwidth, etc.

[0007] The key characteristic of application-specific hardware accelerators is that they are primarily geared towards the needs of one or more specific applications. Compared to general-purpose computing devices used in these applications, they offer advantages such as smaller size, lower power consumption, higher reliability, higher performance, and lower cost. For example, application-specific hardware accelerators for machine learning applications can have hardware characteristics that are not only geared towards the details of the machine learning task (e.g., data modality, data size, etc.) but also towards the computational requirements (e.g., number of computational operations (e.g., convolution, depthwise convolution, feedforward, etc.) of the various types of machine learning models deployed on the application-specific hardware accelerator to perform these tasks, such as the number of computational operations (e.g., convolution, depthwise convolution, feedforward, etc.), model parameter size, instruction size, etc.).

[0008] An application-specific hardware accelerator can be part of a target computing device that includes one or more hardware accelerators. As an example, a target computing device that includes one or more hardware accelerators can be a single, specific edge device, such as a mobile phone, a smart speaker, or another embedded computing device, or other edge devices. As a concrete example, an edge device can be a mobile phone or another device with a specific type of hardware accelerator or other computer chips on which neural networks will be deployed.

[0009] As another example, a target computing device that includes one or more hardware accelerators can be a set of multiple hardware accelerator devices, such as ASICs, FPGAs, or Tensor Processing Units (TPUs) on a real-world agent, such as a vehicle, an autonomous car, or a robot.

[0010] As yet another example, a target computing device that includes one or more hardware accelerators can be a set of hardware accelerators in a data center.

[0011] The subject matter described in this specification may be implemented in particular embodiments to achieve one or more of the following advantages.

[0012] Hardware accelerators are specialized hardware configured to perform specific computations and are generally more computationally efficient than their general-purpose counterparts, but are typically associated with higher operating costs due to the energy required to power and maintain them. Using neural networks deployed on hardware accelerators to efficiently perform machine learning tasks, such as vision, text, audio, or other tasks requiring near real-time responses to users, necessitates specially designed hardware accelerator configurations—configurations already tailored for machine learning tasks, neural networks, or both.

[0013] Existing accelerator configuration design methodologies typically involve exploring a large multimodal candidate configuration space to determine corresponding values ​​for hardware parameters, and then evaluating the performance of the hardware accelerator, whose configuration has been defined by the determined parameter values. For most modern hardware accelerators, such as GPUs or TPUs, this requires selecting a set of parameter values ​​from a large discrete design space, integer design space, ordinal design space, cardinal design space, or a combination thereof. Repeatedly performing this search process is computationally intensive and consumes significant computational resources due to the exhaustive nature of the parameter value sampling step size and the cost of running expensive simulators or other analytical models for hardware performance evaluation.

[0014] Furthermore, the accelerator design space is typically characterized by a narrow manifold of feasible accelerators within a vast number of infeasibility points. While some of these infeasibility points can be identified by simple rules, most correspond to failures during compilation or hardware simulation. These infeasibility points usually cannot be directly expressed as an optimization problem using formulas.

[0015] Furthermore, the optimization objective may exhibit high sensitivity to small changes in some architectural parameters in certain regions of the design space, but remain relatively insensitive in others, leading to complex optimization scenarios. This suggests that optimization algorithms based on local parameter updates (e.g., gradient ascent, evolutionary schemes, etc.) may have challenging tasks traversing near-flat objective scenarios, potentially resulting in poor performance.

[0016] This specification describes a data-driven offline optimization framework that utilizes recorded simulated data from prior design tasks or experiments. The described framework allows for the use of configuration data from recorded datasets in such a way that values ​​of the configuration data are added to determine the optimal hardware accelerator design under various design constraints. Specifically, the described framework can learn a surrogate neural network configured to predict performance metrics of a hardware accelerator from a one-time collected offline dataset, and automatically search for high-performance hardware accelerators (e.g., applying a specific hardware accelerator) by using the surrogate neural network to optimize hardware parameter values ​​that define the hardware configuration.

[0017] The described framework eliminates the recurring costs of large-scale hardware simulation scans by removing any additional queries to expensive hardware simulators used to evaluate the various candidate configurations generated during the design process. Instead, each time, only a more efficient forward pass computation via a proxy neural network is needed to determine the predictive performance metric for the candidate hardware accelerator. Furthermore, the proxy neural network trained using the described framework is more robust than existing methods such as black-box optimization schemes and largely avoids unrealistically optimistic performance metric predictions for any given hardware configuration, even those defined by rarely seen or out-of-distribution hardware parameter values. Therefore, the described framework can be used to design hardware accelerator configurations with performance comparable to those designed using state-of-the-art online methods, albeit with significantly reduced time, computational resources, or both compared to existing techniques.

[0018] Details of one or more embodiments of the subject matter of this specification are set forth in the accompanying drawings and the following description. Other features, aspects, and advantages of this subject matter will become apparent from the description, drawings, and claims. Attached Figure Description

[0019] Figure 1 An example neural network training system and an example hardware configuration determination system are shown.

[0020] Figure 2 This is a flowchart of an example process for training a proxy neural network.

[0021] Figure 3 This is an example illustration of training an agent neural network.

[0022] Figure 4 This is a flowchart of an example process for determining the final hardware configuration of a hardware accelerator.

[0023] The same reference numerals and names in the various figures denote the same elements. Detailed Implementation

[0024] Figure 1 An example neural network training system 100 and an example hardware configuration determination system 150 are illustrated. The neural network training system 100 and the hardware configuration determination system 150 are examples of systems implemented as computer programs on one or more computers at one or more locations, wherein the systems, components and techniques described below can be implemented.

[0025] The neural network training system 100 is a system for training a neural network 110 (“surrogate neural network”) to determine training values ​​of the parameters of the surrogate neural network 110 from initial values ​​of the surrogate network parameters, referred to herein as surrogate network parameters.

[0026] The proxy neural network 110 is a neural network configured to receive input data of a specified target hardware configuration and process the input data according to proxy network parameters to generate output data, wherein the output data specifies a predictive performance metric of a hardware accelerator with the target hardware configuration.

[0027] In some implementations, the input data specifying the target hardware configuration may include a number of corresponding hardware parameters that collectively define the computational characteristics of the hardware accelerator with the target hardware configuration, which will be further described below with reference to Table 1.

[0028] In some implementations, the predictive performance metric may include one or more of the following: the predicted runtime latency of a machine learning model deployed on a hardware accelerator with the target hardware configuration, or the predicted power consumption of the hardware accelerator. Runtime latency is a metric that measures the time required, for example, in milliseconds, for performing inference on a batch of one or more inputs when the machine learning model is deployed on a hardware accelerator with the target hardware configuration; that is, the time required for the machine learning model to process each input in a batch of inputs.

[0029] In some implementations, the input data may also include a description of the target application, represented as a context vector. That is, in some implementations, the prediction performance metric is conditioned on the target application. For example, the context vector may include one or more context variables specifying a set of attributes of the target application. The attributes of the target application may include attributes of the machine learning task and attributes of the machine learning model to be deployed on a hardware accelerator to perform the task. In some examples, the context variables may include one-hot encoded variables that identify which of several machine learning tasks the machine learning model is configured to perform across various domains (e.g., visual, text, and audio domains). Context variables may also include integer or floating-point variables that define the number of computations (e.g., convolutions, depthwise convolutions, feedforwards, etc.) of the machine learning model, the model parameter size, the instruction size, the number of computational operations, etc.

[0030] The proxy neural network 110 can be implemented using any suitable neural network architecture that enables it to perform its described functions. In various examples, the proxy neural network 110 can be a fully connected neural network, i.e., including one or more fully connected neural network layers, or alternatively, a recurrent neural network, for example, including one or more long short-term memory (LSTM) neural network layers.

[0031] As a specific example, the agent neural network 110 can be an attention neural network comprising multiple attention layers. Examples of configurations of attention neural networks are described in more detail in Vaswani et al., “Attention Is All You Need”, arXiv:1706.03762, Raffel et al., “Exploring the Limits of Transfer Learning with a Unified Text-to-TextTransformer”, arXiv:1910.10683, and Devlin et al., “BERT: Pre-training of DeepBidirectional Transformers for Language Understanding”, arXiv:1810.04805.

[0032] To effectively train the agent neural network 110, the neural network training system 100 maintains a hardware configuration training dataset 120 and uses the training dataset 120 to train the agent neural network 110 to determine training values ​​for the agent network parameters based on optimizing the agent objective function. For example, the training dataset 120 may include offline training inputs that have been generated or otherwise determined from recorded simulated data or from prior hardware design tasks or experiments, and may optionally include random training inputs generated by random sampling from the space of candidate hardware accelerator configurations. See below for reference. Figure 2-3 The training process will be described in more detail.

[0033] Specifically, training dataset 120 includes multiple first training inputs specifying feasible hardware configuration 126. Training dataset 120 also includes multiple second training inputs specifying infeasible hardware configuration 128. During training, by training surrogate neural network 110 to generate more conservative predictive performance metrics (i.e., underestimated performance metrics) for these second training inputs specifying infeasible hardware configuration 128, neural network training system 100 can improve the robustness of the performance metrics predictions generated by surrogate neural network 110 relative to uncommon hardware configurations (i.e., hardware configurations defined by rarely seen or out-of-distribution hardware parameter values).

[0034] The feasibility of a hardware configuration can be determined by evaluating it against a set of feasibility rules or criteria, and can be additionally or alternatively determined by compilation or hardware simulation. The feasibility of a hardware configuration can depend on the target software, the underlying hardware, or both. For example, a hardware configuration might be infeasible because it specifies hardware that cannot be built on silicon. As another example, a hardware configuration might be infeasible because it specifies hardware on which the target software (the source code of the neural network) cannot be compiled, for example, due to insufficient memory units in the hardware accelerator.

[0035] After training, the neural network training system 100 can then output training instance data of a specified agent neural network 110 to the hardware configuration determination system 150, for example, via a wired or wireless network. For example, the data may include architectural data of the specified agent neural network architecture and parameter data of the training values ​​of the specified agent network parameters.

[0036] The hardware configuration determination system 150 is a system that deploys training instances of the agent neural network 110 and uses these training instances to determine the hardware configuration (or architecture) of a dedicated hardware accelerator. A specific hardware accelerator is a hardware accelerator on which one or more machine learning models (including one or more neural networks) can be deployed to perform one or more target machine learning tasks.

[0037] Depending on the task(s), the neural network to be deployed on the hardware accelerator can be configured to receive any kind of digital data input and generate any kind of score, classification or regression output based on that input.

[0038] In some cases, neural networks are configured to perform image processing tasks, i.e., to receive an input image and process it to generate a network output. In this specification, processing the input image refers to using a neural network to process the intensity values ​​of image pixels. For example, the task could be image classification, and the output generated by the neural network for a given image could be a score for each of a set of object categories, each score representing an estimated probability that the image contains an object belonging to that category. As another example, the task could be image embedding generation, and the output generated by the neural network could be a digit embedding of the input image. As yet another example, the task could be object detection, and the output generated by the neural network could identify the location of objects depicting a specific type in the input image. As yet another example, the task could be image segmentation, and the output generated by the neural network could define which of several categories each pixel of the input image belongs to.

[0039] As another example, if the input to the neural network is an internet resource (e.g., a webpage), a document, or a portion of a document, or features extracted from an internet resource, document, or portion of a document, then the task can be to classify the resource or document. That is, the output generated by the neural network for a given internet resource, document, or portion of a document can be a score for each of a set of topics, each score representing an estimated probability that the internet resource, document, or portion of a document is relevant to a topic.

[0040] As another example, if the input to the neural network is features of the impression context of a particular ad, the output generated by the neural network can be a score representing the estimated probability that the particular ad will be clicked.

[0041] As another example, if the input to the neural network is features of personalized recommendations for a user, such as features characterizing the context of the recommendation, such as features characterizing the user's previous actions, then the output generated by the neural network could be a score for each of a set of content items, each score representing an estimated probability that the user will make a favorable response to the recommended content item.

[0042] As another example, if the input to a neural network is a sequence of text in one language, the output generated by the neural network can be a score for each of a set of text fragments in another language, where each score represents an estimated probability that the text fragment in the other language is a correct translation of the input text into the other language.

[0043] As another example, if the input to a neural network is a sequence representing spoken utterances, the output generated by the neural network could be a score for each of a set of text segments, each score representing an estimated probability that the text segment is a correct transcription of the utterance.

[0044] As another example, the task could be an audio processing task. For instance, if the input to a neural network is a sequence representing spoken utterances, the output generated by the neural network could be a text fragment that is a correctly predicted transcription of the utterance. As another example, if the input to a neural network is a sequence representing spoken utterances, the output generated by the neural network could indicate whether a specific word or phrase (“hot word”) was spoken in that utterance. As yet another example, if the input to a neural network is a sequence representing spoken utterances, the output generated by the neural network could identify the natural language used to speak that utterance.

[0045] As another example, the task can be a natural language processing or understanding task, where the input is a sequence of natural language text and the output is a natural language processing or understanding output. An example of such a task is the entailment task, where the input consists of multiple natural language sentences and the output indicates the entailment between the sentences. Another example is the paraphrasing task, where the input is a sequence of natural language sentences and the output identifies another natural language sequence that has a similar meaning to the input sequence. Another example is the text similarity task, where the input is multiple natural language sequences and the output indicates how similar the input sequences are, for example, semantically similar. Another example is the sentiment task, where the input is a sequence of natural language sentences and the output represents the sentiment of the input sequence. Another example is the sentence completion task, where the input is a sequence of natural language sentences and the output identifies another natural language sequence that completes the input sequence. Another example is the summarization task, where the input is an input natural language text sequence and the output is a summary natural language sequence that is shorter than the input sequence but summarizes the input sequence, representing the most important or relevant information within the input sequence. In some cases, summarizing tasks are abstraction tasks, where the output sequence is a proper subset of the input sequence, i.e., it consists of text derived from the input sequence. In other cases, summarizing tasks are abstract summarizing tasks, where the output is a new sequence that may contain text different from the input sequence. Another example is grammatical tasks, where the input is a sequence of natural language text, and the output represents the grammaticality of the input sequence, i.e., how grammatically correct the input sequence is.

[0046] As another example, the task could be a text-to-speech task, where the input is text in natural language or text features in natural language, and the network output is a spectrogram, waveform, or other data that defines the audio of the text spoken in natural language.

[0047] As another example, the task could be a health prediction task, where the input is a sequence derived from a patient's electronic health record (EHR), and the output is a prediction related to the patient's future health. In this example, the patient's EHR data could include multiple features representing health events in the patient's EHR, each belonging to a vocabulary of possible features. An example type of possible features is numerical features, which include laboratory test results or patient vital sign measurements—measurements of vital signs captured by medical devices—both of which can take many different possible values. Another example type of possible features is discrete features, which include binary features, such as whether a patient was admitted for treatment within a given time period, or categorical features, such as procedure codes assigned to a specific procedure or event. For a health prediction task, the output could include a predicted treatment that should be prescribed for the patient, the probability that the patient will experience an adverse health event, or a predicted diagnosis for the patient.

[0048] As another example, the task could be a text generation task, where the input is a sequence of text and the output is another sequence of text, such as the completion of the input text sequence, a response to a question posed in the input sequence, or a sequence of text about a topic specified by a first text sequence. As another example, the input to a text generation task could be anything other than text, such as an image, and the output sequence could be text describing that input. As yet another example, the input to a text generation task could include both input from different modalities and text, and the output sequence could be text in response to the input. For example, the task could be a visual question-and-answer task, and the input could include one or more images and a text question about said one or more images, and the output sequence could be the answer to the text question.

[0049] As another example, the task could be an image generation task, where the input is a conditioning input and the output is an input sequence of intensity values ​​of image pixels. The conditioning input could include one or more of the following, for example, class labels identifying the desired categories of objects that should be depicted in the image, a sequence of text describing the desired content of the image, or another image, such as an image of objects that should be included in the new image or a lower-resolution image that should be upscaled to a higher resolution to generate the new image.

[0050] As another example, the task could be an agent control task, where the input is an observation characterizing the state of the environment, and the output defines the action to be performed by the agent in response to the observation. The agent could be, for example, a real-world or simulated robot, a control system for an industrial facility, or a control system that controls different types of agents. Observations could include sensor data captured by sensors measuring the environment (e.g., camera sensors, LiDAR sensors, temperature sensors, humidity sensors, etc.).

[0051] As another example, the task could be a genomics task, where the input is a sequence representing a fragment of DNA or other molecular sequence, and the output is an embedding of the fragment for use in downstream tasks, such as by utilizing unsupervised learning techniques on a dataset of DNA sequence fragments, or the output of a downstream task. Examples of downstream tasks include promoter site prediction, methylation analysis, predicting the functional effects of non-coding variants, and so on.

[0052] In some cases, a machine learning task is a combination of multiple individual machine learning tasks; that is, a neural network is configured to perform multiple different individual machine learning tasks, such as the two or more machine learning tasks mentioned above. For example, a neural network can be configured to perform multiple individual natural language understanding tasks. Optionally, the network input may include identifiers of the individual natural language understanding tasks to be performed on the network input. As another example, a neural network can be configured to perform multiple individual image processing or computer vision tasks, that is, to generate the outputs of multiple different individual image processing tasks in parallel by processing a single input image.

[0053] The hardware configuration determination system 150 obtains data describing the accelerator configuration search space 160 and searches through the accelerator configuration search space 160 to determine the final hardware configuration of the hardware accelerator, such as the final hardware configuration 182a, on which one or more machine learning models can be deployed to perform one or more target machine learning tasks.

[0054] In some implementations, the accelerator configuration search space 160, which typically defines the candidate hardware accelerator configuration space, can be parameterized by a set of adjustable hardware parameters, each of which is associated with one or more values ​​(e.g., discrete values). Table 1 below shows an example accelerator configuration search space 160, where “PE” refers to a processing element capable of performing matrix multiplication in a single instruction multiple data (SIMD) paradigm, and for example, “PEs-X” refers to the number of processing elements along the horizontal dimension of the hardware accelerator.

[0055]

[0056] Table 1

[0057] In the example in Table 1, the multiple hardware parameters that commonly define the computational characteristics of a hardware accelerator with a target hardware configuration and are included in the input data to the proxy neural network 110 may include two or more of the following: the number of processing elements (PEs) along one dimension, the size of the PE memory, the size of the core memory, the size of the instruction memory, the size of the activation memory, the number of cores, the number of computation channels, the size of the parameter memory, and the DRAM bandwidth.

[0058] Specifically, the hardware configuration determination system 150 can determine the final hardware configuration 182a of a hardware accelerator that supports performance for (multiple) machine learning tasks with approximately specified performance metrics, such as runtime latency or power consumption, while simultaneously meeting hardware design constraints, such as area budget. For example, hardware design constraints can be specified by the system user or automatically determined by the system based on the application.

[0059] To determine the final hardware configuration 182a, system 150 utilizes one or more hardware design strategies 172a-k and a surrogate neural network 110. Example hardware design strategies, including stochastic strategies, regularized evolutionary search strategies, model-based optimization strategies, Bayesian optimization strategies, and population-based black-box optimization strategies, are described in PCT patent application PCT / US2021 / 054359, which is incorporated herein by reference.

[0060] By utilizing hardware design strategies 172a-k, system 150 repeatedly, i.e., in each of the multiple iterations of the search process, generates different candidate hardware configurations 162 from the accelerator configuration search space 160, and for each candidate hardware configuration 162, determines the predictive performance metric 164 of the hardware accelerator with the candidate hardware configuration 162 on the target application, for example, by using a proxy neural network 110 instead of relying solely on expensive hardware simulation.

[0061] Unlike expensive hardware performance simulators or other analytical models that typically take an hour or more to evaluate performance metrics for a single hardware accelerator with candidate architecture configurations, the proxy neural network 110 is faster and more resource-efficient when used to determine performance metrics. By incorporating the proxy neural network 110 into the search process, the system 150 reduces the amount of computational resources consumed by hardware simulation because it no longer needs to repeatedly perform costly queries on the hardware performance simulator. Thus, the system 150 can perform the search process with reduced latency and reduced computational resource consumption, while still being able to determine the final hardware configuration that meets the desired performance metrics and hardware design constraints.

[0062] After the search process is complete, for example, once a predetermined number of search iterations have been performed or a certain amount of time has elapsed, the hardware configuration determination system 150 can select (multiple) candidate hardware configurations that have the best performance metrics, best meet various hardware design constraints, or both, as the final hardware configuration 182a of the hardware accelerator. Alternatively or additionally, the system 100 can generate new candidate hardware configurations by using updated hardware design strategies and use the new configurations as the final hardware configuration of the hardware accelerator.

[0063] The hardware architecture search system 100 can then generate hardware configuration data as output, specifying the configuration of the hardware accelerator. The hardware configuration data may include, for example, the layout of the processing elements on the hardware accelerator, the number of computing channels, the size of various memories provided by the hardware accelerator, and the DRAM bandwidth.

[0064] For example, the hardware configuration determination system 150 can output hardware configuration data to a user who provides hardware design constraints. As another example, the hardware configuration determination system 150 can output hardware configuration data to a semiconductor manufacturing facility containing semiconductor manufacturing equipment, for example, via a wired or wireless network, which can be used to manufacture a hardware accelerator with a final hardware configuration.

[0065] In some implementations, the hardware configuration determination system 150 may be included as part of a software tool (e.g., an electronic design automation (EDA) tool) for designing and / or analyzing integrated circuits, and then the hardware configuration data may be provided to another component of that tool for further refinement or evaluation prior to manufacturing the hardware accelerator.

[0066] Figure 2 This is a flowchart of an example process 200 for training an agent neural network. For convenience, process 200 will be described as being executed by a system of one or more computers located in one or more locations. For example, a properly programmed training system, such as... Figure 1 The training system 100 can execute process 200.

[0067] The system can repeatedly execute the iteration of process 200 to repeatedly update the network parameters until a termination criterion has been met, such as until a threshold number of iterations of process 200 has been executed, until a threshold amount of wall time has elapsed, or until the values ​​of the proxy network parameters have converged.

[0068] The system maintains a hardware configuration training dataset. The hardware configuration training dataset includes multiple first hardware configuration training inputs, each specifying a corresponding predetermined feasible hardware configuration, and for each predetermined feasible hardware configuration, a corresponding target performance metric is specified. The hardware configuration training dataset also includes multiple second hardware configuration training inputs, each specifying a corresponding predetermined infeasible hardware configuration.

[0069] Hardware configuration training datasets can typically include offline training inputs that have been generated or otherwise determined from recorded simulation data or from prior hardware design tasks or experiments, and in some implementations may additionally include random training inputs generated by random sampling from the space of candidate hardware accelerator configurations.

[0070] In these implementations, the system can generate random training inputs by receiving data defining an accelerator configuration search space and generating different hardware configurations from the accelerator configuration search space based on random sampling. That is, the random training inputs can include multiple hardware parameters, each with a value randomly selected from a corresponding set of possible values. For each random training input, the system can determine, through hardware simulation, whether a hardware configuration defined by the included multiple hardware parameters is feasible, and if feasible, determine the corresponding target performance metric for the hardware accelerator with that hardware configuration.

[0071] In some implementations, the hardware configuration training dataset is fixed during the training process, while in other implementations, the training dataset is expanded as the training process is performed iteratively. For example, a third hardware configuration training input, each specifying a new hardware configuration generated as part of the training process, can be added to the training dataset, as described below.

[0072] The system selects a batch of training inputs from the hardware configuration training dataset, which includes (i) one or more first hardware configuration training inputs and (ii) one or more second hardware configuration training inputs (step 202). The system typically obtains different hardware configuration training inputs for different iterations, for example, by sampling a fixed number of training inputs from the maintained training dataset at each iteration.

[0073] Each hardware configuration training input includes a plurality of hardware parameters defining the corresponding predetermined hardware configuration specified by the hardware configuration training input, and optionally, a context vector specifying a set of attributes of the corresponding target application of the hardware accelerator. For each first training input, the corresponding target performance metric should be a benchmark ground truth performance metric of the first training input generated by the proxy neural network through processing the first training input.

[0074] The system uses a proxy neural network to process each first hardware configuration training input based on the current values ​​of multiple proxy network parameters to determine the corresponding predictive performance metric for each predetermined feasible hardware configuration specified in one or more first hardware configuration training inputs (step 204).

[0075] In some implementations, the predictive performance metric can be the runtime latency of a machine learning model deployed on a hardware accelerator on a target application, for example, to perform a target machine learning task. Runtime latency can be the latency of the (trained) machine learning model executing the target machine learning task; that is, the latency for processing new inputs to the target task using the machine learning model deployed on the hardware accelerator after the hardware configuration of the hardware accelerator has been determined. In some implementations, the predictive performance metric can be the power (or energy) consumption of the hardware accelerator on the target application; that is, the power (or energy) consumption when supporting the execution of the machine learning model on the target task.

[0076] The system uses a proxy neural network to process each second hardware configuration training input based on the current values ​​of multiple proxy network parameters to determine the corresponding predictive performance metric for each predetermined infeasible hardware configuration specified in one or more second hardware configuration training inputs (step 206).

[0077] Alternatively, in some implementations, during the training process, the system can repeatedly generate unseen hardware configurations that were not previously used (or rarely used) when training the agent neural network by scouring the space of candidate architectures in order to generate new training inputs specifying the unseen hardware configurations.

[0078] In other words, the system can optionally generate one or more third hardware configuration training inputs, each specifying a new hardware configuration (step 208). Specifically, the system can apply one or more hardware design strategies to the accelerator configuration search space to generate new hardware configurations, which, when applied to the hardware accelerator, improve the hardware accelerator's performance metrics.

[0079] In addition to or instead of the example hardware design strategies described above, the system may, for example, use generative or evolutionary optimization strategies, including optimization strategies that run the Firefly Optimization algorithm, to identify each new hardware configuration that maximizes the predicted performance metric of the new hardware configuration determined by using a surrogate neural network. Additionally or alternatively, the system may do this based on proactively modifying or changing existing hardware configurations, such as those specified in an offline training dataset. In some such examples, to allow the hardware design strategy to progressively generate new hardware configurations with improved performance metrics during training, the system may provide the predicted performance metric of the already generated hardware configurations determined by using a surrogate neural network as input to the hardware design strategy.

[0080] In these implementations, the system may optionally use a proxy neural network to process each third hardware configuration training input based on the current values ​​of multiple proxy network parameters to determine the corresponding predictive performance metric for each new hardware configuration specified in one or more third hardware configuration training inputs (step 210).

[0081] The system determines the gradient of the surrogate objective function with respect to multiple surrogate network parameters (step 212), and determines the update of the current values ​​of the multiple surrogate network parameters based on the gradient. For example, the system can determine the gradient through backpropagation, and can determine the update by applying an update rule to the gradient, such as a stochastic gradient descent update rule, an Adam optimizer update rule, or an rmsProp update rule.

[0082] Figure 3 This is an example illustration of training an agent neural network.

[0083] As illustrated, the proxy objective function includes a first term 310, which is specific to each predetermined feasible hardware configuration x specified in one or more first hardware configuration training inputs. i , Measure target performance metric y i and predictive performance metrics The difference between them. The first term measures the accuracy of the surrogate neural network in predicting the performance metric of feasible hardware configurations relative to the corresponding target performance metric included in the training dataset. For example, the first term 310 could be the mean squared error between the target performance metric and the predicted performance metric, for example, The expected value is taken from a feasible sample. It should be understood that the first term 310 may alternatively have other forms.

[0084] The proxy objective function may optionally include a second term 320, which targets each predetermined infeasible hardware configuration x specified in one or more second hardware configuration training inputs. j The value of the predicted performance metric for a pre-defined, infeasible hardware configuration. This optional additional item depends on the predictive performance metric generated using a proxy neural network for infeasible hardware configurations. For example, the second item 320 could include the expected value of the predictive performance on infeasible hardware configurations, such as... It should be understood that the first item 310 may alternatively have other forms.

[0085] The proxy objective function may optionally include a third term 330, which is specific to each new hardware configuration specified in one or more third hardware configuration training inputs. Values ​​of predictive performance metrics for new hardware configurations This further optional addition depends on the predictive performance metric generated using a proxy neural network for unseen hardware configurations. For example, the third item 330 could include the expected value of the predictive performance on negative samples, e.g. It should be understood that the first item 310 may alternatively have other forms.

[0086] Specifically, the surrogate objective function minimizes the first term 310 and maximizes the second and third terms 320, i.e., including any one or both of them. Maximizing the second and third terms encourages the surrogate neural network to generate worse predictive performance metrics (i.e., underestimated performance metrics) for infeasible or unseen hardware configurations. In other words, the additional term minimizes the metric when a higher metric indicates better hardware performance, and maximizes the metric when a lower metric (e.g., lower latency) indicates better hardware performance.

[0087] In some implementations, only the parameters of the surrogate neural network are updated based on the gradient of the surrogate objective function, rather than the hardware design policy (used to generate new hardware configurations during training). This can be achieved by applying a "stop gradient" operator to the hardware design policy during training operations, i.e., by stopping gradient backflow into the hardware design policy.

[0088] Figure 4 This is a flowchart of an example process 400 for determining the final hardware configuration of a hardware accelerator. For convenience, process 400 will be described as being performed by a system of one or more computers located in one or more locations. For example, a properly programmed system, such as... Figure 1 The hardware configuration determines system 150, which can execute process 400.

[0089] The system generates multiple different candidate hardware configurations from the accelerator configuration search space (step 402), and uses the agent neural network that has been trained by executing process 200 to determine the corresponding predictive performance metric for each different candidate hardware configuration (step 404).

[0090] Specifically, the system can use any of the hardware design strategies described above to traverse the accelerator configuration search space to repeatedly, i.e., in each iteration of the search process, determine each distinct candidate hardware configuration. Each distinct candidate hardware configuration minimizes the predictive performance metric determined by training instances of a proxy neural network on the target application, while adhering to one or more hardware design constraints. For example, one or more hardware design constraints may include an area constraint on the hardware accelerator, which defines the maximum allowable area of ​​the hardware accelerator, for example, measured in square millimeters.

[0091] After the search process is completed, for example, once a predetermined number of search iterations have been performed or a certain amount of time has elapsed, the system selects the candidate hardware configuration with the best predictive performance metric from different candidate hardware configurations and based on the corresponding predictive performance metric as the hardware configuration for applying the specific hardware accelerator (step 406).

[0092] This specification uses the term "configured" in conjunction with system and computer program components. For a system of one or more computers configured to perform a specific operation or action, this means that software, firmware, hardware, or a combination thereof are installed on the system, which, in operation, causes the system to perform the operation or action. For one or more computer programs configured to perform a specific operation or action, this means that one or more programs include instructions that, when executed by a data processing device, cause that device to perform the operation or action.

[0093] Embodiments of the subject matter and functional operation described in this specification may be implemented in digital electronic circuits, in tangibly embodied computer software or firmware, in computer hardware (including the structures disclosed in this specification and their structural equivalents), or in a combination of one or more of these. Embodiments of the subject matter described in this specification may be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible, non-transitory storage medium, for execution by a data processing apparatus or for controlling the operation of a data processing apparatus. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of these. Alternatively or additionally, the program instructions may be encoded on artificially generated propagation signals (e.g., machine-generated electrical, optical, or electromagnetic signals) generated to encode information for transmission to a suitable receiver device for execution by the data processing apparatus.

[0094] The term "data processing device" refers to data processing hardware and encompasses all kinds of devices, apparatuses, and machines used for processing data, including programmable processors, computers, or multiple processors or computers. The device may also be or further include special-purpose logic circuitry, such as FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits). In addition to hardware, the device may optionally include code that creates an execution environment for computer programs, such as code constituting processor firmware, protocol stacks, database management systems, operating systems, or combinations thereof.

[0095] A computer program can also be called or described as a program, software, software application, app, module, software module, script, or code. It can be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages; and it can be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A program may, but does not need to, correspond to a file in a file system. A program may be stored as a portion of a file that holds other programs or data, for example, as one or more scripts stored in a markup language document, as a single file dedicated to the program in question, or as multiple collaborative files, for example, as a file storing one or more modules, subroutines, or code portions. A computer program can be deployed to execute on a single computer or on multiple computers located at a site or distributed across multiple sites and interconnected via a data communication network.

[0096] In this specification, the term "database" is used broadly to refer to any collection of data: the data does not need to be structured in any particular way, or not at all, and it can be stored on storage devices in one or more locations. Therefore, for example, an indexed database may include multiple collections of data, each of which can be organized and accessed differently.

[0097] Similarly, in this specification, the term "engine" is used broadly to refer to a software-based system, subsystem, or process programmed to perform one or more specific functions. Typically, an engine will be implemented as one or more software modules or components installed on one or more computers at one or more locations. In some cases, one or more computers will be dedicated to a particular engine; in other cases, multiple engines may be installed and run on the same one or more computers.

[0098] The processes and logical flows described in this specification can be executed by one or more programmable computers that execute one or more computer programs to perform functions by manipulating input data and generating outputs. These processes and logical flows can also be executed by special-purpose logic circuitry (such as FPGAs or ASICs), or by a combination of special-purpose logic circuitry and one or more programmable computers.

[0099] A computer suitable for executing computer programs can be based on a general-purpose or special-purpose microprocessor, or both, or any other type of central processing unit. Typically, the central processing unit receives instructions and data from read-only memory or random access memory, or both. The basic components of a computer are the central processing unit for implementing or executing instructions and one or more memory devices for storing instructions and data. The central processing unit and memory may be supplemented or incorporated therein by special-purpose logic circuitry. Typically, a computer will also include, or be operatively coupled to, one or more mass storage devices for storing data, such as disks, magneto-optical disks, or optical disks, to receive data from or transfer data to, or both. However, a computer does not need to have such devices. Furthermore, a computer can be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device such as a universal serial bus (USB) flash drive, to name just a few examples.

[0100] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, for example, semiconductor memory devices such as EPROM, EEPROM, and flash memory devices; magnetic disks such as internal hard disks or removable disks; magneto-optical disks; and CD ROMs and DVD-ROMs.

[0101] To provide interaction with the user, embodiments of the subject matter described in this specification can be implemented on a computer having a display device for displaying information to the user, such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, and a keyboard and pointing device, such as a mouse or trackball, through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback, such as visual, auditory, or tactile feedback; and input from the user can be received in any form, including sound, speech, or tactile input. Furthermore, the computer can interact with the user by sending and receiving documents from the device used by the user; for example, by sending a webpage to a web browser in response to a request received from a web browser on the user's device. Additionally, the computer can interact with the user by sending text messages or other forms of messages to a personal device (e.g., a smartphone running a messaging application) and subsequently receiving response messages from the user.

[0102] The data processing apparatus used to implement machine learning models may also include, for example, dedicated hardware accelerator units for processing the general and computationally intensive parts of machine learning training or production, namely inference and workloads.

[0103] Machine learning frameworks can be used to implement and deploy machine learning models, such as the TensorFlow framework, the Microsoft Cognitive Toolkit framework, the Apache Singa framework, or the Apache MXNet framework.

[0104] Embodiments of the subject matter described in this specification can be implemented in a computing system that includes backend components, such as a data server; middleware components, such as an application server; or frontend components, such as a client computer with a graphical user interface, a web browser, or an app through which a user can interact with embodiments of the subject matter described in this specification; or any combination of one or more such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication (e.g., a communication network) of any form or medium. Examples of communication networks include local area networks (LANs) and wide area networks (WANs), such as the Internet.

[0105] A computing system may include clients and servers. Clients and servers are typically geographically separated and usually interact via a communication network. The client-server relationship arises from computer programs running on respective computers and having a client-server relationship with each other. In some embodiments, the server sends data, such as HTML pages, to a user device, for example, to display data to a user interacting with the device acting as a client and to receive user input from that user. Data generated at the user device, such as the results of user interactions, may be received at the server from the device.

[0106] While this specification contains numerous specific details of implementation, these should not be construed as limiting the scope of any invention or the scope that may be claimed, but rather as descriptions of features characteristic of particular embodiments of a particular invention. Certain features described in this specification in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented separately or in any suitable sub-combination in multiple embodiments. Furthermore, although features may be described above as functioning in certain combinations, and even initially claimed in this way, one or more features from a claimed combination may be removed from that combination in some cases, and the claimed combination may be for sub-combinations or variations thereof.

[0107] Similarly, although the operations are described in a specific order in the accompanying drawings and recited in a specific order in the claims, this should not be construed as requiring these operations to be performed in the specific or sequential order shown, or requiring all shown operations to be performed to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the above embodiments should not be construed as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0108] Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the appended claims. For example, the actions recited in the claims may be performed in different orders and still achieve the desired result. As an example, the processes depicted in the drawings do not necessarily require the specific order or sequential order shown to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous.

Claims

1. A method for training a neural network having multiple neural network parameters, and for determining a predictive performance metric for a hardware accelerator with a target hardware configuration on a target application, the method comprising: Maintain a hardware configuration training dataset comprising (i) multiple first hardware configuration training inputs, each specifying a corresponding predetermined feasible hardware configuration; (ii) a corresponding target performance metric for each predetermined feasible hardware configuration; and (iii) multiple second hardware configuration training inputs, each specifying a corresponding predetermined infeasible hardware configuration, wherein the feasibility of the hardware configurations is determined by evaluation against a set of feasibility rules or criteria, and / or by compilation or hardware simulation; and Repeatedly perform the training operations, including: Select a batch of training inputs from the hardware configuration training dataset, the batch of training inputs including (i) one or more first hardware configuration training inputs and (ii) one or more second hardware configuration training inputs; Based on the current values ​​of the plurality of neural network parameters, each first hardware configuration training input is processed using the neural network to determine a corresponding predictive performance metric for each predetermined feasible hardware configuration specified in the one or more first hardware configuration training inputs, the predictive performance metric including one or more of the following: runtime latency of the machine learning model deployed on the hardware accelerator on the target application, or power consumption of the hardware accelerator on the target application. Based on the current values ​​of the plurality of neural network parameters, each second hardware configuration training input is processed using the neural network to determine a corresponding predictive performance metric for each predetermined infeasible hardware configuration specified in the one or more second hardware configuration training inputs; and The gradient of a surrogate objective function with respect to multiple neural network parameters is determined. The surrogate objective function includes (i) a first term measuring the difference between a target performance metric and a predicted performance metric for each predetermined feasible hardware configuration specified in the training inputs of the one or more first hardware configurations, and (ii) a second term measuring the value of a predicted performance metric for each predetermined infeasible hardware configuration specified in the training inputs of the one or more second hardware configurations.

2. The method according to claim 1, wherein, The training operation further includes: Generate one or more third hardware configuration training inputs, each third hardware configuration training input specifying a new hardware configuration; Based on the current values ​​of the plurality of neural network parameters, the neural network is used to process the training input for each third hardware configuration to determine a predictive performance metric for the new hardware configuration; and The proxy objective function includes a third term that measures the value of a predictive performance metric for each new hardware configuration specified in the training input of the one or more third hardware configurations.

3. The method according to any one of claims 1-2, wherein, The proxy objective function minimizes the first term and maximizes the second and third terms.

4. The method according to claim 2, wherein, The training input for generating the third hardware configuration specifying the new hardware configuration includes: An optimizer is applied to the accelerator configuration search space to identify each new hardware configuration that maximizes the predictive performance metric of the new hardware configuration determined by the neural network.

5. The method according to claim 4, wherein, The optimizer is configured to receive a predictive performance metric based on a new hardware configuration determined using the neural network.

6. The method of claim 5, further comprising applying a stopping gradient operator to the optimizer when performing the training operation.

7. The method according to any one of claims 4-6, wherein, The optimizer includes a generative optimizer or an evolutionary optimizer, including an optimizer that runs the Firefly optimization algorithm.

8. The method according to any one of claims 1-2, wherein, The neural network is further configured to process a context vector of a set of attributes for a corresponding target application when processing a first hardware configuration training input, a second hardware configuration training input, or a third hardware configuration training input, wherein the context vector specifies a different set of attributes for each different target application.

9. The method according to any one of claims 1-2, wherein, Each hardware configuration training input in the hardware configuration training dataset includes a plurality of corresponding hardware parameters, which define a predetermined hardware configuration specified by the hardware configuration training input.

10. The method according to claim 9, wherein, The multiple hardware parameters include two or more of the following: the number of processing elements (PEs) along one dimension, the size of the PE memory, the size of the core memory, the size of the instruction memory, the size of the activation memory, the number of cores, the number of compute channels, the size of the parameter memory, or the DRAM bandwidth.

11. The method according to any one of claims 1-2, wherein, The predicted performance metrics for the hardware configuration of a hardware accelerator having the target hardware configuration include one or more of the following: The runtime latency of the machine learning model deployed on the hardware accelerator on the target application, or The power consumption of the hardware accelerator.

12. The method according to claim 11, wherein, The target applications include visual machine learning applications.

13. The method according to any one of claims 1-2, further comprising generating the hardware configuration training dataset, the generation comprising: Receive data defining the accelerator configuration search space; Different hardware configurations are generated from the accelerator configuration search space based on random sampling; and Use a hardware simulator to determine (i) whether each different hardware configuration is feasible or not, and (ii) the corresponding target performance metric for any different hardware configuration that is determined to be feasible.

14. A method for determining a hardware configuration for applying a specific hardware accelerator, the method comprising: Generate multiple different candidate hardware configurations; The corresponding predictive performance metric for each different candidate hardware configuration is determined using a neural network that has been trained by performing the method described in any of the preceding claims; and From the different candidate hardware configurations and based on the corresponding predicted performance metrics, the candidate hardware configuration with the best predicted performance metric is selected as the hardware configuration of the application-specific hardware accelerator.

15. The method according to claim 14, wherein, Generating the plurality of different candidate hardware configurations includes using an optimizer according to any one of claims 4-7 to traverse the accelerator configuration search space to determine each different candidate hardware configuration, each different candidate hardware configuration minimizing a predictive performance metric determined by using the neural network on the target application while conforming to one or more hardware design constraints.

16. The method according to claim 15, wherein, The one or more hardware design constraints include the area constraints of the hardware accelerator.

17. An application-specific hardware accelerator, the hardware configuration of which has a predictive performance metric determined by performing a process comprising the corresponding operations of any one of claims 14-16.

18. A system comprising one or more computers and one or more storage devices storing instructions, wherein when the instructions are executed by the one or more computers, the one or more computers perform the operation of a corresponding method according to any one of claims 1-16.

19. One or more computer storage media storing instructions that, when executed by one or more computers, cause the one or more computers to perform the operation of the corresponding method according to any one of claims 1-16.

Images