Resource Allocation for Tuning Hyperparameters of Large-Scale Deep Learning Workloads

Abstract

The method, computer system, and computer program product may perform hyperparameter tuning. Hyperparameter search input data may be received. A search algorithm may be invoked to generate a set of hyperparameters. A hyperparameter tuning resource allocator may receive physical resource data for a cluster. The hyperparameter tuning resource allocator may determine a required workload and may determine the required workload using the received multiple parallel experiments. The hyperparameter tuning resource allocator may determine whether an option to command workload batch size is selected. The cluster may run the multiple parallel experiments to search for and tune a set of hyperparameters. The hyperparameter tuning resource allocator may manage the cluster by using tensor swapping according to the physical resource data and the required workload.

Description

[Technical field]

[0001] The present invention relates generally to the field of computing, and more specifically to deep learning. [Background technology]

[0002] Deep learning as a technology has helped solve many problems that were previously considered difficult to solve. Newer techniques in deep learning also require the availability of larger datasets on which learning can be performed. The need for larger datasets translates in the field to the use of many accelerators on a single server or multiple servers. The accuracy of deep learning problems is affected by the hyperparameters used to tune the model. If hyperparameters for larger deep learning problems need to be tuned, then a computing cluster needs to be available for that tuning. Summary of the Invention

[0003] According to an exemplary embodiment, a method, computer system, and computer program product may perform hyperparameter tuning. Hyperparameter search input data may be received, including a plurality of parallel experiments and an option to mandate a workload batch size for the hyperparameter search. A search algorithm may be invoked to generate a set of hyperparameters. A hyperparameter tuning resource allocator may receive physical resource data for a cluster. The hyperparameter tuning resource allocator may determine a required workload. The hyperparameter tuning resource allocator may determine a required workload using the received plurality of parallel experiments. The hyperparameter tuning resource allocator may determine whether the option to mandate a workload batch size is selected. The cluster may run the plurality of parallel experiments such that a set of hyperparameters is explored and the set of hyperparameters is tuned. The hyperparameter tuning resource allocator may manage the cluster during the execution in response to the physical resource data and the required workload, and by using tensor swapping.

[0004] These and other objects, features and advantages of the present invention will become apparent from the following detailed description of illustrative embodiments thereof, which is to be read in connection with the accompanying drawings, in which various features of the drawings are not drawn to scale because the figures are for clarity in facilitating understanding of the invention by those skilled in the art in conjunction with the detailed description. [Brief description of the drawings]

[0005] [Figure 1] FIG. 1 illustrates a networked computing environment in accordance with at least one embodiment. [Diagram 2] 1 is an operational flowchart illustrating an end-to-end process of a deep learning workflow in accordance with at least one embodiment. [Diagram 3] 1 is an operational flowchart illustrating a process for hyperparameter tuning in accordance with at least one embodiment. [Figure 4] FIG. 1 illustrates an example of a hyperparameter tuning program and its interaction with a cluster. [Diagram 5] FIG. 1 illustrates an example of a computing node that may be used in the hyperparameter tuning process. [Figure 6] FIG. 1 illustrates an example of a computing node with a virtual accelerator that may be used in the hyperparameter tuning process. [Figure 7] FIG. 2 is a block diagram of internal and external components of the computer and server shown in FIG. 1 according to at least one embodiment. [Figure 8] FIG. 2 is a block diagram of an exemplary cloud computing environment including the computer system shown in FIG. 1, according to one embodiment of the disclosure. [Figure 9] FIG. 9 is a block diagram of functional layers of the exemplary cloud computing environment of FIG. 8 in accordance with one embodiment of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0006] Although detailed embodiments of the claimed structures and methods are disclosed herein, it will be understood that the disclosed embodiments are merely exemplary of the claimed structures and methods that may be embodied in various forms. However, the present invention may be embodied in many different forms and should not be construed as being limited to the exemplary embodiments described herein. Rather, these exemplary embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the present invention to those skilled in the art. In this description, details of well-known features and techniques may be omitted so as not to unnecessarily obscure the presented embodiments.

[0007] Exemplary embodiments described below provide systems, methods, and program products for hyperparameter tuning. Embodiments of the present invention are capable of improving the field of deep learning by enabling hyperparameter exploration of large-scale deep learning workloads utilizing limited computing resources, thereby eliminating the need to own or have access to large accelerator clusters.

[0008] Specifically, data scientists and researchers generally do not have access to large clusters of accelerators, e.g., large clusters of graphic processing units (GPUs). Most deep learning experiments are conducted in shared cloud environments, where large clusters of accelerators are either unavailable or prohibitively expensive for many data scientists and researchers.

[0009] It would therefore be advantageous to provide a way for hyperparameter searches to be performed in an economically feasible manner, particularly by allowing data scientists and researchers in academia, to run hyperparameter searches in cloud environments using the regular accelerator cluster resources available to them.

[0010] Exemplary embodiments described below enable hyperparameter exploration and tuning for large deep learning workloads with limited computing resources and enable data scientists and researchers, such as in academia, to perform hyperparameter exploration and tuning in an economically feasible manner in cloud environments. Exemplary embodiments enable avoiding infrastructure costs associated with acquiring or accessing large accelerator clusters. Exemplary embodiments enable improved resource management, providing dynamic adjustment of accelerator usage in response to hyperparameter exploration job requirements. Exemplary embodiments enable organizations to reduce resources required for hyperparameter exploration and tuning during the model development phase, allowing large accelerator clusters to be dedicated for production workloads.

[0011] Referring to FIG. 1, an exemplary networked computing environment 100 is shown according to one embodiment. The networked computing environment 100 may include a computer 102 having a processor 104 and a data storage device 106 enabled to execute a software program 108 and a hyperparameter tuning program 110a. The networked computing environment 100 may also include a server 112 enabled to execute a hyperparameter tuning program 110b, which may interact with a database 114 and a communications network 116. The networked computing environment 100 may include multiple computers 102 and servers 112, only one of which is shown. The communications network 116 may include various types of communications networks, such as, for example, a wide area network (WAN), a local area network (LAN), a telecommunications network, a wireless network, a public switched network, or a satellite network, or combinations thereof. It should be understood that FIG. 1 provides only an example of one implementation and is not intended to imply any limitations with respect to the environments in which various embodiments may be implemented. Many modifications to the depicted environment may be made based on design and implementation requirements.

[0012] The client computer 102 may communicate with the server computer 112 via a communication network 116. The communication network 116 may include connections such as, for example, wired, wireless communication links, or fiber optic cables. As described with reference to FIG. 7, the server computer 112 may include internal components 902a and external components 904a, respectively, and the client computer 102 may include internal components 902b and external components 904b, respectively. The server computer 112 may also operate in a cloud computing service model, such as, for example, Software as a Service (SaaS), Platform as a Service (PaaS), or Infrastructure as a Service (IaaS). The server 112 may also be deployed in a cloud computing deployment model, such as, for example, a private cloud, a community cloud, a public cloud, or a hybrid cloud. The client computer 102 may be, for example, a mobile device, a phone, a personal digital assistant, a netbook, a laptop computer, a tablet computer, a desktop computer, or any type of computing device capable of executing a program, accessing a network, and accessing a database 114. According to various implementations of embodiments of the present invention, the hyperparameter tuning programs 110a, 110b may interact with the database 114, which may be embodied in, for example, but not limited to, a computer / mobile device 102, a networked server 112, or various storage devices such as a cloud storage service. According to various implementations of embodiments of the present invention, the hyperparameter tuning programs 110a, 110b may interact with a cluster as shown in FIG. 4. The cluster may include computing nodes 500 as shown in FIG. 5, or computing nodes 600 with virtual accelerators as shown in FIG. 6, or both.

[0013] According to an embodiment of the invention, a user using a client computer 102 or a server computer 112 (respectively) may perform hyperparameter tuning of hyperparameters of a deep learning process using a hyperparameter tuning program 110a, 110b. The hyperparameter tuning method is described in more detail below with respect to FIGS. 2-9. As also described in more detail below and shown in particular in FIG. 4, the hyperparameter tuning program 110a, 110b may include a job input generator, a search algorithm, a hyperparameter tuning resource allocator, and a training controller. Some or all of the job input generator, the search algorithm, the hyperparameter tuning resource allocator, and the training controller may be provided in the form of software. The hyperparameter tuning program 110a, 110b may operate in conjunction with a cluster.

[0014] Referring to FIG. 2, an operational flowchart illustrating an exemplary end-to-end deep learning workflow process 200 in which hyperparameter tuning programs 110a and 110b perform actions is shown in accordance with at least one embodiment.

[0015] The development phase of a deep learning model may include the subphases of advanced data processing, model selection, and hyperparameter tuning. Developers or researchers involved in this development phase typically use cloud infrastructure, which has limited resources. Large dedicated clusters are typically only available for production environments. Provisioning resources on the cloud may involve data scientists, developers, and / or DevOps to execute this development phase with the help of resource allocators and orchestrators, which work with a centralized infrastructure on the cloud.

[0016] As shown in FIG. 2, an end-to-end deep learning workflow process 200 may include a step of training a model to eventually recognize patterns or data presented. In step 202, a training task may be defined. This step 202 may be performed, for example, by identifying a specific business problem, obtaining subject matter expertise, defining a unit of analysis and a prediction target, prioritizing model criteria, considering risk and success criteria, or identifying a desired search scope, or a combination thereof. One training task may be to develop a model that may receive as input pictures of animals and recognize and correctly identify the type of animal in various pictures. Another training task may be to develop a model that may receive as input audio sounds or recordings, perform speech-to-text conversion, and recognize the physical or age characteristics of a speaker based on the speaker's voice. Another training task may be to use image classification to identify product defects at various stages of a production line.

[0017] In step 204, training data may be provided. For example, training data may be uploaded. The training data may be raw data obtained from a data source. In step 206, advanced data may be processed. This processing of advanced data may also be referred to as data pre-processing. For example, this processing may address missing data, outliers, or data with a long tail. Data with a long tail may be corrected by filtering the raw data. In this step 206, data may be merged. Following the above example of image classification of a manufacturing line, the training data may include photos of defective and non-defective products. This step 206 may alternatively or additionally include data enrichment, data standardization, and data format conversion to provide compatibility with the training algorithm.

[0018] In step 208, a base model may be chosen from various candidate models that have been used previously and have been successful in recognition. In step 210, hyperparameters or hyperparameter configurations may be tuned. An exemplary process of substeps for performing step 210 is described below in FIG. 3 as a hyperparameter tuning process. In step 212, a model may be trained. This training may be performed using a training data set and may generate model artifacts. The model training in step 212 is performed using a consistent set of hyperparameters tuned in step 210, e.g., no hyperparameters are changed during step 212. After the model is trained, the deep learning model or neural network may then be ready to perform inferences or to recognize patterns or data in step 214. For example, depending on the possible embodiments described above, the trained model may then be able to identify the type of animal shown in a photo, may be able to perform text conversion or natural language processing from a received audio signal, or may be able to recognize features of a person speaking. The trained deep learning model may be run on a test data set to generate inferences. The trained deep learning model may be able to recognize whether a picture shows a cat, a dog, an elephant, a camel, etc. The steps of the end-to-end deep learning workflow process 200 may be repeated or performed iteratively, as indicated by the return arrow after the model training step 212 back to the beginning of the process, so that, for example, steps 204 through 212 may be performed again.

[0019] The end-to-end deep learning workflow process 200 may use an iterative hyperparameter exploration and tuning phase. Hyperparameter tuning may require multiple training runs to try different sets of hyperparameters to be explored. Thus, in a typical scenario, the amount of resources required for hyperparameter exploration may approach, equal, or exceed the amount of resources required for training a model in a production environment. The need for such large resources may create resource scarcity, as using a large cluster for hyperparameter exploration experiments reduces the resources available to data scientists for training models. For large-scale deep learning workloads, the problem size and the amount of resources required grow exponentially. However, hyperparameter exploration is performed in the research and development phase. While model training and deployment have performance bounds, hyperparameter exploration does not. Therefore, in hyperparameter exploration, resolving resource limitations is more important than performance. Therefore, it would be advantageous to establish a hyperparameter exploration / tuning of step 210 for large-scale workloads that can be performed when computational resources are limited.

[0020] 3, an operational flowchart is shown illustrating an exemplary hyperparameter tuning 210 used by the hyperparameter tuning programs 110a, 110b, in accordance with at least one embodiment. The flowchart shown in FIG. 3 fits the hyperparameter tuning 210 of the end-to-end deep learning workflow process 200 shown in FIG.

[0021] Hyperparameter tuning in deep learning may involve the selection of an optimal set of hyperparameters for a learning algorithm. This selection may involve a search job in the configuration space defined according to the algorithm. The hyperparameter search job may involve the use of various search algorithms that help narrow down the search space. Workload details and the search range of hyperparameters may be provided as input to the search algorithm. The search algorithm may generate various sets of hyperparameters for experimentation based on the algorithm used. The experiments may be conducted on a cluster, e.g., on compute nodes, over multiple training runs.

[0022] According to at least some embodiments of the present invention, a hyperparameter tuning resource allocator (HTRA) is implemented as an additional module. HTRA understands the resource requirements of the exploration jobs and simulates large-scale workloads by seamlessly managing the use of physical and virtual resources. HTRA uses the workload details and resource requirements to automatically manage computing resources for the requested number of exploration experiments. HTRA can manage resources when batch size scaling is allowed. HTRA can also manage resources when batch size scaling is not allowed. When batch size scaling is not allowed, it is expected that the same batch size will be used for each exploration experiment.

[0023] In step 302 of hyperparameter tuning 210, hyperparameter search input data is received as part of a hyperparameter search job. The hyperparameter search input data may include details of the workload, the search scope, resources required for tuning (e.g., multiple accelerators required to run tuning experiments), multiple parallel experiments for tuning, an option to command the workload batch size for the hyperparameter search, and the workload batch size, if commanded. The hyperparameter search input data may include which hyperparameters are to be used. For example, momentum and weight decay may be selected as the hyperparameters to be tuned. The workload batch size may also be considered a hyperparameter. The hyperparameter search information may be obtained via a graphical user interface (GUI) that prompts a user to enter the hyperparameter search input data.

[0024] For embodiments involving deep learning for image classification, batch size may be the number of images that can be processed in a single batch. For example, in a data set of a thousand images, it may be possible to process 10 images at a time. Thus, the batch size in this example would be 10. If only one image could be processed at a time, the batch size would be 1. If the batch size were larger, processing could process the entire data set more quickly if sufficient computing resources were available. However, to address larger batch sizes, more computing resources are typically required, which are often not available.

[0025] In step 304, the hyperparameter search job calls a search algorithm that generates a set of hyperparameters for experiments based on the number of parallel experiments requested. For example, a search algorithm Hyperband, a random search algorithm, a grid search algorithm, a Bayesian optimization algorithm, a gradient-based optimization algorithm, an evolutionary optimization algorithm, a population-based algorithm, a radial basis function algorithm, or a spectral algorithm, or a combination thereof, may be used in step 304 to generate a set of hyperparameters for experiments. The hyperparameters obtained in step 302, such as momentum and weight decay and batch size, may be fed to the search algorithm, which may generate test numbers or test values ​​for these hyperparameters, which are tested and tuned in subsequent steps. A deep learning model may have many hyperparameters, for example, 15-20 hyperparameters, and a choice is made as to which values ​​are generated, which are tuned, and used in the deep learning model once the model is trained.

[0026] In step 306, the HTRA receives physical resource data for the cluster, such as available computing nodes to run the data and perform the experiments. The physical resource data may include the number of accelerators in the cluster. The accelerators may be, for example, graphics processing units (GPUs), field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), or other accelerators. The physical resource data may also include the amount of memory in the accelerators, such as, for example, the memory held by each graphics processing unit (GPU memory). The physical resource data may also include the number of central processing units (CPUs) in the cluster and the amount of memory held by the CPU or held by each CPU (CPU memory). The physical resource data may also include the amount of system memory accessible by the CPU or accessible by each CPU. Knowing or having physical resource data available in the cluster helps the HTRA determine the amount of scaling required and how to allocate the scaling on the GPUs.

[0027] In step 308, the HTRA determines or understands the resource requirements of the search job and determines the workload required to perform the hyperparameter search and tuning. These resource requirements of the search job may be in the hyperparameter search input data received in step 302. The HTRA may perform one or more calculations to determine the required workload used to perform the hyperparameter search and tuning.

[0028] In step 310, the HTRA determines whether a workload batch size has been commanded. The HTRA may make this determination in step 310 using the hyperparameter search input data received in step 302. The hyperparameter search input data received in step 302 may include a binary variable indicating, positively or negatively, whether a workload batch size has been commanded. If a workload batch size has been commanded, the hyperparameter search input data also includes the workload batch size provided as a variable. For example, the hyperparameter search input data may include a command for a workload batch size of 1.

[0029] In step 312, the HTRA schedules experiments on the cluster using a training controller and initiates experiments on the cluster. The training controller has knowledge of run-time resources and is configured to schedule jobs on available resources. The training controller can be software configured to be compatible with a particular computing cluster.

[0030] In step 314, experiments are run on the cluster. A training experiment or multiple training experiments may be run on the computing nodes of the cluster. Data including hyperparameters may be divided into smaller groups and loaded into accelerator cores in the cluster for computation with a set of possible hyperparameter values. The cluster runs the experiments using tensor swapping. For example, the cluster uses tensor swapping to manage memory requirements. Thousands of tests may be run using various hyperparameters to identify optimal hyperparameter values. A particular group of hyperparameters may be tested on the computing cluster and inference time and accuracy may be observed. Various groups of hyperparameters may be tested by experimentation in the cluster until an optimal or most efficient set of hyperparameter values ​​is determined. The efficient set may then be used for model training step 212 shown in FIG. 2 and recognition step 214 shown in FIG. 2. Recognition step 214 may be performed in deep learning when information such as an image or audio signal is fed into a deep learning model and the deep learning model presents an output of analyzing the image or audio signal.

[0031] In step 316, HTRA manages the cluster. HTRA manages the use of the cluster's physical and virtual resources to simulate a large workload. The cluster uses tensor swapping to manage memory requirements when running experiments, and HTRA manages the tensor swapping. During or upon completion of a set of experiments, the cluster may return training outputs or metrics to the training controller or HTRA, or both, which may determine, initiate, or control the next set of experiments, or a combination thereof. HTRA may use the training controller to initiate the next set of experiments on the cluster.

[0032] Using the inputs provided, HTRA will automatically manage resources for the requested number of experiments.

[0033] 4 illustrates an example of a hyperparameter tuning program 110a, 110b and its interaction with a cluster 408 for running a hyperparameter tuning experiment on the cluster 408. The cluster 408 is a computing cluster that may include an accelerator, as described below with respect to FIGS. 5 and 6. The hyperparameter tuning program 110a, 110b may include subcomponents such as a hyperparameter job input generator, a search algorithm 402, a hyperparameter tuning resource allocator (HTRA) 404, and a training controller 406.

[0034] The hyperparameter job input generator may use a graphical user interface (HI GUI) 400 to obtain input for the hyperparameter job search to be performed, i.e., perform step 302. The HI GUI 400 may prompt a developer or researcher to enter hyperparameter job search input data, such as details of the workload, the search scope, resources required for tuning (e.g., accelerators required), multiple parallel experiments, an option to dictate the workload batch size for the hyperparameter search, and the workload batch size if selected. The specific hyperparameters to be tested may be entered into the HI GUI 400 by a user.

[0035] Upon receiving the hyperparameter search input data by the HI GUI 400, a search algorithm 402 may be invoked such that step 304 may be performed, i.e., a set of hyperparameters, i.e., a set of hyperparameter values, may be generated based at least in part on the number of parallel experiments requested as part of the hyperparameter job search in the hyperparameter job search input data. Various examples of the search algorithm 402 are described above.

[0036] The HTRA 404 performs step 306 by receiving physical resource data for the cluster 408 that may be used to run experiments for hyperparameter tuning. The HTRA 404 may have a data connection to the cluster 408 for receiving the physical resource data, e.g., a physical connection or a wireless connection to the cluster 408. The HTRA 404 may also perform step 308 to determine the required workload. The HTRA 404 may use the hyperparameter search input data received in step 302 to determine the required workload in step 308. The HTRA 404 may also use the hyperparameter search input data received in step 302 to determine whether a workload batch size is commanded in step 310. The HTRA also uses the training controller 406 to schedule experiments on the cluster 408 and initiate the experiments on the cluster 408. During or after experiments are being performed on the cluster 408, the cluster may return training outputs / metrics to the HTRA 404 and / or the training controller 406 so that further experiments in hyperparameter tuning can be determined and controlled.

[0037] FIG. 5 illustrates an example of a cluster 408 including a computing node 500 that may be useful for performing experiments in at least some embodiments of the method according to the present invention. The computing node 500 may include a system memory 502, a central processing unit (CPU) 504, and multiple accelerators. In this exemplary embodiment, the multiple accelerators are graphics processing units (GPUs). Although FIG. 5 illustrates a first GPU 506a and a second GPU 506b, in some embodiments, the computing node may include three, four, five, six, or eight GPUs. The first GPU 506a and the second GPU 506b may be general processing graphics processing units (GPGPUs). The first GPU 506a and the second GPU 506b may be NVIDIA® Tesla® V100 Tensor Core GPUs. (NVIDIA and Tesla and all NVIDIA based trademarks and logos are trademarks or registered trademarks of Nvidia, Inc.) In other embodiments, other accelerators may be provided instead of a GPU.

[0038] A high-speed memory bus 508 may connect the CPU 504 to the system memory 502. The high-speed memory bus 508 may pass information at a rate of 170 GB / s or more. A mesh multi-way interface 510c may connect the first and second GPUs 506a,b to each other. The mesh multi-way interface 510a may connect the first GPU 506a to the CPU 504, and the mesh multi-way interface 510b may connect the second GPU 506b to the CPU 504. The computing node 500 may generate tensor swapping, which may allow the present systems and methods to scale beyond normal computing capabilities. The mesh multi-way interfaces 510a-c may be ingrained into the silicon of the processor.

[0039] With regard to tensor swapping, tensors generated during computation of an experiment on a GPU may be temporarily passed or transferred from the accelerator, e.g., the first and second GPUs 506a, 506b, to the CPU 504 and by the CPU 504 to the system memory 502, so that the accelerator is not overloaded with data and freezes. Tensor swapping allows one or more new tensors to be generated by using the accelerator's resources to perform additional computations and experiments. After an experiment is completed or while part of the experiment is still in progress, the tensors temporarily stored in the CPU 504 and / or system memory 502 may be passed or transferred backloaded to the accelerator, e.g., the first and second GPUs 506a,b, and to the hyperparameters being explored within the accelerator, if required for a training job running on the accelerator.

[0040] The mesh multi-directional interfaces 510a-c may be bidirectional interfaces that may include eight differential pairs in each direction, for a total of 32 wires. The pairs may be DC-coupled and may use 85 Ω differential termination with an embedded clock. To facilitate routing, the mesh multi-directional interfaces 510a-c may support lane reversal and lane polarity, meaning that the physical lane order and their polarity between two devices may be reversed. The mesh multi-directional interfaces 510a-c may each be an NVLink® 1.0 or NVLink® 2.0 connection. (NVLink is a registered trademark of Nvidia, Inc.) The mesh multi-directional interfaces 510a-c may each include four NVLink® connections. In some embodiments, the cluster 408 may include four accelerators, such as four GPUs, and two CPUs, where the four accelerators are fully connected to each other and the fourth link goes to one of the two CPUs. In some embodiments, the accelerators and the CPU 504 may access each other's memory in a flat address space, for example for direct loads and stores. A flat address space may be supported via a new address translation service. Additionally, in some embodiments, the CPU 504 and accelerators may natively support atomic operations. A flat address space may enable cache coherence and allow the CPU 504 to efficiently cache accelerator memory, thereby significantly improving latency and therefore performance. In a cluster with eight accelerators on a single baseboard, a switch may be provided that connects each of the eight accelerators to all seven other accelerators on the same baseboard. For example, a cluster may include eight GPUs with the described connectivity.

[0041] In response to HTRA404 identifying that batch size scaling is possible, HTRA404 simulates a large workload by dynamically scaling the batch size according to the workload requirements. If the option to dictate the workload batch size for hyperparameter search is not selected, batch size scaling is possible. HTRA404 simulates a large workload by dynamically scaling the batch size according to the workload batch size provided and the resource requirements.

[0042] As part of this dynamic scaling, HTRA 404 identifies a cumulative batch size requested for the exploratory experiments. The cumulative batch size is the workload batch size multiplied by the number of accelerators requested. HTRA 404 has details of the number of physical accelerators in the cluster 408 and the available CPU memory and accelerator memory from step 306. HTRA 404 also has the number of experiments requested. The cumulative batch size is distributed among the available physical accelerators in the cluster 408 for the requested number of experiments.

[0043] Thus, in this embodiment with no batch size directive and dynamic scaling, each physical accelerator would need to accommodate larger batch sizes. The memory and communication requirements resulting from batch size scaling are seamlessly accommodated by HTRA 404 using tensor swapping between the accelerators (e.g., any GPUs, such as the first GPU 506a and the second GPU 506b) and the CPU 504. The batch size scaling mechanism of HTRA 404 may be based on the available cumulative accelerator memory and CPU memory.

[0044] If batch size scaling is not available, e.g., if a workload batch size is commanded in the hyperparameter search input data, HTRA 404 may identify that and manage resources accordingly by creating and managing virtual accelerators that can emulate physical accelerators. In this manner, large-scale simulations of large workloads may be performed. The commanded workload batch size is used for the hyperparameter search. For example, the hyperparameter search input data from step 302 may include a command for a workload batch size of 1.

[0045] In this embodiment with workload batch size directive, HTRA 404 first allocates virtual accelerators according to the number of accelerators requested for the total number of parallel experiments requested for the hyperparameter search. The allocation also takes into account the physical accelerators, accelerator memory, and available CPU memory available on each computing node in the cluster 408. The creation of virtual accelerators depends on the cumulative accelerator memory and CPU memory available.

[0046] Certain search algorithms, such as Hyperband, use techniques such as Successive Halving, where the number of parallel experiments is reduced at each step of the hyperparameter search. The reduction is done by selecting the most likely configuration and discarding other settings. In such cases, HTRA404 reallocates virtual accelerators as necessary at each step to take advantage of the additional resources available. The reallocation is done so that the remaining virtual accelerators required in further steps have more resources available, such as memory and floating-point operations per second (flops). This reallocation implicitly speeds up the execution of the experiments at that step. Communication patterns may also be changed in response to the reallocation.

[0047] In this manner, the use of virtual accelerators such as virtual GPUs may allow available accelerator resources to be allocated efficiently, and physical resource utilization of the cluster 408 may be maximized by dynamically adjusting resources according to hyperparameter search job requirements.

[0048] The HTRA 404 may manage the creation of virtual accelerators and the memory requirements of the virtual accelerators by using tensor swapping between multiple accelerators and the CPU 504.

[0049] An example of HTRA 404 allocating virtual accelerators within a physical accelerator is shown in FIG. 6. FIG. 6 illustrates how HTRA 404 operates when a workload batch size is dictated in the hyperparameter search input data. FIG. 6 illustrates a compute node 600 in which a CPU 504 has CPU memory 620 and virtual GPUs 612a-i are created within the illustrated first and second GPUs 506a,b. The virtual GPUs 612a-i may access the CPU memory 620 for tensor swapping. A particular virtual GPU, for example, virtual GPU 612a, may consistently access the same address space in the CPU memory 620 or the same address space in the system memory 502 for tensor swapping. FIG. 6 also illustrates inter-GPU and intra-GPU communication patterns.

[0050] Hyperparameter training performed without the methods and systems described herein is performed without a hyperparameter tuning resource allocator and without the ability to scale up with limited computing resources. In this example, without the ability to scale up and where the cluster has 8 GPUs, a hyperparameter search job for a workload requiring 8 GPUs with batch size 1 can only run one training experiment. This tuning is limited by the available physical resources.

[0051] This example is in contrast to the example systems and methods described herein for hyperparameter exploration / tuning of large-scale deep learning workloads with limited resources. In the invention described herein, a hyperparameter tuning resource allocator, e.g., HTRA404, is used. In this example, the cluster 408 only has 8 GPUs, but the hyperparameter exploration job is allowed to run 8 experiments in parallel with a batch size of 1. In a typical scenario, this operation would require 64 GPUs to operate, with 8 GPUs used per experiment. HTRA404 is able to simulate large-scale workloads using limited resources. Based on whether the option to use the workload batch size for hyperparameter exploration is selected or not, HTRA404 can take one of two approaches to seamlessly perform the simulation of large-scale workloads.

[0052] If the option to dictate workload batch size is not selected, HTRA 404 may use a batch size scaling mechanism to seamlessly manage the floating point operations per second (flops) and memory requirements of the physical accelerator.

[0053] In the given example where there is no workload batch size directive and the workload requirement is to run eight experiments in parallel, each experiment can be run on one physical accelerator by seamlessly scaling to a batch size of eight experiments per accelerator, e.g., eight experiments per GPU. This dynamic scaling allows eight experiments to be run in parallel even with limited resources. The present system and method allows a tuning operation to be run on as few as eight accelerators when 64 accelerators were originally required.

[0054] Additionally, when a search algorithm such as Hyperband is used that may cut the number of experiments in half in successive steps, HTRA404 may reallocate resources after each run. In this example, where the number of experiments is cut in half in successive steps, the next step of running four experiments in parallel may have each of the eight accelerators use a batch size of four. This reallocation speeds up the execution and maximizes resource utilization.

[0055] If the option to command workload batch size is selected and a batch size of 1 is commanded, HTRA 404 may create a virtual accelerator to address the resource requirements. Figure 6 illustrates a cluster 408 that includes a computing node 600 on which virtual GPUs 612a-i are created by HTRA 404. Figure 6 illustrates that computing node 600 has many components in common with computing node 500 shown in Figure 5, and thus some common reference numbers are used in both Figures 5 and 6.

[0056] In a given example where a workload batch size is commanded, HTRA 404 may create 64 virtual accelerators, e.g., 64 GPUs, with 8 virtual GPUs on each physical GPU, each virtual GPU handling batch size 1. Creating virtual accelerators such as virtual GPUs 612a-i in this manner allows 8 experiments to be run in parallel, even with limited resources. A tuning operation that originally required 64 physical accelerators can now be run on 8 physical accelerators. The memory, floating point operations per second (flops), and communication requirements of virtual GPUs 612a-i are all seamlessly handled by HTRA 404. The memory 620 of virtual GPUs 612a-i and CPU 504 are shown in FIG. 6.

[0057] Moreover, in this alternative example, if a search algorithm such as Hyperband is used where the number of experiments is reduced by half in successive steps, HTRA404 reallocates virtual accelerators to utilize all available resources. In this example, the next step of running 4 experiments in parallel will result in 32 virtual accelerators being created, each dealing with a batch size of 1 as directed.

[0058] HTRA 404 may manage intra-accelerator communications, i.e., communications between virtual accelerators within a particular physical accelerator. For example, HTRA 404 may manage intra-GPU communications 614a between virtual GPUs 612a and 612b in a first GPU 506a (the first GPU 506a is a physical GPU), intra-GPU communications 614b between virtual GPUs 612d and 612e in a first GPU 506a, intra-GPU communications 614c between virtual GPUs 612f and 612g in a second GPU 506b (the second GPU 506b is a physical GPU), and intra-GPU communications 614d between virtual GPUs 612h and 612i in a second GPU 506b. HTRA 404 may also manage inter-accelerator communications. For example, HTRA 404 may manage inter-GPU communication 616 that occurs between a virtual GPU of one physical GPU and a virtual GPU of another physical GPU. This inter-GPU communication 616 may occur between a virtual GPU 612c of a first GPU 506a and a virtual GPU 612f of a second GPU 506b. The inter-GPU communication 616 may occur through a mesh multi-way interface, such as mesh multi-way interface 510c shown in FIG. 5. HTRA 404 may manage communication between virtual accelerators, such as virtual GPUs 612a-i, using communication techniques such as NCCL (NVIDIA® Collective Communications Library) or MPI (message passing interface).

[0059] If the option to dictate workload batch size is not selected, HTRA 404 can still manage the experiment in a manner similar to that described for the embodiment in which the option to dictate workload batch size is selected, by creating virtual accelerators depending on the size of the problem to be solved.

[0060] It will be appreciated that Figures 2-6 provide only examples of particular embodiments and are not intended to impose any limitations on how various embodiments may be implemented. Many variations may be made to the illustrated embodiments based on design and implementation requirements.

[0061] As described in the above embodiments, the hyper-parameter tuning programs 110a, 110b may improve the capabilities of a computer by scaling up limited computing resources to perform hyper-parameter tuning for large workloads.

[0062] Figure 7 is a block diagram 900 of internal and external components of the computer shown in Figure 1 in accordance with an exemplary embodiment of the invention. It should be understood that Figure 7 is intended as only an example of one implementation and is not intended to impose any limitations with respect to the environments in which different embodiments may be implemented. Many modifications to the depicted environment may be made based on design and implementation requirements.

[0063] The data processing systems 902, 904 represent any electronic device capable of executing machine-readable program instructions. The data processing systems 902, 904 may represent a smart phone, a computer system, a PDA, or other electronic device. Examples of computing systems, environments, or configurations, or combinations thereof, that may be represented by the data processing systems 902, 904 include, but are not limited to, personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, multiprocessor systems, microprocessor-based systems, network PCs, minicomputer systems, and distributed cloud computing environments that include any of the above systems or devices.

[0064] The user client computer 102 and the network server 112 may include respective sets of internal components 902a,b and external components 904a,b shown in Figure 7. Each of the sets of internal components 902a,b includes one or more processors 906, one or more computer-readable RAMs 908, and one or more computer-readable ROMs 910 on one or more buses 912, one or more operating systems 914, and one or more computer-readable tangible storage devices 916. The one or more operating systems 914, software programs 108, and hyperparameter tuning program 110a in the client computer 102, and hyperparameter tuning program 110b in the network server 112 may be stored in one or more computer-readable tangible storage devices 916 for execution by the one or more processors 906 via one or more RAMs 908 (typically including cache memory). 6, each of the computer-readable tangible storage devices 916 is an internal hard drive magnetic disk storage device, or alternatively, each of the computer-readable tangible storage devices 916 is a semiconductor storage device, such as ROM 910, EPROM, flash memory, or any other computer-readable tangible storage device capable of storing computer programs and digital information.

[0065] Each set of internal components 902a,b also includes an R / W drive or interface 918 for reading from and writing to one or more portable computer-readable tangible storage devices 920, such as, for example, a CD-ROM, a DVD, a memory stick, a magnetic tape, a magnetic disk, an optical disk, or a semiconductor storage device. Software programs, such as software program 108 and hyper-parameter tuning programs 110a, 110b, may be stored in one or more of the respective portable computer-readable tangible storage devices 920 and read via the respective R / W drive or interface 918 and loaded onto the respective hard drive 916.

[0066] Each set of internal components 902a,b may also include a network adapter (or switch port card) or interface 922, e.g., a TCP / IP adapter card, a wireless wi-fi interface card, or a 3G or 4G wireless interface card, or other wired or wireless communication link. The software program 108 and the hyperparameter tuning program 110a in the client computer 102, and the hyperparameter tuning program 110b in the network server computer 112 may be downloaded from an external computer (e.g., a server) via a network (e.g., the Internet, a local area network, or other wide area network) and the respective network adapter or interface 922. From the network adapter (or switch port adapter) or interface 922, the software program 108 and the hyperparameter tuning program 110a in the client computer 102, and the hyperparameter tuning program 110b in the network server computer 112 are loaded onto the respective hard drives 916. The network may include copper wire, fiber optics, wireless transmissions, routers, firewalls, switches, gateway computers, or edge servers, or any combination thereof.

[0067] Each of the set of external components 904a,b can include a computer display monitor 924, a keyboard 926, and a computer mouse 928. The external components 904a,b can also include touch screens, virtual keyboards, touch pads, pointing devices, and other human interface devices. Each of the set of internal components 902a,b also includes a device driver 930 for interfacing to the computer display monitor 924, the keyboard 926, and the computer mouse 928. The device driver 930, the R / W drive or interface 918, and the network adapter or interface 922 include hardware and software (stored in the storage device 916 or ROM 910, or both).

[0068] The present invention may be a system, method, or computer program product, or combination thereof, at any possible level of technical detail integration. The computer program product may include a computer-readable storage medium (or media) having computer-readable program instructions thereon for causing a processor to carry out aspects of the present invention.

[0069] A computer readable storage medium may be any tangible device capable of holding and storing instructions for use by an instruction execution device. A computer readable storage medium may be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. A non-exhaustive list of more specific examples of computer readable storage media includes portable computer diskettes, hard disks, random access memory (RAM), read only memory (ROM), erasable programmable read only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disk read only memory (CD-ROM), digital versatile disk (DVD), memory stick, floppy disk, mechanically encoded devices such as punch cards or groove ridge structures having instructions recorded thereon, and any suitable combination thereof. A computer-readable storage medium, as used herein, should not be construed to be a transitory signal per se, such as, for example, radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission medium (e.g., light pulses passing through a fiber optic cable), or electrical signals transmitted over a wire.

[0070] The computer readable program instructions described herein may be downloaded from a computer readable storage medium to the respective computing / processing device or to an external computer or storage device over a network, such as, for example, the Internet, a local area network, a wide area network, or a wireless network, or a combination thereof. The network may include copper transmission cables, optical transmission fiber, wireless transmission, routers, firewalls, switches, gateway computers, or edge servers, or a combination thereof. A network adapter card or network interface of each computing / processing device receives the computer readable program instructions from the network and transfers the computer readable program instructions for storage in a computer readable storage medium within the respective computing / processing device.

[0071] The computer readable program instructions for carrying out the operations of the present invention may be source or object code written in any combination of one or more programming languages, including assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state setting data, configuration data for an integrated circuit, or object oriented programming languages ​​such as Smalltalk, C++, and procedural programming languages ​​such as the "C" programming language or similar programming languages. The computer readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the last scenario, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or wide area network (WAN), or a connection may be made to an external computer (e.g., via the Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example, a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be personalized by utilizing state information of the computer readable program instructions to execute the computer readable program instructions to perform aspects of the present invention.

[0072] Aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer readable program instructions.

[0073] These computer readable program instructions may be provided to a processor of a computer or other programmable data processing apparatus to produce a machine such that, when the instructions are executed via the processor of the computer or other programmable data processing apparatus, means are generated for implementing the functions / acts specified in one or more blocks of the flowcharts and / or block diagrams. These computer readable program instructions may also be stored on a computer readable storage medium capable of directing a computer, programmable data processing apparatus, or other device, or combination thereof, to function in a particular manner, such that the computer readable storage medium on which the instructions are stored constitutes an article of manufacture including instructions that implement aspects of the functions / acts specified in one or more blocks of the flowcharts and / or block diagrams.

[0074] Furthermore, computer-readable program instructions may be loaded into a computer, other programmable data processing apparatus, or other device and caused to perform a series of operational steps on the computer, other programmable apparatus, or other device to generate a computer-implemented process that, when executed on the computer, other programmable apparatus, or other device, implements the functions / acts specified in one or more blocks of the flowcharts and / or block diagrams.

[0075] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in the flowcharts or block diagrams may represent a module, segment, or part of instructions that includes one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions described in the blocks may occur out of the order shown. For example, depending on the functionality involved, two blocks shown in succession may in fact be realized as one step, may be executed simultaneously, substantially simultaneously, partially or completely overlapping in time, or the blocks may be executed in reverse order, if necessary. It will also be noted that each block of the block diagrams and / or flowchart illustrations, and combinations of blocks in the block diagrams and / or flowchart illustrations, may be implemented by a dedicated hardware-based system that performs the specified functions or acts or executes a combination of dedicated hardware and computer instructions.

[0076] Although this disclosure includes detailed descriptions of cloud computing, it is understood that implementation of the teachings recited herein is not limited to a cloud computing environment. Rather, embodiments of the invention may be implemented in conjunction with any other type of computing environment now known or later developed.

[0077] Cloud computing is a model of service delivery for enabling convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, network bandwidth, servers, processing, memory, storage, applications, virtual machines, and services) that can be rapidly provisioned and released with minimal administrative effort or interaction with a service provider. The cloud model may include at least five characteristics, at least three service models, and at least four deployment models.

[0078] Its features are as follows.

[0079] On-Demand Self-Service: Cloud consumers can unilaterally provision computing capacity, such as server time and network storage, automatically as they need it, without any human interaction with the provider of the service.

[0080] Broad Network Access: Capabilities are available over the network and are accessed through standard mechanisms that facilitate use by heterogeneous thin or thick client platforms (e.g., cell phones, laptops, and PDAs).

[0081] Resource Pooling: Pooling provider computing resources to serve multiple consumers using a multi-tenant model where various physical and virtual resources are dynamically allocated and reallocated according to demand. Consumers generally have no control over or knowledge of the exact location of the resources provided to them, but there is a sense of location independence in that the location may be identifiable at a higher level of abstraction (e.g., country, state, or data center).

[0082] Rapid Elasticity: Capacity can be rapidly and elastically provisioned, sometimes automatically, to quickly scale out and rapidly release to quickly scale in. To the consumer, provisionable capacity often appears to be unlimited and can be purchased in any quantity at any time.

[0083] Metered Services: Cloud systems leverage metering capabilities at some level of abstraction appropriate to the type of service (e.g., storage, processing, bandwidth, and active user accounts) to automatically control and optimize resource usage. Resource usage can be monitored, managed, and reported to provide transparency to both providers and consumers of the services utilized.

[0084] The service model is as follows:

[0085] Software as a Service (SaaS): The capability offered to the consumer is to use the provider's applications running on a cloud infrastructure. The applications are accessible from a variety of client devices through thin-client interfaces such as web browsers (e.g., web-based email). The consumer has no management or control over the underlying cloud infrastructure, including networks, servers, operating systems, storage, or even individual application features, with the possible exception of limited user-specific application configuration settings.

[0086] Platform as a Service (PaaS): The capability offered to a consumer is to deploy applications that the consumer creates or acquires, written using programming languages ​​and tools supported by the provider, onto a cloud infrastructure. The consumer does not manage or control the underlying cloud infrastructure, including networks, servers, operating systems, or storage, but does control the deployed applications and possibly the application hosting environment configuration.

[0087] Infrastructure as a Service (IaaS): The ability offered to consumers is to provision processing, storage, network, and other basic computing resources onto which the consumer can deploy and run any software, which may include operating systems and applications. The consumer does not manage or control the underlying cloud infrastructure, but does control the operating systems, storage, deployed applications, and possibly limited control over selected networking components (e.g., host firewalls).

[0088] The deployment model is as follows:

[0089] Private Cloud: The cloud infrastructure is operated exclusively for an organization. It can be managed by the organization or a third party and can be on-premise or off-premise.

[0090] Community Cloud: The cloud infrastructure is shared by several organizations to support a specific community with common concerns (e.g., mission, security requirements, policies, and compliance considerations). It may be managed by the organization or a third party and may exist on-premise or off-premise.

[0091] Public Cloud: The cloud infrastructure is made available to the general public or large industry organizations and is owned by an organization that sells cloud services.

[0092] Hybrid Cloud: A cloud infrastructure is a composite of two or more clouds (private, community, or public) that remain unique entities but are joined by standardized or proprietary technologies that enable data and application portability (e.g., cloud bursting for load balancing between clouds).

[0093] Cloud computing environments are service-oriented with an emphasis on statelessness, low coupling, modularity, and semantic interoperability. At the core of cloud computing is an infrastructure that includes a network of interconnected nodes.

[0094] Referring now to FIG. 8, an exemplary cloud computing environment 1000 is shown. As shown, the cloud computing environment 1000 includes one or more cloud computing nodes 100 with which local computing devices used by cloud consumers, such as, for example, a personal digital assistant (PDA) or cell phone 1000A, a desktop computer 1000B, a laptop computer 1000C, or an automobile computer system 1000N, or combinations thereof, may communicate. The nodes 100 may communicate with each other. They may be physically or virtually grouped (not shown), for example, in one or more networks, such as the private, community, public, or hybrid clouds described above, or combinations thereof. This enables the cloud computing environment 1000 to provide infrastructure-as-a-service, platform-as-a-service, or software-as-a-service, or combinations thereof, without the cloud consumer having to maintain resources on a local computing device. It should be understood that the types of computing devices 1000A-N illustrated in FIG. 8 are intended to be exemplary only, and that the computing node 100 and cloud computing environment 1000 can communicate with any type of computerized device via any type of network or network-addressable connection (e.g., using a web browser) or both.

[0095] 9, there is shown a set of functional abstraction layers 1100 provided by the cloud computing environment 1000. It should be understood in advance that the components, layers, and functions shown in FIG. 9 are intended to be exemplary only, and embodiments of the present invention are not limited thereto. As shown, the following layers and corresponding functions are provided:

[0096] The hardware and software layer 1102 includes hardware and software components. Examples of hardware components include a mainframe 1104, a RISC (reduced instruction set computer) architecture based server 1106, a server 1108, a blade server 1110, a storage device 1112, and a network and networking component 1114. In some embodiments, the software components include network application server software 1116 and database software 1118.

[0097] The virtualization layer 1120 provides an abstraction layer from which the following examples of virtual entities may be provided: virtual servers 1122, virtual storage 1124, virtual networks including virtual private networks 1126, virtual applications and operating systems 1128, and virtual clients 1130.

[0098] In one example, the management layer 1132 may provide the following functions: Resource provisioning 1134 provides dynamic procurement of computing and other resources utilized to execute tasks within the cloud computing environment. Metering and pricing 1136 provides cost tracking as resources are utilized within the cloud computing environment and accounting or billing for the consumption of these resources. In one example, these resources may include application software licenses. Security provides identity verification of cloud consumers and tasks as well as protection for data and other resources. User portal 1138 provides consumers and system administrators access to the cloud computing environment. Service level management 1140 provides allocation and management of cloud computing resources such that requested service levels are met. Service level agreement (SLA) planning and fulfillment 1142 provides advance arrangement and procurement of cloud computing resources anticipated to be required in the future according to SLAs.

[0099] The workload layer 1144 provides examples of functionality for which a cloud computing environment may be utilized. Examples of workloads and functions that may be provided from this layer include mapping and navigation 1146, software development and lifecycle management 1148, virtual classroom instructional delivery 1150, data analytics processing 1152, transaction processing 1154, and hyperparameter tuning 1156. The hyperparameter tuning programs 110a, 110b provide a way to scale resources and perform hyperparameter training for large workloads using limited computing resources.

[0100] The terms used herein are merely for the purpose of describing particular embodiments and are not intended to limit the present invention. As used herein, the singular forms "a", "an" and "the" are intended to include the plural unless the context clearly indicates otherwise. It will be further understood that as used herein, terms such as "comprises", "comprising", "includes", "including", "has", "have", "having", "with" and the like refer to the presence of stated features, integers, steps, operations, elements, or components, or combinations thereof, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, or groups thereof, or combinations thereof.

[0101] The description of various embodiments of the present invention has been presented for illustrative purposes, but is not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope of the described embodiments. The terms used in this specification are selected to best explain the principles of the embodiments, practical applications, or technical improvements to the technology found in the market, or to enable those skilled in the art to understand the embodiments disclosed herein.

Claims

1. 1. A computer-implemented method executed by one or more processors in an information processing system of a computer, comprising: receiving hyperparameter search input data, the hyperparameter search input data including multiple parallel experiments and an option to dictate a workload batch size for the hyperparameter search; Invoking a search algorithm to generate a set of hyperparameters; In a hyperparameter tuning resource allocator, receiving physical resource data for a cluster; determining a required workload via the hyperparameter tuning resource allocator, the hyperparameter tuning resource allocator determining the required workload using the received multiple parallel experiments; determining whether the option to dictate the workload batch size via the hyperparameter tuning resource allocator was selected; running the plurality of parallel experiments to explore and tune the sets of hyperparameters over the cluster, wherein the hyperparameter tuning resource allocator manages the cluster during the execution in response to the physical resource data and the requested workload and by using tensor swapping; 4. A computer-implemented method comprising:

2. the cluster comprises at least one accelerator and at least one central processing unit; the at least one accelerator comprises an accelerator memory; the at least one central processing unit comprises a CPU memory; the physical resource data for the cluster relates to the at least one accelerator, the at least one central processing unit, the accelerator memory, and the CPU memory; The method of claim 1.

3. 3. The method of claim 2, wherein in response to determining that the option to dictate the workload batch size is selected, the hyperparameter tuning resource allocator creates and manages virtual accelerators within the at least one accelerator.

4. The method of any one of claims 1 to 3, wherein the hyper-parameter tuning resource allocator initiates the execution of the multiple parallel experiments on the cluster using a training controller.

5. The method of claim 2 , wherein the hyperparameter tuning resource allocator manages floating point operations per second of the at least one accelerator.

6. 6. The method of claim 1, wherein in response to determining that the option to dictate the workload batch size was not selected, the hyperparameter tuning resource allocator dynamically scales a batch size of the parallel experiment based on the requested workload.

7. the cluster comprises at least one accelerator and at least one central processing unit; the tensor swapping includes transferring at least one tensor from the at least one accelerator to the at least one central processing unit, and the at least one central processing unit transferring the at least one tensor back to the at least one accelerator; the at least one tensor is generated via the at least one accelerator. The method according to any one of claims 1 to 6.

8. 1. A computer system for hyperparameter tuning, comprising: and one or more processors, one or more computer readable memories, one or more computer readable tangible storage media, and program instructions stored in at least one of the one or more computer readable tangible storage media for execution by at least one of the one or more processors via at least one of the one or more computer readable memories, the computer system being capable of performing a method, the method comprising: receiving hyperparameter search input data, the hyperparameter search input data including multiple parallel experiments and an option to dictate a workload batch size for the hyperparameter search; Invoking a search algorithm to generate a set of hyperparameters; In a hyperparameter tuning resource allocator, receiving physical resource data for a cluster; determining a required workload via the hyperparameter tuning resource allocator, the hyperparameter tuning resource allocator determining the required workload using the received multiple parallel experiments; determining whether the option to dictate the workload batch size via the hyperparameter tuning resource allocator was selected; running the plurality of parallel experiments to explore and tune the sets of hyperparameters over the cluster, wherein the hyperparameter tuning resource allocator manages the cluster during the execution in response to the physical resource data and the requested workload and by using tensor swapping; 1. A computer system comprising:

9. the cluster comprises at least one accelerator and at least one central processing unit; the at least one accelerator comprises an accelerator memory; the at least one central processing unit comprises a CPU memory; the physical resource data for the cluster relates to the at least one accelerator, the at least one central processing unit, the accelerator memory, and the CPU memory; 9. The computer system of claim 8.

10. 10. The computer system of claim 9, wherein in response to determining that the option to dictate the workload batch size is selected, the hyperparameter tuning resource allocator creates and manages virtual accelerators within the at least one accelerator.

11. The computer system of any one of claims 8 to 10, wherein the hyperparameter tuning resource allocator initiates the execution of the multiple parallel experiments on the cluster using a training controller.

12. The computer system of any one of claims 9 to 11, wherein the hyperparameter tuning resource allocator manages floating point operations per second of the at least one accelerator.

13. 13. The computer system of claim 8, wherein in response to determining that the option to dictate the workload batch size was not selected, the hyperparameter tuning resource allocator dynamically scales a batch size of the parallel experiment in response to the requested workload.

14. the cluster comprises at least one accelerator and at least one central processing unit; the tensor swapping includes transferring at least one tensor from the at least one accelerator to the at least one central processing unit, and the at least one central processing unit transferring the at least one tensor back to the at least one accelerator; the at least one tensor is generated via the at least one accelerator. A computer system according to any one of claims 8 to 13.

15. A computer program for causing a computer to execute the method according to any one of claims 1 to 7.

16. A computer readable storage medium having recorded thereon the computer program of claim 15.

Images