Resource allocation for tuning hyperparameters of large-scale deep learning workloads

Abstract

A method, computer system, and computer program product can perform hyperparameter tuning. Hyperparameter search input data can be received. A search algorithm that generates a set of hyperparameters can be invoked. A hyperparameter tuning resource allocator can receive physical resource data about a cluster. The hyperparameter tuning resource allocator can determine a required workload and can determine the required workload using a received number of parallel experiments. The hyperparameter tuning resource allocator can determine whether an option for authorized workload batch size is selected. The cluster can perform a number of parallel experiments to search and adjust the set of hyperparameters. The hyperparameter tuning resource allocator can manage the cluster according to the physical resource data and the required workload and by using tensor exchange.

Description

Background Technology

[0001] This invention relates generally to the field of computing, and more specifically to deep learning.

[0002] Deep learning, as a technology, has helped solve many problems previously considered intractable. Newer deep learning techniques have also made the availability of large datasets for learning necessary. The large datasets required in this field translate to the use of numerous accelerators on a single server or multiple servers. The accuracy of deep learning problems is sensitive to the hyperparameters used to tune the model. If hyperparameters must be tuned for large deep learning problems, the computational cluster needs to be available for tuning. Summary of the Invention

[0003] According to an exemplary embodiment, a method, computer system, and computer program product can perform hyperparameter tuning. Hyperparameter search input data can be received, including multiple parallel experiments and an option to manage the workload batch size for the hyperparameter search. A search algorithm for generating a set of hyperparameters can be invoked. A hyperparameter tuning resource allocator can receive physical resource data about the cluster. The hyperparameter tuning resource allocator can determine the required workload. The hyperparameter tuning resource allocator can use the received number of parallel experiments to determine the required workload. The hyperparameter tuning resource allocator can determine whether the option to manage the workload batch size has been selected. The cluster can execute multiple parallel experiments to search for and tune the set of hyperparameters. The hyperparameter tuning resource allocator can manage the cluster during execution based on the physical resource data and the required workload and by using tensor exchanges. Attached Figure Description

[0004] These and other objects, features, and advantages of the invention will become apparent from the following detailed description of exemplary embodiments of the invention, which will be read in conjunction with the accompanying drawings. The various features in the drawings are not to scale, as they are illustrated for clarity to facilitate understanding of the invention by those skilled in the art in conjunction with specific embodiments. In the drawings:

[0005] Figure 1 A networked computer environment according to at least one embodiment is shown;

[0006] Figure 2 This is an operational flowchart illustrating an end-to-end process for a deep learning workflow according to at least one embodiment;

[0007] Figure 3 This is an operational flowchart illustrating a process for hyperparameter tuning according to at least one embodiment;

[0008] Figure 4 The hyperparameter tuning procedure and an example of its interaction with the cluster are shown;

[0009] Figure 5 An example of a computation node that can be used in the hyperparameter tuning process is shown;

[0010] Figure 6 An example of a computing node with a virtual accelerator that can be used during hyperparameter tuning is shown;

[0011] Figure 7 According to at least one embodiment Figure 1 A block diagram of the internal and external components of the computer and server shown;

[0012] Figure 8 According to embodiments of this disclosure, including Figure 1 A block diagram illustrating a cloud computing environment for a computer system; and

[0013] Figure 9 According to embodiments of this disclosure Figure 7 A block diagram illustrating the functional layers of an illustrative cloud computing environment. Detailed Implementation

[0014] Detailed embodiments of the claimed structures and methods are disclosed herein; however, it should be understood that the disclosed embodiments are merely illustrative of the claimed structures and methods, and they may be implemented in different forms. The invention can be embodied in many different forms and should not be construed as limited to the exemplary embodiments set forth herein. Rather, these exemplary embodiments are provided to make this disclosure thorough and complete, and to fully convey the scope of the invention to those skilled in the art. Details of well-known features and techniques may be omitted in the description to avoid unnecessarily obscuring the presented embodiments.

[0015] The exemplary embodiments described below provide systems, methods, and procedures for hyperparameter tuning. These embodiments offer the ability to improve the field of deep learning by allowing the search for hyperparameters for large-scale deep learning workloads using limited computational resources. This eliminates the need for having or access to large-scale accelerator clusters.

[0016] Specifically, data scientists and researchers typically do not have access to large clusters of accelerators, such as large clusters of graphics processing units (GPUs). Most deep learning experiments are conducted in shared cloud environments. In shared cloud environments, large accelerator clusters are either unavailable or prohibitively expensive for many data scientists and researchers.

[0017] Therefore, it may be advantageous to provide ways to allow, for example, data scientists and researchers from academia to perform hyperparameter searches on cloud environments using common accelerator cluster resources available in cloud environments, so that hyperparameter searches can be performed in an economically viable manner.

[0018] The exemplary embodiments described below allow for the search and tuning of hyperparameters with limited computing resources for large-scale deep learning workloads, and also allow data scientists and researchers, for example, from academia, to perform hyperparameter search and tuning in a cost-effective manner in cloud environments. The exemplary embodiments allow for avoiding the infrastructure costs associated with acquiring or accessing large-scale accelerator clusters. The exemplary embodiments allow for improved resource management and dynamic adjustment of accelerator usage based on the requirements of hyperparameter search jobs. The exemplary embodiments will allow organizations to reduce the resources required for hyperparameter search and tuning during the model development phase, enabling large-scale accelerator clusters to be dedicated to production workloads.

[0019] Reference Figure 1 This describes an exemplary networked computer environment 100 according to one embodiment. The networked computer environment 100 may include a computer 102 having a processor 104 and a data storage device 106 capable of running software program 108 and hyperparameter tuning program 110a. The networked computer environment 100 may also include a server 112 capable of running hyperparameter tuning program 110b, which can interact with a database 114 and a communication network 116. The networked computer environment 100 may include multiple computers 102 and servers 112, only one of which is shown. The communication network 116 may include different types of communication networks, such as wide area networks (WANs), local area networks (LANs), telecommunications networks, wireless networks, public switched networks, and / or satellite networks. It should be understood that... Figure 1 This illustration provides only one possible implementation and does not imply any limitation regarding the environment in which different embodiments may be implemented. Many modifications can be made to the depicted environment based on design and implementation requirements.

[0020] Client computer 102 can communicate with server computer 112 via communication network 116. Communication network 116 may include connections such as wired, wireless communication links, or fiber optic cables. (See reference...) Figure 7The server computer 112 discussed may include internal component 902a and external component 904a, and the client computer 102 may include internal component 902b and external component 904b. The server computer 112 may also operate in a cloud computing service model (such as Software as a Service (SaaS), Platform as a Service (PaaS), or Infrastructure as a Service (IaaS)). The server 112 may also reside in a cloud computing deployment model, such as a private cloud, community cloud, public cloud, or hybrid cloud. The client computer 102 may be, for example, a mobile device, telephone, personal digital assistant, netbook, laptop computer, tablet computer, desktop computer, or any type of computing device capable of running programs, accessing a network, and accessing database 114. According to different implementations of this embodiment, hyperparameter tuning programs 110a, 110b may interact with database 114, which may be embedded in different storage devices, such as, but not limited to, computer / mobile device 102, networked server 112, or cloud storage service. According to various embodiments of this embodiment, hyperparameter tuning programs 110a, 110b may interact with, for example, Figure 4 The cluster interactions shown are illustrated. A cluster can include, for example, cluster interactions. Figure 5 The computing node 500 shown and / or as shown Figure 6 The computing node 600 shown has a virtual accelerator.

[0021] According to this embodiment, a user using client computer 102 or server computer 112 can use hyperparameter tuning programs 110a and 110b (respectively) to perform hyperparameter tuning for the deep learning process. See below for further details. Figure 2-9 The hyperparameter tuning method will be explained in more detail below. Figure 4 Specifically shown, hyperparameter tuning programs 110a and 110b may include a job input generator, a search algorithm, a hyperparameter tuning resource allocator, and a training controller. Some or all of the job input generator, search algorithm, hyperparameter tuning resource allocator, and training controller may be provided in software form. Hyperparameter tuning programs 110a and 110b can work in conjunction with a cluster.

[0022] Now for reference Figure 2 The diagram depicts an operational flowchart of an exemplary end-to-end deep learning workflow process 200 according to at least one embodiment, wherein hyperparameter tuning procedures 110a and 110b will take actions.

[0023] The deep learning model development phase can include sub-phases such as processing high-level data, selecting a model, and tuning hyperparameters. The developers or researchers involved in this phase typically utilize resource-constrained cloud infrastructure. Large, dedicated clusters are usually only available in production environments. Providing resources in the cloud can involve data scientists, developers, and / or DevOps personnel who leverage resource allocators and orchestrators to perform this development phase, working in conjunction with centralized infrastructure located in the cloud.

[0024] like Figure 2 As shown, the end-to-end deep learning workflow process 200 may include steps of training a model and ultimately identifying submitted patterns or data. A training task may be specified in step 202. This step 202 may be performed, for example, by identifying specific business problems, acquiring subject-specific expertise, defining analytical units and prediction objectives, prioritizing model criteria, considering risk and success criteria, and / or identifying the desired search scope. The training task may be developing a model that can receive images of animals as input and can identify and correctly identify the types of animals in different images. Another training task may be developing a model that can receive audio sounds or recordings as input and can perform speech-to-text conversion and identify physical or age characteristics of the speaker based on their voice. Yet another training task may be using image classification at various stages of a production line to identify defects in products.

[0025] Training data can be provided in step 204. For example, training data can be uploaded. The training data can be raw data obtained from a data source. Advanced data processing can be performed in step 206. This advanced data processing can also be referred to as data preprocessing. For example, this processing can handle missing data, outliers, or data with long tails. Data with long tails can be remedied by filtering the raw data. Data merging can be performed in step 206. Following the example given above for image classification in a production line, the training data can include photos of defective products and photos of products without any defects. Step 206 may alternatively or additionally include data enrichment, data normalization, and data format conversion to create compatibility with the training algorithm.

[0026] In step 208, a base model can be selected from different candidate models that have been previously used and successfully used for identification. Hyperparameters or hyperparameter configurations can be tuned in step 210. The following... Figure 3An exemplary process for performing the sub-step of step 210 is described as a hyperparameter tuning process. In step 212, the model can be trained. This training can be done using a training dataset and can produce model artifacts. The model training in step 212 is performed using a consistent set of hyperparameters tuned in step 210; for example, the hyperparameters do not change during step 212. After the model has been trained, in step 214, the deep learning model or neural network can then be prepared to perform inference or can be prepared to recognize patterns or data. For example, corresponding to the possible implementations mentioned above, the trained model can then be able to identify the type of animal shown in an image, perform text-to-speech or natural language processing from received audio signals, or recognize characteristics about the speaker. The trained deep learning model can be run on a test dataset to produce inference. The trained deep learning model can be able to identify whether an image shows a cat, dog, elephant, or camel, etc. The steps of the end-to-end deep learning workflow process 200 can be performed repeatedly or iteratively, as indicated by the return arrows that propagate back to earlier steps in the process after the model training step 212, so that steps (e.g., steps 204 to 212) can be executed again.

[0027] The end-to-end deep learning workflow process 200 can utilize iterative hyperparameter search and tuning phases. Hyperparameter tuning may require multiple training runs to experiment with different sets of the searched hyperparameters. Therefore, in typical scenarios, the amount of resources required for hyperparameter search can approach, equal to, or exceed the amount of resources required for model training in a production environment. This need for large resources can lead to resource constraints, as dedicating large-scale clusters to hyperparameter search experiments reduces the resources available for data scientists to train models. For large-scale deep learning workloads, the size of the problem and the amount of resources required increase exponentially. However, hyperparameter search occurs during the research and development phase. Training and deploying models are performance-constrained, but hyperparameter search is not. Therefore, addressing resource constraints is more important than the performance of hyperparameter search. Thus, for large-scale workloads, establishing the hyperparameter search / tuning of step 210 would be advantageous, as this hyperparameter search / tuning can be performed if computational resources are limited.

[0028] Now for reference Figure 3 The document describes an operational flowchart illustrating an exemplary hyperparameter tuning 210 used by hyperparameter tuning procedures 110a, 110b according to at least one embodiment. Figure 3 The flowchart shown is suitable for Figure 2 Hyperparameter tuning 210 of the end-to-end deep learning workflow process 200 shown.

[0029] Hyperparameter tuning in deep learning can involve selecting an optimal set of hyperparameters for a learning algorithm. This selection can involve a search job within a configuration space defined according to the algorithm. The hyperparameter search job can involve using different search algorithms that help narrow down the search space. The workload details and search range of the hyperparameters can be provided as input to the search algorithm. The search algorithm can generate different sets of hyperparameters for experimentation based on the algorithms used. Experiments can occur across multiple training runs on a cluster, such as on compute nodes.

[0030] According to at least some embodiments of the invention, a hyperparameter-tuned resource allocator (HTRA) is implemented as an add-on module. The HTRA captures the resource requirements of the search job and simulates large-scale workloads by seamlessly managing the use of physical and virtual resources. Using workload details and resource requirements, the HTRA automatically manages the computational resources for the requested number of search experiments. The HTRA can manage resources when batch size scaling is allowed. The HTRA can also manage resources when batch size scaling is not allowed. When batch size scaling is not allowed, the same batch size is required for each search experiment.

[0031] In step 302 of hyperparameter tuning 210, hyperparameter search input data is received as part of the hyperparameter search job. The hyperparameter search input data may include workload details, search range, resources required for tuning (e.g., the number of accelerators required to perform tuning experiments), the number of parallel experiments to be tuned, options for authorizing a workload batch size for the hyperparameter search, and (if authorized) the workload batch size. The hyperparameter search input data may include which hyperparameters to use. For example, momentum and weight decay may be selected as hyperparameters to be tuned. The workload batch size may also be considered a hyperparameter. Hyperparameter search information may be obtained via a graphical user interface (GUI) prompting the user to enter hyperparameter search input data.

[0032] For those implementations of deep learning involving image classification, the batch size can be the number of images that can be processed in a single batch. For example, in a dataset of one thousand images, it might be possible to process 10 images at a time. Therefore, the batch size for this instance is 10. If only one image can be processed at a time, the batch size is 1. If the batch size is larger, processing can work faster across the entire dataset if sufficient computational resources are available. However, larger computational resources are typically required to process larger batch sizes and are often unavailable.

[0033] In step 304, the hyperparameter search job invokes a search algorithm that generates a set of hyperparameters for the experiments based on the requested number of parallel experiments. For example, search algorithms such as Hyperband, random search, grid search, Bayesian optimization, gradient-based optimization, evolutionary optimization, population-based algorithms, radial basis function algorithms, and / or spectral algorithms may be used in step 304 to generate the set of hyperparameters for the experiments. The hyperparameters obtained in step 302 (e.g., momentum and weight decay, and batch size) may be fed into the search algorithm, which may generate test numbers or test values ​​for these hyperparameters that will be tested and tuned in subsequent steps. The deep learning model may have many hyperparameters, for example, fifteen to twenty, and the values ​​selected for them will be generated and tuned once the model is trained and subsequently used in the deep learning model.

[0034] In step 306, HTRA receives physical resource data about the cluster (e.g., compute nodes), which can be used to run the data to perform the experiment. The physical resource data may include the number of accelerators in the cluster. Accelerators may be, for example, graphics processing units (GPUs), field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), or other accelerators. The physical resource data may also include the amount of memory in the accelerators, such as the memory held by each GPU (GPU memory). The physical resource data may also include the number of central processing units (CPUs) in the cluster and the amount of memory held by the CPUs or held by each CPU (CPU memory). The physical resource data may also include the amount of system memory accessible by the CPUs or accessible by each CPU. Knowing or having the available physical resource data in the cluster will help HTRA determine how much scaling will be needed and how to allocate the scaling in the GPUs.

[0035] In step 308, HTRA determines or captures the resource requirements of the search job and determines the workload required to perform the hyperparameter search and tuning. These resource requirements of the search job can be found in the hyperparameter search input data received in step 302. HTRA may perform one or more calculations to determine the amount of workload required to perform the hyperparameter search and tuning.

[0036] In step 310, HTRA determines whether to authorize a workload batch size. HTRA can make this determination in step 310 using the hyperparameter search input data received in step 302. The hyperparameter search input data received in step 302 may include binary variables that positively or negatively indicate whether a workload batch size is required. When a workload batch size is required, the hyperparameter search input data will also include the workload batch size provided as a variable. For example, the hyperparameter search input data may include an authorization for a workload batch size of one (1).

[0037] In step 312, HTRA uses the training controller to schedule and launch experiments on the cluster. The training controller has knowledge of runtime resources and is configured to schedule jobs on available resources. The training controller can be software configured to be compatible with a specific computing cluster.

[0038] In step 314, experiments are performed on the cluster. A single training experiment or multiple training experiments can be performed on the compute nodes of the cluster. Data including hyperparameters can be broken down into smaller groups and loaded into the cores of the accelerators in the cluster for computation with possible sets of hyperparameter values. The cluster uses tensor comprehensions to perform experiments. For example, the cluster uses tensor comprehensions to manage memory requirements. Thousands of tests can be performed with different hyperparameters to identify the optimal hyperparameter values. Specific sets of hyperparameters can be tested in the compute cluster, and inference time and accuracy can be observed. Different sets of hyperparameters can be tested through experiments in the cluster until the optimal or most efficient set of hyperparameter values ​​is determined. The efficient set can then be used for, for example... Figure 2 The model training steps 212 and as shown are as follows Figure 2 The recognition step 214 is shown. Recognition step 214 can occur in deep learning when information, such as image or audio signal information, is fed into a deep learning model and the deep learning model submits an output analyzing the image or audio signal.

[0039] In step 316, HTRA manages the cluster. HTRA simulates large-scale workloads by managing the cluster's physical and virtual resource usage. During experiment execution, the cluster uses tensor swapping to manage memory requirements, and HTRA manages tensor swapping. During or at the end of a set of experiments, the cluster can provide training outputs or metrics back to the training controller and / or HTRA, allowing the determination, initiation, and / or control of the next set of experiments. HTRA can use the training controller to initiate the next set of experiments on the cluster.

[0040] Using the provided input, HTRA automatically manages the resources for the requested number of experiments.

[0041] Figure 4Examples of hyperparameter tuning procedures 110a and 110b and their interaction with cluster 408 to perform hyperparameter tuning experiments on cluster 408 are shown. Cluster 408 is a computing cluster that may include accelerators, as described below. Figure 5 and 6 As described, the hyperparameter tuning programs 110a and 110b may include sub-components such as a hyperparameter job input generator, a search algorithm 402, a hyperparameter tuning resource allocator (HTRA) 404, and a training controller 406.

[0042] The hyperparameter job input generator can use the graphical user interface (HI GUI) 400 to obtain input about the hyperparameter job search to be performed, i.e., to execute step 302. The HI GUI 400 can prompt and request developers or researchers to enter hyperparameter job search input data, such as workload details, search scope, required resources for tuning (e.g., the number of accelerators required), number of parallel experiments, options for requesting the workload batch size for the hyperparameter search, and (if selected) the workload batch size. Specific hyperparameters to be tested can be entered into the HI GUI 400 by the user.

[0043] The HI GUI 400 receiving hyperparameter search input data can invoke search algorithm 402, enabling the execution of step 304, i.e., enabling the generation of a set of hyperparameters (i.e., a set of hyperparameter values) based at least in part on the number of parallel experiments requested as part of the hyperparameter job search input data. Different instances of search algorithm 402 have been mentioned above.

[0044] HTRA 404 performs step 306 by receiving physical resource data about cluster 408, which can be used to perform hyperparameter tuning experiments. HTRA 404 may have a data connection to cluster 408, such as a physical or wireless connection, to receive the physical resource data. HTRA 404 may also perform step 308 to determine the required workload. HTRA 404 can use the hyperparameter search input data received in step 302 to determine the required workload in step 308. HTRA 404 can also use the hyperparameter search input data received in step 302 to determine whether a workload batch size is required in step 310. HTRA also uses training controller 406 to schedule experiments on cluster 408 and launch those experiments on cluster 408. During or after performing experiments on cluster 408, the cluster may provide training outputs / metrics to HTRA 404 and / or training controller 406 to determine and control further experiments in hyperparameter tuning.

[0045] Figure 5 An example of cluster 408 is shown, which includes computing nodes 500 that can assist in performing experiments in at least some embodiments of the method according to the invention. Computing node 500 may include system memory 502, a central processing unit (CPU) 504, and multiple accelerators. In this exemplary embodiment, the multiple accelerators are graphics processing units (GPUs). Figure 5 The diagram illustrates a first GPU 506a and a second GPU 506b, but in some embodiments, the compute node may include three, four, five, six, or eight GPUs. The first GPU 506a and the second GPU 506b may be general-purpose graphics processing units (GPGPUs). Tensor Core GPU. (NVIDIA and Tesla, and all NVIDIA-based trademarks and logos, are trademarks or registered trademarks of Nvidia, Inc.) In other embodiments, an additional accelerator may be provided instead of the GPU.

[0046] A high-speed memory bus 508 connects the CPU 504 to the system memory 502. The high-speed memory bus 508 can transfer information at a rate of 170 GB / s or higher. A mesh multi-directional interface 510c connects the first GPU 506a and the second GPU 506b to each other. Mesh multi-directional interface 510a connects the first GPU 506a to the CPU 504, and mesh multi-directional interface 510b connects the second GPU 506b to the CPU 504. This compute node 500 allows tensor swapping, enabling the system and methods to scale beyond typical computing capabilities. Mesh multi-directional interfaces 510a-c can be embedded in the processor's silicon.

[0047] For tensor swapping, tensors generated during experimental computations in the GPU can be temporarily transferred by CPU 504 from the accelerator (e.g., from the GPUs (e.g., first GPU 506a and second GPU 506b)) or to CPU 504 and system memory 502, preventing the accelerator from becoming overloaded due to data and freezes. Through tensor swapping, the accelerator's resources can be used to perform additional computations and experiments, thereby generating new tensors or new tensors. After an experiment is completed or while some experiments are still in progress, tensors temporarily stored in CPU 504 and / or system memory 502 can be transferred or transferred and reloaded back into the accelerator, for example, back into the first and second GPUs 506a, b, and back into hyperparameters searched in the accelerator, such as those required by training jobs running on the accelerator.

[0048] The mesh multidirectional interface 510a-c can be a bidirectional interface, comprising eight differential pairs in each direction, for a total of thirty-two wires. These pairs can be DC-coupled and can use 85Ω differential terminations with embedded clocks. For ease of routing, the mesh multidirectional interface 510a-c can support channel reversal and channel polarity, meaning the physical channel ordering and their polarities can be reversed between two devices. The mesh multidirectional interface 510a-c can be respectively... 1.0 or 2.0 connectivity. (NVLink is a registered trademark of Nvidia Corporation.) The 510a-c mesh multi-directional interface can include four ports each. Connectivity. In some embodiments, cluster 408 may include four accelerators (e.g., four GPUs) and two CPUs, wherein the four accelerators are fully interconnected with each other using a fourth link to one of the two CPUs. In some embodiments, the accelerators and CPU 504 may access each other's memory in a flat address space, for example, for direct load and store. A flat address space can be supported by a new address translation service. Additionally, in some embodiments, CPU 504 and the accelerators may have native support for atomic operations. A flat address space enables cache coherency and allows CPU 504 to efficiently cache the accelerator's memory, thereby significantly improving latency and thus performance. In a cluster with eight accelerators on a single substrate, a switch can be provided to connect each of the eight accelerators to all other seven accelerators on the same substrate. For example, the cluster may include eight GPUs with the described connectivity features.

[0049] Where batch size scaling is possible, in response to HTRA 404 determining this, HTRA 404 simulates large-scale workloads by dynamically scaling the batch size according to workload requirements. If the option to search for an authorized workload batch size for hyperparameters is not selected, batch size scaling is possible. HTRA 404 simulates large-scale workloads by dynamically scaling the batch size based on the provided workload batch size and resource requirements.

[0050] As part of this dynamic scaling, HTRA 404 identifies the cumulative batch size required for the search experiments. The cumulative batch size is the workload batch size multiplied by the number of accelerators required. HTRA 404, starting at step 306, has details of the number of physical accelerators in cluster 408, as well as the available CPU memory and accelerator memory. HTRA 404 also has the required number of experiments. The cumulative batch size is allocated among the available physical accelerators in cluster 408 for the required number of experiments.

[0051] Therefore, in this embodiment, without batch size requirements and dynamic scaling, each physical accelerator would have to handle higher batch sizes. HTRA 404 seamlessly handles the memory and communication requirements arising from batch size scaling using tensor swapping between the accelerators (e.g., any GPU, such as the first GPU 506a and the second GPU 506b) and the CPU 504. HTRA 404's batch size scaling mechanism can be based on the available accumulated accelerator memory and CPU memory.

[0052] For cases where batch size scaling cannot be used, such as when a workload batch size is required in the hyperparameter search input data, HTRA 404 determines it and then, in response, manages resources by creating and managing virtual accelerators that can simulate physical accelerators. In this way, large-scale simulations of large workloads can be performed. The workload batch size will be forced for the hyperparameter search. For example, the hyperparameter search input data from step 302 may include an authorization for a workload batch size of one (1).

[0053] For this embodiment with workload batch size requirements, HTRA 404 initially allocates virtual accelerators based on the number of accelerators required for the total number of parallel experiments for the hyperparameter search request. The allocation also considers the physical accelerators, accelerator memory, and CPU memory available for each compute node in cluster 408. The creation of virtual accelerators depends on the accumulated accelerator memory and available CPU memory.

[0054] Some search algorithms (like Hyperband) use techniques such as successive halving, where the number of parallel experiments is reduced at each step of the hyperparameter search. This reduction occurs by selecting the most likely configurations and discarding others. In such cases, the HTRA 404 will reallocate virtual accelerators as needed at each step to utilize the additional resources available. This reallocation makes more resources (e.g., memory per second and floating-point operations (triggers)) available for the remaining virtual accelerators needed in another step. This reallocation implicitly makes the experiment run faster for that step. The communication mode can also change based on the reallocation.

[0055] This use of virtual accelerators (e.g., virtual GPUs) ensures efficient allocation of available accelerator resources and maximizes the utilization of the physical resources of cluster 408 by dynamically adjusting resources based on job requirements searched for hyperparameters.

[0056] The HTRA 404 can manage virtual accelerator creation and virtual accelerator memory requirements by using tensor swapping between multiple accelerators and CPU 504.

[0057] HTRA 404 is an instance of allocating a virtual accelerator within a physical accelerator. Figure 6 As shown in the image. Figure 6 This demonstrates how the HTRA 404 operates when the workload batch size is required in the hyperparameter search input data. Figure 6 A compute node 600 is shown, in which a CPU 504 has a CPU memory 620, and virtual GPUs 612a-i are created in the first and second GPUs 506a, b shown. Virtual GPUs 612a-i can access the CPU memory 620 for tensor swapping. A particular virtual GPU (e.g., virtual GPU 612a) can always access the same address space within the CPU memory 620 or the same address space within the system memory 502 for tensor swapping. Figure 6 It also shows the communication modes between and within GPUs.

[0058] Hyperparameter training that occurs without the methods and systems described herein takes place without a hyperparameter tuning resource allocator and without the ability to scale up using limited computational resources. In this example, which lacks scaling capabilities and has a cluster of eight GPUs, a hyperparameter search job for a workload requiring eight GPUs with a batch size of one (1) can only perform one training experiment. The tuning is limited by the available physical resources.

[0059] This example contrasts with examples of systems and methods for hyperparameter search / tuning of large-scale deep learning workloads with limited resources, as described herein. In the invention described herein, a hyperparameter tuning resource allocator, such as the HTRA 404, is used. In this example, even though the cluster 408 has only eight GPUs, the hyperparameter search job is able to run eight experiments in parallel with a batch size of 1. This operation would typically require sixty-four (64) GPUs to run in a conventional scenario, with eight (8) GPUs dedicated to each experiment. The HTRA 404 can simulate large-scale workloads with limited resources. Based on the option of choosing whether to use the workload batch size for hyperparameter search, the HTRA 404 can take one of two approaches to seamlessly perform simulations of large-scale workloads.

[0060] If no option to authorize workload batch size is selected, the HTRA 404 can use the batch size scaling mechanism and can seamlessly manage the physical accelerator's floating-point operations per second (triggers) and memory requirements.

[0061] In a given example without a workload batch size authorization, where the workload requires running eight experiments in parallel, each experiment can be run on a single physical accelerator by seamlessly scaling to a batch size of eight experiments per accelerator (e.g., a batch size of eight experiments per GPU). This dynamic scaling allows the eight experiments to run in parallel even with limited resources. When the initial tuning operation required 64 accelerators, this system and method allow the operation to be performed with only 8 accelerators.

[0062] Furthermore, if a search algorithm similar to Hyperband is used, which can halve the number of experiments in successive steps, the HTRA 404 can reallocate resources after each run. In this instance where the number of experiments is halved for successive steps, the next step of four parallel experiments can use each of eight promoters with a batch size of 4. This reallocation speeds up the run and maximizes resource utilization.

[0063] If the option to authorize the workload batch size is selected and the batch size is authorized as one (1), HTRA 404 can create a virtual accelerator to handle resource requirements. Figure 6 Cluster 408, including compute node 600, is shown, in which virtual GPU 612a-i has been created by HTRA 404. Figure 6 The compute node 600 and Figure 5 The computing node 500 shown has many common components, making some common reference figures used for Figure 5 and Figure 6 middle.

[0064] In the given example, when a large workload batch size is required, the HTRA 404 can create sixty-four virtual accelerators, for example, sixty-four GPUs, with eight virtual GPUs in each physical GPU, and each virtual GPU handling a batch size of 1. This creation of virtual accelerators (e.g., virtual GPU 612a-i) allows eight experiments to run in parallel, even with limited resources. The tuning operation initially required 64 physical accelerators, but can now be performed using only 8 physical accelerators. All memory, floating-point operations per second (flip-flops), and communication requirements of the virtual GPU 612a-i will be seamlessly handled by the HTRA 404. Figure 6 The image shows the memory 620 of the virtual GPU 612a-i and CPU 504.

[0065] Furthermore, in this alternative, if a Hyperband-like search algorithm is used, where the number of experiments is halved in successive steps, HTRA 404 will reallocate virtual accelerators using all available resources. In this example, the next step after four parallel experiments will have 32 virtual accelerators created, each with a batch size of 1 based on the authorized processing size.

[0066] HTRA 404 can manage intra-accelerator communication, that is, communication between virtual accelerators within a specific physical accelerator. For example, HTRA 404 can manage intra-frame GPU communication 614a between virtual GPUs 612a and 612b in the first GPU 506a (where the first GPU 506a is a physical GPU), intra-frame GPU communication 614b between virtual GPUs 612d and 612e in the first GPU 506a, intra-frame GPU communication 614c between virtual GPUs 612f and 612g in the second GPU 506b (where the second GPU 506b is a physical GPU), and intra-frame GPU communication 614d between virtual GPUs 612h and 612i in the second GPU 506b. HTRA 404 can also manage inter-accelerator communication. For example, HTRA 404 can manage inter-GPU communication 616 occurring between a virtual GPU on one physical GPU and a virtual GPU on another physical GPU. Inter-GPU communication 616 can occur between the virtual GPU 612c of the first GPU 506a and the virtual GPU 612f of the second GPU 506b. Inter-GPU communication 616 can be achieved via, for example... Figure 5 The mesh multidirectional interface 510c shown in the diagram occurs due to the mesh multidirectional interface. HTRA 404 can use interfaces like NCCL ( Communication technologies such as collective communication libraries (CLCs) or MPI (Message Passing Interface) are used to manage communication between virtual accelerators (such as virtual GPUs 612a-i).

[0067] If the option to authorize workload batch size is not selected, the HTRA 404 can still create virtual accelerators to manage experiments, similar to the implementation described for which the option to authorize workload batch size is selected.

[0068] Understandable. Figure 2-6 Illustrations of certain embodiments are provided only and do not imply any limitation on how different embodiments may be implemented. Many modifications may be made to the depicted embodiments based on design and implementation requirements.

[0069] As described in the above embodiments, hyperparameter tuning procedures 110a, 110b can improve computer functionality by allowing limited computing resources to be expanded to perform hyperparameter tuning for large-scale workloads.

[0070] Figure 7 This is an illustrative embodiment of the present invention. Figure 1 Block diagram 900 depicts the internal and external components of a computer. It should be understood that... Figure 7 This illustration provides only one possible implementation and does not imply any limitation regarding the environment in which different embodiments may be implemented. Many modifications can be made to the depicted environment based on design and implementation requirements.

[0071] Data processing systems 902 and 904 represent any electronic device capable of executing machine-readable program instructions. Data processing systems 902 and 904 may represent smartphones, computer systems, PDAs, or other electronic devices. Examples of computing systems, environments, and / or configurations that data processing systems 902 and 904 may represent include, but are not limited to, personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, multiprocessor systems, microprocessor-based systems, network PCs, minicomputer systems, and distributed cloud computing environments that include any of the above systems or devices.

[0072] The user client computer 102 and the network server 112 may include Figure 7 The internal components 902a, 902b and external components 904a, 904b of the corresponding groups are shown. Each group of internal components 902a, 902b includes one or more processors 906, one or more computer-readable RAMs 908 and one or more computer-readable ROMs 910 on one or more buses 912, as well as one or more operating systems 914 and one or more computer-readable tangible storage devices 916. One or more operating systems 914, software programs 108, and hyperparameter tuning programs 110a in client computer 102, and hyperparameter tuning programs 110b in network server 112, may be stored on one or more computer-readable tangible storage devices 916 for execution by one or more processors 906 via one or more RAMs 908 (which typically include cache memory). Figure 6 In the embodiment shown, each computer-readable tangible storage device 916 is a disk storage device of an internal hard disk drive. Alternatively, each computer-readable tangible storage device 916 is a semiconductor storage device, such as ROM 910, EPROM, flash memory, or any other computer-readable tangible storage device capable of storing computer programs and digital information.

[0073] Each set of internal components 902a, 902b also includes an R / W drive or interface 918 for reading from and writing to one or more portable computer-readable physical storage devices 920, such as CD-ROM, DVD, Memory Stick, magnetic tape, disk, optical disc, or semiconductor storage devices. Software programs (such as software program 108 and hyperparameter tuning programs 110a, 110b) can be stored on one or more of the corresponding portable computer-readable physical storage devices 920, read from via the corresponding R / W drive or interface 918, and loaded into the corresponding hard disk drive 916.

[0074] Each set of internal components 902a, 902b may also include a network adapter (or switch port card) or interface 922, such as a TCP / IP adapter card, a wireless Wi-Fi interface card, or a 3G or 4G wireless interface card, or other wired or wireless communication links. Software program 108 and hyperparameter tuning program 110a from client computer 102 and hyperparameter tuning program 110b from network server computer 112 can be downloaded from an external computer (e.g., a server) via a network (e.g., the Internet, a local area network, or another wide area network) and the corresponding network adapter or interface 922. Software program 108 and hyperparameter tuning program 110a from client computer 102 and hyperparameter tuning program 110b from network server computer 112 are loaded into the corresponding hard disk drive 916 from the network adapter (or switch port adapter) or interface 922. The network may include copper wire, fiber optic, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers.

[0075] Each set of external components 904a, 904b may include a computer display monitor 924, a keyboard 926, and a computer mouse 928. External components 904a, 904b may also include a touchscreen, virtual keyboard, touchpad, pointing device, and other human-computer interface devices. Each set of internal components 902a, 902b also includes a device driver 930 connected to the computer display monitor 924, keyboard 926, and computer mouse 928. The device driver 930, R / W driver or interface 918, and network adapter or interface 922 include hardware and software (stored in storage device 916 and / or ROM 910).

[0076] This invention can be a system, method, and / or computer program product with any possible level of technical detail integration. The computer program product may include a computer-readable storage medium (or media) having computer-readable program instructions thereon for causing a processor to execute aspects of the invention.

[0077] Computer-readable storage media can be tangible means for retaining and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example, but not limited to, electronic storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of computer-readable storage media includes: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disk read-only memory (CD-ROM), digital universal disk (DVD), memory sticks, floppy disks, mechanical encoding devices such as punch cards or protrusions in slots having instructions recorded thereon, and any suitable combination of the foregoing. As used herein, computer-readable storage media should not be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses passing through fiber optic cables), or electrical signals transmitted through wires.

[0078] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to a suitable computing / processing device via a network (e.g., the Internet, a local area network, a wide area network, and / or a wireless network), or to an external computer or external storage device. The network may include copper cables, optical fibers, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to a computer-readable storage medium within the suitable computing / processing device.

[0079] Computer-readable program instructions used to perform the operations of this invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, integrated circuit configuration data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​(such as Smalltalk, C++, etc.) and procedural programming languages ​​(such as the "C" programming language or similar programming languages). The computer-readable program instructions may be executed entirely on a user's computer, partially on a user's computer, as a standalone software package, partially on a user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter case, the remote computer may be connected to the user's computer via any type of network (including a local area network (LAN) or a wide area network (WAN)) or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry including, for example, programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs) may execute computer-readable program instructions by utilizing state information from the computer-readable program instructions to personalize the electronic circuitry in order to perform aspects of this invention.

[0080] The present invention will now be described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0081] These computer-readable program instructions may be provided to a processor of a computer or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / actions specified in one or more blocks of a flowchart and / or block diagram. These computer-readable program instructions may also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner, such that the computer-readable storage medium storing the instructions includes an article of manufacture containing instructions that implement aspects of the functions / actions specified in one or more blocks of a flowchart and / or block diagram.

[0082] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus, or other device to produce computer-implemented processing, such that the instructions executed on the computer, other programmable apparatus, or other device perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0083] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. Each block in a flowchart or block diagram may represent a module, segment, or portion of instructions, including one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the figures. For example, two blocks shown consecutively may actually be completed as a single step, executed simultaneously, substantially simultaneously, or with partial or complete temporal overlap, or the blocks may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action or executes a combination of dedicated hardware and computer instructions.

[0084] It should be understood that while this disclosure includes a detailed description of cloud computing, the implementation of the teachings cited herein is not limited to cloud computing environments. Rather, embodiments of the invention can be implemented in conjunction with any other type of computing environment now known or developed hereafter.

[0085] Cloud computing is a service delivery model that enables convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, network bandwidth, servers, processing, memory, storage, applications, virtual machines, and services), which can be rapidly provisioned and released with minimal management effort or interaction with the service provider. This cloud model may include at least five features, at least three service models, and at least four deployment models.

[0086] The features are as follows:

[0087] On-demand self-service: Cloud consumers can unilaterally and automatically provide computing power, such as server time and network storage, as needed, without requiring human interaction with the service provider.

[0088] Extensive network access: Capabilities are available through networks and accessed via standard mechanisms that facilitate the use of heterogeneous thin client or thick client platforms (e.g., mobile phones, laptops, and PDAs).

[0089] Resource pooling: A provider's computing resources are pooled to serve multiple consumers using a multi-tenant model, where different physical and virtual resources are dynamically assigned and reassigned as needed. There is a sense of location independence because consumers typically do not have control or knowledge of the exact location of the resources provided, but may be able to specify the location at a higher level of abstraction (e.g., country, state, or data center).

[0090] Rapid flexibility: The ability to provide capacity quickly and flexibly, automatically scaling down and up rapidly in some situations to scale up rapidly. For consumers, the available supply capacity often appears unlimited and can be purchased in any quantity at any time.

[0091] Measuring services: Cloud systems automatically control and optimize resource usage by leveraging metering capabilities at a level of abstraction appropriate to the service type (e.g., storage, processing, bandwidth, and active user accounts). Resource usage can be monitored, controlled, and reported, providing transparency to both service providers and consumers.

[0092] The service model is as follows:

[0093] Software as a Service (SaaS): This provides consumers with the ability to use the provider's applications running on cloud infrastructure. Applications can be accessed from different client devices via thin client interfaces such as web browsers (e.g., web-based email). Consumers do not manage or control the underlying cloud infrastructure, including the network, servers, operating system, storage, or even individual application capabilities, with possible exceptions such as limited user-specific application configuration settings.

[0094] Platform as a Service (PaaS): This provides consumers with the ability to deploy applications created or acquired by the consumer using programming languages ​​and tools supported by the provider onto cloud infrastructure. Consumers do not manage or control the underlying cloud infrastructure, including networks, servers, operating systems, or storage, but they have control over the deployed applications and the configuration of any application hosting environment.

[0095] Infrastructure as a Service (IaaS): The capabilities offered to consumers are processing, storage, networking, and other basic computing resources that enable consumers to deploy and run arbitrary software, which may include operating systems and applications. Consumers do not manage or control the underlying cloud infrastructure, but rather have control over the operating system, storage, deployed applications, and potentially limited control over selected networking components (e.g., host firewalls).

[0096] The deployment model is as follows:

[0097] Private cloud: A cloud infrastructure that operates solely for an organization. It can be managed by the organization or a third party and can exist on-site or off-site.

[0098] Community cloud: A cloud infrastructure shared by several organizations and supporting a specific community with shared concerns (e.g., tasks, security requirements, policies, and compliance considerations). It can be managed by an organization or a third party and can exist on-site or off-site.

[0099] Public cloud: Makes cloud infrastructure available to the public or large industry groups and is owned by an organization that sells cloud services.

[0100] Hybrid cloud: A cloud infrastructure is a combination of two or more clouds (private, community, or public) that remain a single entity but are bound together by standardized or proprietary technologies that enable data and applications to be ported (e.g., cloud bursting for load balancing between clouds).

[0101] Cloud computing environments are service-oriented, focusing on statelessness, loose coupling, modularity, and semantic interoperability. At the heart of cloud computing is the infrastructure comprising a network of interconnected nodes.

[0102] See now Figure 8 This describes an illustrative cloud computing environment 1000. As shown, the cloud computing environment 1000 includes one or more cloud computing nodes 100 that can communicate with local computing devices used by cloud consumers, such as, for example, personal digital assistants (PDAs) or cellular phones 1000A, desktop computers 1000B, laptop computers 1000C, and / or automotive computer systems 1000N. The nodes 100 can communicate with each other. They can be physically or virtually grouped (not shown) in one or more networks, such as private clouds, community clouds, public clouds, or hybrid clouds, or combinations thereof, as described above. This allows the cloud computing environment 1000 to provide infrastructure, platforms, and / or software as services that cloud consumers do not need to maintain on their local computing devices. It should be understood that... Figure 8 The types of computing devices 1000A-N shown are intended to be illustrative only, and computing node 100 and cloud computing environment 1000 can communicate with any type of computerized device via any type of network and / or network-addressable connection (e.g., using a web browser).

[0103] See now Figure 9 This illustrates a set of functional abstraction layers 1100 provided by the cloud computing environment 1000. It should be understood beforehand that... Figure 9 The components, layers, and functions shown are merely illustrative, and embodiments of the invention are not limited thereto. As described, the following layers and corresponding functions are provided:

[0104] The hardware and software layer 1102 includes hardware and software components. Examples of hardware components include: a mainframe 1104; a server 1106 based on a RISC (Reduced Instruction Set Computer) architecture; a server 1108; a blade server 1110; a storage device 1112; and a network and networking component 1114. In some embodiments, the software components include network application server software 1116 and database software 1118.

[0105] The virtualization layer 1120 provides an abstraction layer from which the following examples of virtual entities can be provided: virtual server 1122; virtual storage 1124; virtual network 1126, including virtual private network; virtual application and operating system 1128; and virtual client 1130.

[0106] In one example, management layer 1132 may provide the functionality described below: Resource Provisioning 1134 Provides dynamic procurement of computing resources and other resources used to perform tasks within the cloud computing environment. Metering and Pricing 1136 Provides cost tracking as resources are utilized within the cloud computing environment and bills or invoices for the consumption of these resources. In one example, these resources may include application software licenses. Security Provides authentication for cloud consumers and tasks, as well as protection for data and other resources. User Portal 1138 Provides consumers and system administrators with access to the cloud computing environment. Service Level Management 1140 Provides allocation and management of cloud computing resources to ensure that required service levels are met. Service Level Agreement (SLA) Planning and Fulfillment 1142 Provides pre-scheduling and procurement of cloud computing resources, anticipating future requirements for those resources according to the SLA.

[0107] Workload layer 1144 provides examples of functionalities that can leverage a cloud computing environment. Examples of workloads and functionalities that can be provided from this layer include: mapping and navigation 1146; software development and lifecycle management 1148; virtual classroom education delivery 1150; data analytics and processing 1152; transaction processing 1154; and hyperparameter tuning 1156. Hyperparameter tuning procedures 110a and 110b provide a way to scale resources and perform hyperparameter training on large-scale workloads using limited computing resources.

[0108] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used herein, unless the context clearly indicates otherwise, the singular forms “a,” “an,” and “the” are intended to also include the plural forms. It will be further understood that when the terms “comprises,” “comprising,” “includes,” “including,” “has,” “having,” “with,” etc., are used in this specification, they specify the presence of the stated feature, integral, step, operation, element, and / or component, but do not preclude the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or combinations thereof.

[0109] Various embodiments of the invention have been described for illustrative purposes, but are not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope of the described embodiments. The terminology used herein has been chosen to best explain the principles of the embodiments, their practical application, or technical improvements to technologies found in the market, or to enable those skilled in the art to understand the embodiments disclosed herein.

Claims

1. A computer-implemented method executed by one or more processors, the method comprising: Receive hyperparameter search input data, which includes the number of parallel experiments; Use a search algorithm to generate a set of hyperparameters; Physical resource data about a cluster is received at the hyperparameter tuning resource allocator, wherein the cluster includes at least one accelerator and at least one central processing unit, the at least one accelerator including accelerator memory and the at least one central processing unit including CPU memory. The required workload is determined via the hyperparameter tuner resource allocator, wherein the hyperparameter tuner resource allocator uses the number of received parallel experiments to determine the required workload; The workload batch size for whether to force parallel experiments is determined by analyzing the hyperparameter search input data via the hyperparameter tuner resource allocator. The number of parallel experiments are performed via the cluster to search for and tune the hyperparameter set, wherein the hyperparameter tuning resource allocator manages the cluster during the execution of the number of parallel experiments based on the physical resource data and the required workload, and by using tensor interchange; and Multiple virtual accelerators are created and managed within at least one accelerator via the hyperparameter tuning resource allocator to perform the number of parallel experiments, wherein specific virtual accelerators of the multiple virtual accelerators always access the same address space in the CPU memory for tensor swapping.

2. The method according to claim 1, wherein, In response to determining the workload batch size for the forced parallel experiment, the hyperparameter tuner resource allocator creates and manages virtual accelerators within the at least one accelerator.

3. The method according to claim 1, wherein, The hyperparameter tuner resource allocator uses a training controller to initiate the number of parallel experiments performed on the cluster.

4. The method according to claim 1, wherein, The hyperparameter tuning resource allocator manages the floating-point operations per second for the at least one accelerator.

5. The method according to claim 1, wherein, In response to determining that the workload batch size for parallel experiments is not mandatory, the hyperparameter tuner dynamically scales the batch size of the parallel experiments based on the required workload.

6. The method according to claim 1, wherein, The tensor swap includes transferring at least one tensor from the at least one accelerator to the at least one central processing unit, and includes the at least one central processing unit returning the at least one tensor to the at least one accelerator; and The at least one tensor is generated via the at least one accelerator.

7. A computer system for hyperparameter tuning, comprising: One or more processors, one or more computer-readable storage devices, one or more computer-readable tangible storage media, and program instructions stored on at least one of the one or more computer-readable tangible storage media for execution by at least one of the one or more processors via at least one of the one or more computer-readable storage devices, wherein the computer system is capable of performing a method including the following steps: Receive hyperparameter search input data, which includes the number of parallel experiments; Use a search algorithm to generate a set of hyperparameters; Physical resource data about a cluster is received at the hyperparameter tuning resource allocator, wherein the cluster includes at least one accelerator and at least one central processing unit, the at least one accelerator including accelerator memory and the at least one central processing unit including CPU memory. The required workload is determined via the hyperparameter tuner resource allocator, wherein the hyperparameter tuner resource allocator uses the number of received parallel experiments to determine the required workload; The workload batch size for whether to force parallel experiments is determined by analyzing the hyperparameter search input data via the hyperparameter tuner resource allocator. The number of parallel experiments are performed via the cluster to search for and tune the hyperparameter set, wherein the hyperparameter tuning resource allocator manages the cluster during the execution of the number of parallel experiments based on the physical resource data and the required workload, and by using tensor interchange; and Multiple virtual accelerators are created and managed within at least one accelerator via the hyperparameter tuning resource allocator to perform the number of parallel experiments, wherein specific virtual accelerators of the multiple virtual accelerators always access the same address space in the CPU memory for tensor swapping.

8. The computer system according to claim 7, wherein, In response to determining the workload batch size for the forced parallel experiment, the hyperparameter tuner resource allocator creates and manages virtual accelerators within the at least one accelerator.

9. The computer system according to claim 7, wherein, The hyperparameter tuner resource allocator uses a training controller to initiate the number of parallel experiments performed on the cluster.

10. The computer system according to claim 7, wherein, The hyperparameter tuning resource allocator manages the floating-point operations per second for the at least one accelerator.

11. The computer system according to claim 7, wherein, In response to determining that the workload batch size for parallel experiments is not mandatory, the hyperparameter tuner dynamically scales the batch size of the parallel experiments based on the required workload.

12. The computer system according to claim 7, wherein, The tensor swap includes transferring at least one tensor from the at least one accelerator to the at least one central processing unit, and includes the at least one central processing unit returning the at least one tensor to the at least one accelerator; and The at least one tensor is generated via the at least one accelerator.

13. A computer program product for hyperparameter tuning, comprising a computer-readable storage medium having program instructions contained therein, the program instructions being executable by a processor to cause the processor to perform a method comprising the steps of: Receive hyperparameter search input data, which includes the number of parallel experiments; Use a search algorithm to generate a set of hyperparameters; Physical resource data about a cluster is received at the hyperparameter tuning resource allocator, wherein the cluster includes at least one accelerator and at least one central processing unit, the at least one accelerator including accelerator memory and the at least one central processing unit including CPU memory. The required workload is determined via the hyperparameter tuner resource allocator, wherein the hyperparameter tuner resource allocator uses the number of received parallel experiments to determine the required workload; The workload batch size for whether to force parallel experiments is determined by analyzing the hyperparameter search input data via the hyperparameter tuner resource allocator. The number of parallel experiments are performed via the cluster to search for and tune the hyperparameter set, wherein the hyperparameter tuning resource allocator manages the cluster during the execution of the number of parallel experiments based on the physical resource data and the required workload, and by using tensor interchange; and Multiple virtual accelerators are created and managed within at least one accelerator via the hyperparameter tuning resource allocator to perform the number of parallel experiments, wherein specific virtual accelerators of the multiple virtual accelerators always access the same address space in the CPU memory for tensor swapping.

14. The computer program product according to claim 13, wherein, In response to determining the workload batch size for the forced parallel experiment, the hyperparameter tuner resource allocator creates and manages virtual accelerators within the at least one accelerator.

15. The computer program product according to claim 13, wherein, The hyperparameter tuner resource allocator uses a training controller to initiate the number of parallel experiments performed on the cluster.

16. The computer program product according to claim 13, wherein, The hyperparameter tuning resource allocator manages the floating-point operations per second for the at least one accelerator.

17. The computer program product according to claim 13, wherein, In response to determining that the workload batch size for parallel experiments is not mandatory, the hyperparameter tuner dynamically scales the batch size of the parallel experiments based on the required workload.

Images