Deep learning automatic tuning task optimization

Abstract

The present disclosure relates to deep learning auto-tuning task optimization. Systems and methods for improving the auto-tuning process are provided. For example, the system can implement a task launcher, a scheduler, and an agent to respectively launch, schedule, and execute the decomposed auto-tuning phases. The scheduling policy implemented by the scheduler can perform operations that exceed simple scheduling policies (e.g., FIFO-based scheduling policies) that produce high queuing delays. By leveraging the specific domain knowledge of auto-tuning, this can help reduce queuing delays and improve resource utilization in traditional systems.

Description

Background Art

[0001] Applications that require real-time performance, such as industrial monitoring or autonomous driving, are increasingly supported by deep learning (DL) or other artificial intelligence models. Dedicated hardware (such as graphics processing units (GPUs), optimized central processing units (CPUs), or field programmable gate arrays (FPGAs)) are used to accelerate or speed up the computationally intensive reasoning output by DL or other artificial intelligence models. However, these models are typically executed using device-specific low-level optimizations to fully exploit the potential of the hardware. Each of these hardware devices has different characteristics (such as dedicated instructions, memory size and layout, cache, and parallelization support).

[0002] Thus, optimization can vary parameters based on a variety of factors. These factors can include the relative ability of the hardware to acquire and interpret information, the speed at which information is transmitted through different parts of the hardware, or the speed at which outputs from traditional or vector processing cores are sent to the processing logic. These inference optimizations are often performed manually, which can be inefficient. When individuals optimizing accelerators lack a deep understanding of the hardware, manual optimization can also use processes that may miss certain low-level optimizations specific to the device. BRIEF DESCRIPTION OF THE DRAWINGS

[0003] The present disclosure according to one or more embodiments is described in detail with reference to the following drawings. These drawings are provided for illustration purposes only and merely depict typical or exemplary embodiments.

[0004] Figure 1A-1B An example of automatically tuning infrastructure according to an embodiment of the present application is provided.

[0005] Figure 2 An alternative example of an automatic tuning infrastructure according to an embodiment of the present application is provided.

[0006] Figure 3 Automatic tuning and advanced resource utilization according to an embodiment of the present application are shown.

[0007] Figure 4 Automatic tuning and advanced resource utilization according to embodiments of the present application are provided.

[0008] Figure 5 Exclusive access to resources by a scheduler circuit according to an embodiment of the present application is shown.

[0009] Figure 6 FIG. 4 shows a shortest job first (SJF) scheduler circuit for resources according to an embodiment of the present application.

[0010] Figure 7 The diagram shows the scheduling of a single automatic tuning task according to an embodiment of the present application.

[0011] Figure 8 The figure shows a computing component for providing a dynamic automatic tuning process according to an embodiment of the present application.

[0012] Figure 9 are exemplary computing components that may be used to implement the various features of the embodiments described in this disclosure.

[0013] The drawings are not exhaustive and do not limit the disclosure to the precise form disclosed. DETAILED DESCRIPTION

[0014] The embodiments described herein can improve the hardware-specific tuning process to extract maximum performance from available infrastructure resources. Automatic performance optimization (referred to as autotuning) creates optimized low-level implementations for specific hardware to improve inference performance without the need for vendor-independent human experts. Specifically, autotuning finds the best configuration for a loop block, which can be a combination of tunable parameters available for a specific hardware. Sorting, caching, and loop unrolling can be configured in the best configuration to reduce memory access costs, maximize parallelism (e.g., CUDA threads), and take advantage of novel hardware primitives (e.g., tensor cores) for the specific hardware.

[0015] For example, auto-tuning can define a design space and a code template. The design space can correspond to tunable parameters for a specific hardware. The code template can include all potential tunable parameters, either individually or in combination, including the number of blocks (e.g., tile_f, tile_y, tile_x, etc.), blocks and binding information for multiple filters, the height and width of the feature map, blocks and binding information for reducing axes (e.g., channels, filter height, width, etc.), "auto_unroll_max_step" corresponding to a threshold for the number of steps in the loop to be automatically expanded during code generation, and "unroll_explicit" corresponding to an explicit hint for expanding the loop during the code generation phase. The auto-tuning system can generate an executable file from a selected combination of tunable parameters and a code template. The selected combination can include loop blocks, sorting, and / or caching for implementation. The auto-tuning system can repeat these processes until the optimal configuration is found. For example, the auto-tuning system can measure the performance of the generated executable file based on each combination of tunable parameters.

[0016] These operations can directly affect the efficiency of machine learning (ML) or other artificial intelligence (AI) inference calculations. For example, several ML and AI calculations are matrix operations (e.g., addition and multiplication), and the speed of these operations generally affects the overall computation time of the system. Because loop blocks, sorting, and caches can determine the performance of inference calculations on specialized hardware, finding the optimal configuration can be important for improving the overall efficiency of the system.

[0017] In the context of deep learning, auto-tuning a deep learning model refers to optimizing the model to improve its efficiency when running on a specific hardware architecture, such as a specific type of graphics processing unit (GPU) or central processing unit (CPU). Furthermore, auto-tuning optimizes each trained model for a specific hardware based on specific configuration parameters (e.g., quantization, auto-tuner type, etc.). Unlike manual optimization, auto-tuning fosters innovation and helps manage the growing demand for performance across a growing variety of models and accelerator devices.

[0018] In the automatic tuning framework, these processes use a sequential operation consisting of four stages to find the optimal low-level implementation before finding the optimal implementation. The first stage can select a batch of candidate implementations. For example, the process can select a batch of candidates in the search space based on the results from the exploration algorithm that utilizes the ML cost model managed in the update cost model stage. In the absence of initial training data, random candidates can be selected. The second stage can include a construction stage. For example, the process can generate one or more executable files based on the batch of candidates. The third stage can include a profiling stage, which can run one or more executable files and measure the execution time on the target device. The fourth stage can update the model and use machine learning (ML) and / or search exploration algorithms to find the next promising configuration candidate.

[0019] Although existing automatic tuning frameworks may exhibit good inference performance, they have fundamental design limitations on the automatic tuning process. For example, in existing frameworks, the automatic tuning process is iteratively executed as sequential operations in asynchronous stages. Moreover, these sequential operations can be run on different computing resources. However, when such a sequential automatic tuning process is run on different computing resources (e.g., CPU, GPU, etc.), inefficient computing resource utilization may result. Specifically, since each stage has a strong dependency, in existing automatic tuning frameworks, there is always one computing resource that is idle. For example, when the third stage (e.g., the simplification stage) is executed on the GPU, the CPU resources are idle because the fourth stage (e.g., the update stage) executed on the CPU requires the output data from the third stage.

[0020] To avoid this inefficient resource utilization, multiple auto-tuning tasks can be run on the same resource. Typically, a task can include processes running on the device's hardware, including data dependencies, that are handled by the runtime system rather than the application-layer software. However, if tasks are not carefully scheduled, performance can degrade due to task interference. Furthermore, because existing auto-tuning frameworks have a rigid and monolithic design that continuously iterates each stage based on an initial auto-tuning configuration, the ability to coordinate different auto-tuning tasks until the entire auto-tuning task is completed can be limited.

[0021] Embodiments of the present disclosure address the aforementioned limitations of existing auto-tuning frameworks by providing a redesigned auto-tuning framework that maximizes resource utilization, reduces overall auto-tuning completion time, and reduces the cost of auto-tuning. Embodiments of the present invention achieve this at least in part by decomposing the existing monolithic and rigid auto-tuning process into a set of functional sub-processes and scheduling the functional sub-processes to simultaneously schedule phases requiring different resources. For a particular resource, exclusive resource access can be guaranteed to only one phase at a time, avoiding incorrect profile measurements and performance degradation due to interference.

[0022] Some embodiments of the present disclosure may include a system that includes a task launcher, a scheduler, and an agent to respectively launch, schedule, and execute decomposed auto-tuning stages. The scheduling policy implemented by the scheduler can perform better than simple scheduling policies (e.g., FIFO-based scheduling policies) that produce high queuing delays. By automatically adapting auto-tuning specific domain knowledge (e.g., the state of auto-tuning tasks and resources and their relationships), this can help reduce queuing delays and improve resource utilization that exists in traditional systems. For example, the scheduler can achieve this in three ways: 1) leveraging fine-grained domain knowledge (e.g., the cost of each stage in terms of execution time, update model interval, and queue state) through a shortest job first (SJF) scheduling policy to mitigate long queuing delays, 2) dynamically adjusting parameters of each stage to reduce queuing delays, and 3) leveraging hardware-based multi-process service (MPS) capabilities to help ensure isolation of active threads from multiple GPU applications, reusing a GPU for multiple profiled stages (e.g., in the case where inference services are deployed on a shared GPU, etc.).

[0023] In an example embodiment of the present disclosure, a scheduler may receive initial auto-tuning parameters from a job manager and generate a schedule of auto-tuning tasks based on the initial auto-tuning parameters. The scheduler may repeatedly schedule multiple layers in parallel, or multiple layers in a single auto-tuning job, to update and improve auto-tuning across the layers. Additional details of the task launcher, scheduler, and agent are provided throughout the disclosure.

[0024] Figure 1A-1B Provides examples of automatically tuning infrastructure. Figure 1A The machine learning (ML) lifecycle is shown. The lifecycle can start with training 110 an ML model. This can generate an ML model based on data. To improve inference performance (e.g., low execution time for ML inference), inference optimization 120 can be implemented based on the ML trained model generated from training 110. Inference optimization 120 can be implemented before deploying the ML inference stage for a specific target device and can include automatic tuning. In inference optimization 120, high-level transformations rewrite the computational graph, for example by fusing tensor operators, pre-computing constant parts, or transforming the data layout in memory. These high-level graph optimizations can be applied first, and then the optimal low-level implementation is executed on the modified graph on the specific hardware. The last stage is inference 130, which services inference requests based on the trained model and automatic tuning.

[0025] Figure 1B An exemplary automatic tuning infrastructure for inference optimization 120 is shown, which may include three components: a builder 150, a tracker 160, and a runner 170. Builder 150, tracker 160, and runner 170 may collaborate to complete the automatic tuning task. For example, the automatic tuning framework performed by these components may include four phases as discussed herein. The first phase may select a batch of candidates for implementation. For example, the process may select a batch of candidates in the search space based on the results of an exploration algorithm utilizing the ML cost model managed in the update cost model phase. In the absence of initial training data, random candidates may be selected. The second phase may include a build phase. For example, the process may generate one or more executable files based on the batch of candidates. The third phase may include a profile phase, which may run one or more executable files and measure execution time on a target device. The fourth phase may update the model and use machine learning (ML) and / or search exploration algorithms to find the next promising configuration candidate.

[0026] Builder 150 may execute the first, second, and fourth phases, while runner 170 may execute the third phase. Tracker 160 may track resource availability (e.g., GPU) in runner 170 and help builder 150 discover unused resources for profiling. When runner 170 is launched on the target device, runner 170 may contact tracker 160 and register with tracker 160. Tracker 160 then matches incoming requests from builder 150 and uses a FIFO-based scheduling algorithm to allocate available resources to builder 150.

[0027] Figure 1A-1B The auto-tuning process shown is monolithic and continuously iterates each stage based on the initial auto-tuning configuration (e.g., model, hardware, quantization) until the entire auto-tuning task is completed. These monolithic and sequential stages may make fine-grained management and deployment of auto-tuning stages infeasible. Therefore, in some embodiments of the present disclosure, an improved auto-tuning process is described that decomposes these components and processes (e.g., builder 150, tracker 160, and runner 170) into a set of functional sub-processes and schedules the functional sub-processes so that stages requiring different resources are scheduled simultaneously to optimize and improve the automated auto-tuning process.

[0028] Figure 2 An alternative example of an automatic tuning infrastructure according to an embodiment of the present application is provided. The automatic tuning infrastructure can be decomposed into schedulable units, which include a scheduler circuit 230 and an agent circuit 220 to start, schedule, and execute the decomposed automatic tuning phases. In some examples, the scheduler circuit 230 can implement the task launcher circuit 210 inside the scheduler circuit 230, or the task launcher circuit 210 can be external (e.g., a called service or device, etc.). The decomposition of the scheduler circuit 230 and the agent circuit 220 can enable communication between each phase and the scheduler circuit 230 to exchange metrics or other information, as well as dynamic adjustment of each phase (e.g., the interval for updating the model and the number of simplifications, etc.) to facilitate context-aware automatic tuning scheduling.

[0029] The task launcher circuitry 210 is configured to submit tasks or jobs to the agent circuitry 220. The task launcher circuitry 210 may reside with the scheduler circuitry 230 to initiate launching of tasks on the agent circuitry 220 based on computing resource availability information, which may be determined and / or stored by the scheduler circuitry 230.

[0030] The agent circuit 220 may run on a resource (e.g., a device having a GPU or CPU, etc.) and act as an intermediary between the resource and the automatic tuning infrastructure including the scheduler circuit 230. The agent circuit 220 may enable the task launcher circuit 210 to submit and manage computations on the resource.

[0031] The agent circuit 220 may manage network connections between multiple circuits including the agent circuit 220 and the scheduler circuit 230 .

[0032] In some examples, the task launcher circuit 210 may receive one or more automatic tuning requests from a user. The task launcher circuit 210 and the scheduler circuit 230 may exchange information to find available resources (e.g., specified hardware including a GPU, etc.). The task launcher circuit 210 may send a job launch request with parameters (e.g., DNN model, precision (e.g., float32, float8, int), etc.) to the agent circuit 220.

[0033] The proxy circuit 220 may receive an automatic tuning task from the task launcher circuit 210 and add information to the request with the automatic tuning task. For example, since the proxy circuit 220 is located at the resource, the proxy circuit 220 may access hardware or software information about the resource and include the hardware or software information in the request. The information may include information about the GPU and / or automatic tuning options. As an illustrative example, the proxy circuit 220 may include GPU-specific information (e.g., "max_shared_memory_per_block", "max_threads_per_block", "max_thread_x", "max_thread_y", "max_thread_z", "cuda_arch", "sm_50", etc.) and / or automatic tuning specific parameters (e.g., the interval of the "update model" phase, the number of layers in the model, the search space corresponding to the model, the number of build executables and profile stages running simultaneously, etc.), etc.

[0034] The agent circuitry 220 may send a request to the scheduler circuitry 230 to enable automatic tuning of the task registers, and receive a response from the scheduler circuitry 230 .

[0035] The agent circuit 220 may include a job manager. The job manager at the agent circuit 220 of the resource may receive a scheduling request message associated with the scheduling of a task from the scheduler circuit 230. The scheduling request message may include the automatic tuning phase to be executed by the job manager at the agent circuit 220. After the execution phase, the job manager at the agent circuit 220 sends a scheduling response message to the scheduler circuit 230 including the success or failure of the execution and the time to run those pages.

[0036] Scheduler circuitry 230 can implement scheduling strategies beyond simple scheduling strategies (e.g., first-in, first-out (FIFO)) to reduce high queuing delays. The delays can correspond to automatic tuning based on specific domain knowledge. Domain knowledge can include the states of automatically tuned tasks and resources and their relationships identified by agent circuitry 220.

[0037] The scheduler circuit 230 is a context-aware auto-tuning scheduler that manages decomposed schedulable units and different computing resources (e.g., CPU, GPU) according to multiple or single auto-tuning tasks. To be context-aware, the scheduler circuit 230 receives the status of the auto-tuning tasks (e.g., current stage, average runtime of each stage) and resources (e.g., GPU, CPU availability) and their relationships (i.e., mapping each stage to a specific resource). To track the context, the scheduler circuit 230 and the agent circuit 220 send and share information to give the scheduler circuit 230 a global view of multiple auto-tuning tasks.

[0038] The scheduler circuit 230 may schedule the next phase (e.g., selection, construction, profiling, and updating) for the automatic tuning task, and the job manager at the agent circuit 220 may maintain these configurations. The scheduler circuit 230 may schedule each phase of the automatic tuning task based in part on the execution time returned for each phase. The job manager at the agent circuit 220 may execute the phase scheduled from the scheduler circuit 230.

[0039] During the loop, the system can form multiple auto-tuning states into complete processes, and the scheduler circuit 230 can schedule tasks based on resource availability and other auto-tuning work. The scheduler circuit 230 can guarantee exclusive access to resources so that each task can be fully executed on the resources. Figures 4 to 6 Additional details are provided.

[0040] The scheduler circuit 230 can use auto-tuning specific knowledge (e.g., the state of auto-tuning tasks and resources and their relationships) to improve resource utilization and significantly reduce queuing delays. The scheduler circuit 230 obtains the auto-tuning specific domain context from the agent circuit 220. The auto-tuning specific domain context includes the scheduled phase, the parameters for running the phase, and fine-grained metrics, so as to make optimal scheduling decisions based on them.

[0041] The scheduler circuit 230 can mitigate long queuing delays by 1) leveraging fine-grained domain knowledge (e.g., the cost of each stage in terms of execution time, update model interval, and queue status) with a shortest job first (SJF) scheduling policy; 2) dynamically adjusting parameters of each stage to reduce queuing delays; and 3) leveraging hardware-based multi-process service (MPS) capabilities to help ensure isolation of active threads from multiple GPU applications to reuse one GPU for multiple reduced-profile stages when inference services are deployed on a shared GPU.

[0042] In some examples, multiple auto-tuning tasks can be scheduled, such as Figure 2 As shown in boxes 1 to 5 in the figure.

[0043] At block 1, the task initiator circuit 210 may receive a new auto-tuning task. The task initiator circuit 210 may initiate the auto-tuning task by sending an electronic communication to the agent circuit 220. The task initiator circuit 210 sends information (e.g., auto-tuned deep neural network (DNN) model, precision (e.g., float32, float8, int)) to the agent circuit 220.

[0044] At block 2, the agent circuitry 220 may register a new job with the scheduler circuitry 230. The job manager in the agent circuitry 220 may send a registration request with auto-tuning options to the scheduler circuitry 230. The auto-tuning options may include, for example, automatically tuning specific parameters (e.g., the interval of the "update model" phase, the number of layers in the model, the search space of the model, the number of simultaneous executable builds and profiles, etc.). In some examples, the model and quantization information are sent from the task launcher circuitry 210 to the agent circuitry 220.

[0045] At block 3, the scheduler circuitry 230 may begin scheduling a new job based on the current scheduling policy (e.g., FIFO, SJF, etc.). When a new task is scheduled, a scheduling request from the scheduler circuitry 230 to the agent circuitry 220 is sent to the job manager in the agent circuitry 220. If an auto-tuning task is required to adjust auto-tuning parameters (e.g., the interval for updating the model) in order to more efficiently schedule multiple auto-tuning tasks, this information may also be included in the scheduling request.

[0046] At block 4 , the job manager in the agent circuitry 220 may execute the functionally decomposed sub-processes based on the scheduling request.

[0047] At block 5, the agent circuitry 220 and the scheduler circuitry 230 may update the auto-tuning task metrics. Blocks 3-5 may be repeated for each submitted auto-tuning task.

[0048] Dynamic parameters for adjusting the auto-tuning task can be implemented, including the number of profiles and executable files generated, as well as the intervals for updating model operations, retraining, and exploration. For example, when the auto-tuning task is launched, it configures multiple inputs for each stage (e.g., the interval for updating the model, the number of executable files analyzed and generated, etc.). In traditional systems, these inputs can persist until the auto-tuning task completes. When many tasks are in the target device queue (e.g., GPU) for the profile phase, this can lead to head-of-line (HOL) blocking. For example, when two auto-tuning tasks are waiting to run the profile phase on a single GPU resource, the second task should wait for the first task to complete the profile phase. During this time, the CPU resources are idle. In some embodiments described herein, dynamic parameters are implemented to address this issue. Parameters can be dynamically adjusted by considering the stage of the task and the state of the queue. Scheduler circuitry 230 can send a profile scheduling message with a small amount of profile to the first task. The first task can complete the profile phase earlier than the previous profile phase and utilize CPU resources. At the same time, the second task occupies GPU resources early to run the profiling phase. As a result, the overall resource utilization is increased, which leads to a reduction in the total auto-tuning completion time for multiple tasks.

[0049] In some examples, in addition to exclusive access to the target device (e.g., GPU) for the reduction phase and exclusive access to the CPU for other stages, the GPU can also be shared for multiple reductions. Exclusive access to the target device can help ensure correct measurements by avoiding interference and performance degradation from other reduction phases. Although the system can guarantee exclusive access to the target device, which can help ensure correct measurements by avoiding interference and performance degradation from other reduction phases, in some examples, in addition to exclusive access to the target device (e.g., GPU) for the reduction phase, the GPU can also be shared for multiple reductions. This can be useful when the automatic tuning inference model service is deployed on a shared GPU.

[0050] In some examples, hardware-based multi-process services (MPS) can be implemented to isolate computing resources (i.e., active thread percentages) between different applications on a GPU. Multiple reduction phases can be run simultaneously to increase target resource utilization and reduce auto-tuning completion time. For example, the scheduler circuit 230 can be configured to determine the configuration of the target device on an MPS-enabled target device. The scheduler circuit 230 can treat multiple logical GPUs divided by the active thread percentages from the MPS-enabled GPU as separate GPU resources and schedule the multiple reduction phases to the logical GPUs.

[0051] In some examples, a context-aware scheduler based on short job first (SJF) can be implemented as scheduler circuit 230 instead of a FIFO-based scheduler. This can reduce queuing delays caused by head-of-line (HOL) blocking problems. For example, the system can correspond to strong process dependencies (i.e., selecting, building, simplifying, and updating models) in the automatic tuning tasks and their different execution times. When two automatic tuning tasks are waiting to run the update model phase on one CPU resource, the second phase can wait for the completion of the first phase update model phase. When the update model phase ends in the first phase, the second phase occupies CPU resources, and since the first phase must first complete the selection and construction phases, it hinders the first phase from using GPU resources for the simplifying phase. When the update model performs model retraining and exploration algorithms, both phases require much shorter time than the update model phase. Since the FIFO-based scheduler does not take this fine-grained context into account, this will lead to HOL blocking problems, resulting in increased queuing delays and resource idle time for automatic tuning.

[0052] To address this issue, the scheduler circuit 230 can be implemented as a context-aware scheduler based on Short Job First (SJF). The context-aware scheduler based on SJF can prioritize some stages that require short completion times over stages with long completion times. Sample code is provided here to illustrate the context-aware scheduler based on SJF.

[0053]

[0054]

[0055] The SJF-based context-aware scheduler can utilize fine-grained context, such as the interval between the update model and the queued state, in addition to the process-granular context (e.g., the current stage of the auto-tuning task, the resources required for the stage, and the resource availability). The SJF-based context-aware scheduler allows the task of the last completed stage to be preempted, provided that the last completed stage is the profiling stage and the stage of the first ready task in the queue is the update model, which will execute the retraining and exploration algorithms. Therefore, the selection and construction stages of the last completed task are scheduled first. After the execution of the two stages, the task is scheduled for the profile on the GPU, and at the same time the preempted task starts the update model on the CPU resources. This can effectively avoid long queuing delays.

[0056] Figure 3 Automatic tuning and advanced resource utilization according to an embodiment of the present application are shown. For example, Figure 3The resource idle time caused by strong dependencies between stages in one auto-tuning task in a traditional auto-tuning system can be shown. This can highlight resource inefficiencies (e.g., long idle time) in a traditional auto-tuning system when only one auto-tuning task is running.

[0057] Compared to resource utilization 320, stages 310, selection and construction stage 312, reduction stage 314, and update stage 316 of a conventional automatic tuning system are shown. For example, in a conventional automatic tuning system, stage 310 may be sequentially executed by builder circuitry 322 and runner circuitry 324 to implement automatic tuning of CPU and GPU devices. When builder circuitry 322 completes processing of construction stage 312, runner circuitry 324 may initiate processing of reduction stage 314, where each stage begins after the previous stage ends.

[0058] Figure 4 Automatic tuning and advanced resource utilization according to embodiments of the present application are provided. For example, Figure 4 This can include solutions Figure 3 An illustrative solution to the inefficiency shown in

[15] may involve interleaving multiple autotuning tasks (or multiple layers from one autotuning task) with an exclusive shortest job first scheduler.

[0059] The improved automatic tuning system stage 410 supporting interleaving is shown compared to resource utilization 420. Figure 4 As shown, two simultaneous automatic tuning tasks T1 and T2 are available in the scheduler circuit 230. The scheduler circuit 230 can associate the tasks of each stage with the available resources. For example, the selection and construction stage 412, the simplification stage 414, and the update stage 416 of each task can be intertwined to run together to achieve automatic tuning of CPU and GPU devices. The scheduling of tasks can be aware of the task load to determine which task is in which stage and which task will use each resource (for example, based on the correspondence and simplification of the agent circuit 220, etc.). As determined by the scheduler circuit 230, the interleaved stages can be implemented on the available resources. In some examples, the dependencies between the stages 410 can be maintained.

[0060] exist Figure 5 and Figure 6 In Figure 1, a round-robin scheduling approach is shown. Multiple auto-tuning tasks scheduled in a traditional system using round-robin scheduling (e.g., time-sharing of GPU resources) can result in high idle time. This can be compared to methods using exclusive and SJF allocations. This reduces idle time and improves resource utilization.

[0061] Figure 52 shows the exclusive access to resources by the scheduler circuit according to an embodiment of the present application. For example, the scheduling circuit 230 can guarantee exclusive access to resources of a stage to avoid incorrect profiling measurements and performance degradation due to interference. Figure 5 As shown, two simultaneous auto-tuning tasks (T1 and T2) can be used for scheduling 510. Figure 5 The different shading of the boxes shows which resources will run the tasks, and the first and second rows can correspond to the scheduling phase of each task (e.g., profile, update, etc.).

[0062] Two types of scheduling are available, including round-robin scheduling 520 and exclusive allocation 530. In round-robin scheduling 520, tasks run on resources, each of which has significant idle time. After the build phase, T1 should run the next task on the next resource, but without exclusive access, tasks can share GPU resource time through time multiplexing (for example, to run the reduction phase). This may result in idle time for each of tasks T2 and T1 in order to complete the reduction phase. In another example, exclusive allocation 530 may be implemented (for example, for the reduction phase). T1 does not implement time multiplexing of GPU resources, but waits for the completion of T2's reduction state because the scheduler does not allow time sharing. The description of resource utilization only includes a short idle time, but both will continue to work on different resources. As shown, exclusive allocation 530 may include less idle time than round-robin scheduling 520. This may require less time to complete the automatic tuning task, and resource utilization is high. For example, Figure 5 The diagram in shows that the ends of the rectangles in the exclusive allocation 530 are shorter than the ends of the rectangles in the round-robin allocation 520 .

[0063] Figure 6 The shortest job first of the resource scheduler circuit according to an embodiment of the present application is shown. For example, the scheduling circuit 230 can implement the shortest job first (SJF) to prioritize some stages that require shorter completion times over stages with longer completion times. This can allow the scheduling circuit 230 to utilize fine-grained context, such as the interval between the update model and the queue state in addition to the process-grained context (e.g., the current stage of the automatically tuned task, the resources required for that stage, and the availability of resources).

[0064] The context-aware scheduler based on SJF can preempt tasks in the last completed phase, provided that the last completed phase is the profiling phase and the first ready task in the queue is in the update model phase, which will execute the retraining and exploration algorithms. In some examples, the last completed task's selection and construction phases can be scheduled first. After executing both phases, the task is scheduled for profiling on the GPU, while the preempted task starts the update model on the CPU resources. Using the context-aware scheduler based on SJF, the scheduling circuit 230 effectively avoids long queuing delays.

[0065] In some examples, the scheduling methods for reducing resource idle time and avoiding interference from sharing resources between multiple auto-tuning tasks simultaneously are applicable to other use cases where applications need to be executed sequentially on different resources (e.g., CPU, GPU) to complete their tasks. In some examples, other sophisticated scheduling methods (e.g., machine learning-based scheduling) can be used to maximize resource utilization while maintaining low auto-tuning completion time without departing from the present disclosure.

[0066] like Figure 6 As shown, two automatic tuning tasks are available for scheduling 610, including a reduction phase T1 and an update phase T2. Different processes can be used to schedule the tasks, including round-robin scheduling 620 and shortest job first (SJF) 630. As shown by the length of each rectangle, in any scheduling implementation, the update phase can be estimated to take longer than the other phases, and the reduction phase can take less time than the update phase.

[0067] In the round-robin scheduling approach 620, the scheduling circuitry 230 may schedule tasks as they appear in the request queue. The estimated duration to complete tasks (eg, as identified by the agent circuitry 220) may not be relevant in scheduling these tasks.

[0068] In Shortest Job First (SJF) 630, the scheduling circuitry 230 may schedule shorter tasks before tasks that are estimated to take longer (e.g., as identified by the proxy circuitry 220). In this example, a scheduled task with an update phase in T1 that has a longer estimated processing time may be scheduled after the selection and build phase in T2 is completed. When the update phase in T1 is queued earlier in the scheduling circuitry 230 than the selection and build phase in T2, the scheduling circuitry 230 schedules the phase in T2 first. With this scheduling, T1 and T2 can utilize CPU and GPU resources for updating and profiling in parallel. Compared to the round-robin scheduling approach, SJF exhibits less resource idle time and short self-tuning completion time.

[0069] Figure 7The number of layers in each CNN model is shown. For example, each layer in a deep learning (DL) model is independent for self-tuning, and the system can initiate self-tuning for each layer simultaneously, although it may not fully utilize transfer learning in the ML cost model between layers. The system may only receive one submitted auto-tuning task. As described herein, the scheduler circuit 230 can interact with the agent circuit 220 to run auto-tuning of multiple layers simultaneously from the auto-tuning task and schedule them.

[0070] Figure 8 An exemplary iterative process performed by a computing component 800 is shown for providing a dynamic automatic tuning process. The computing component 800 may be a server computer, a controller, or any other similar computing component capable of processing data. Figure 8 In an example implementation of , the computing component 800 includes a hardware processor 802 and a machine-readable storage medium 804. In some embodiments, the computing component 800 may include Figure 2 An embodiment of a system of task initiator circuitry 210, agent circuitry and / or scheduler circuitry 230.

[0071] The hardware processor 802 may be one or more central processing units, semiconductor-based microprocessors, and / or other hardware devices suitable for retrieving and executing instructions stored in the machine-readable storage medium 804. The hardware processor 802 may fetch, decode, and execute instructions, such as instructions 806-812, to control processing or operations for optimizing the system during runtime. As an alternative to or in addition to retrieving and executing instructions, the hardware processor 802 may include one or more electronic circuits including electronic components (such as a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), or other electronic circuits) for performing the functions of one or more instructions.

[0072] A machine-readable storage medium, such as machine-readable storage medium 804, can be any electronic, magnetic, optical, or other physical storage device that contains or stores executable instructions. Thus, machine-readable storage medium 804 can be, for example, random access memory (RAM), non-volatile RAM (NVRAM), electrically erasable programmable read-only memory (EEPROM), a storage device, an optical disk, or the like. In some embodiments, machine-readable storage medium 804 can be a non-transitory storage medium, where the term "non-transitory" does not include transient propagating signals. As described in detail below, machine-readable storage medium 804 can be encoded with executable instructions (e.g., instructions 806-812).

[0073] In some examples, instructions 806-812 can be implemented by a computer system to perform a set of automatic tuning tasks on a machine learning (ML) model. The computer system can include a memory storing a scheduler circuit, wherein the scheduler circuit operates according to machine-executable instructions (e.g., instructions 806-812) and one or more processors configured to access the memory and execute the machine-readable instructions.

[0074] Hardware processor 802 may execute instructions 806 to initiate the first portion of the task. The task may be initiated using the GPU or the CPU based on a scheduling policy.

[0075] Hardware processor 802 may execute instructions 808 to schedule the second portion of the task. The task may be scheduled using the GPU or CPU based on a scheduling policy. In some examples, starting with the first portion of the task, the GPU or CPU may be scheduled to be utilized when the GPU or CPU is not being used.

[0076] The hardware processor 802 may execute instructions 810 to receive an update (e.g., one update, multiple updates, etc.) to an auto-tuning task metric (e.g., one auto-tuning task metric, multiple auto-tuning task metrics, etc.) based on the execution of a scheduled auto-tuning phase. The receiving may be based on a job manager of the computer system, which executes the scheduling phase and measures the execution time associated with the execution of the scheduled auto-tuning phase. The scheduled auto-tuning phase may include a selection phase, a construction phase, a reduction phase, or an update phase.

[0077] Hardware processor 802 may execute instructions 812 to update the auto-tuning task metric based at least in part on the received update.

[0078] In some examples, a computer system is configured to optimize inference performance of a machine learning (ML) model for a specific hardware configuration.

[0079] In some examples, the machine-readable instructions are further for receiving, by the scheduler circuitry, a registration request with an auto-tuning option from the job manager.

[0080] In some examples, the scheduler circuitry utilizes exclusive resource access for a particular resource only at one stage at a time.

[0081] In some examples, the scheduler circuit utilizes Shortest Job First (SJF) to prioritize a first phase requiring a short completion time over a second phase having a longer completion time.

[0082] In some examples, a multi-process service (MPS) is implemented to isolate computing resources.

[0083] Figure 9A block diagram of an exemplary computer system 900 is shown in which various embodiments described herein may be implemented. The computer system 900 includes a bus 902 or other communication mechanism for communicating information, and one or more hardware processors 904 coupled with the bus 902 for processing information. The hardware processors 904 may be, for example, one or more general-purpose microprocessors.

[0084] The computer system 900 also includes a main memory 906, such as a random access memory (RAM), a cache, and / or other dynamic storage device, coupled to the bus 902 for storing information and instructions to be executed by the processor 904. The main memory 906 may also be used to store temporary variables or other intermediate information during execution of instructions by the processor 904. Such instructions, when stored in a storage medium accessible to the processor 904, present the computer system 900 as a special-purpose machine customized to perform the operations specified in the instructions.

[0085] The computer system 900 also includes a read-only memory (ROM) 908 or other static storage device coupled to the bus 902 for storing static information and instructions for the processor 904. A storage device 910, such as a magnetic disk, optical disk, or USB thumb drive (flash drive), is provided and coupled to the bus 902 for storing information and instructions.

[0086] The computer system 900 may be coupled to a display 912, such as a liquid crystal display (LCD) (or touch screen), via bus 902 for displaying information to a computer user. An input device 914, including alphanumeric and other keys, is coupled to bus 902 for communicating information and command selections to processor 904. Another type of user input device is a cursor controller 916, such as a mouse, trackball, or cursor direction keys, for communicating directional information and command selections to processor 904 and for controlling cursor movement on display 912. In some embodiments, directional information and command selections similar to cursor control may be achieved by receiving touches on a touch screen without a cursor.

[0087] The computing system 900 may include a user interface module to implement a GUI, which may be stored in a mass storage device as executable software code executed by the computing device. For example, this module and other modules may include components such as software components, object-oriented software components, class components and task components, processes, functions, properties, procedures, subroutines, program code segments, drivers, firmware, microcode, circuits, data, databases, data structures, tables, arrays, and variables.

[0088] In general, as used herein, the terms "component," "engine," "system," "database," "data store," and the like may refer to logic embodied in hardware or firmware, or to a collection of software instructions, which may have entry and exit points, written in a programming language (e.g., Java, C, or C++). Software components may be compiled and linked into an executable program, installed in a dynamic link library, or may be written in an interpreted programming language (e.g., BASIC, Perl, or Python). It should be understood that software components may be called from other components or from themselves, and / or may be called in response to detected events or interrupts. Software components configured to execute on a computing device may be provided on a computer-readable medium (such as a compact disk, digital video disk, flash drive, magnetic disk, or any other tangible medium), or as a digital download (and may be stored originally in a compressed or installable format that requires installation, decompression, or decryption prior to execution). Such software code may be stored in part or in whole on a memory device of the executing computing device for execution by the computing device. Software instructions may be embedded in firmware (such as an EPROM). It will be further understood that hardware components may include connected logic units (such as gates and flip-flops), and / or may include programmable units (such as programmable gate arrays or processors).

[0089] The computer system 900 can implement the techniques described herein using custom hard-wired logic, one or more ASICs or FPGAs, firmware, and / or program logic that, in combination with the computer system, makes the computer system 900 a special-purpose machine or programs the computer system 900 to be a special-purpose machine. According to one embodiment, the techniques herein are performed by the computer system 900 in response to the processor 904 executing one or more sequences of one or more instructions contained in the main memory 906. Such instructions can be read into the main memory 906 from another storage medium, such as the storage device 910. Execution of the sequences of instructions contained in the main memory 906 causes the processor 904 to perform the process steps described herein. In alternative embodiments, hard-wired circuitry can be used in place of or in combination with software instructions.

[0090] As used herein, the term "non-transitory medium" and similar terms refer to any medium that stores data and / or instructions that cause a machine to operate in a particular manner. Such non-transitory media may include non-volatile media and / or volatile media. Non-volatile media include, for example, optical or magnetic disks (such as storage device 910). Volatile media include dynamic memory (such as main memory 906). Common forms of non-transitory media include, for example, floppy disks, diskettes, hard disks, solid-state drives, magnetic tape or any other magnetic data storage medium, CD-ROMs, any other optical data storage medium, any physical medium with a pattern of holes, RAM, PROM and EPROM, flash EPROM, NVRAM, any other memory chip or cartridge, and networked versions thereof.

[0091] Non-transitory media are distinct from, but may be used in conjunction with, transmission media. Transmission media participate in the transmission of information between non-transitory media. Examples of transmission media include coaxial cables, copper wire, and optical fiber, including the wires that comprise bus 902. Transmission media may also take the form of acoustic or light waves, such as those generated during radio wave and infrared data communications.

[0092] Computer system 900 also includes a communication interface 918 coupled to bus 902. Communication interface 918 provides two-way data communication coupled to one or more network links connected to one or more local area networks. For example, communication interface 918 can be an integrated services digital network (ISDN) card, a cable modem, a satellite modem, or a modem that provides data communication connections to the telephone line of the corresponding type. As another example, communication interface 918 can be a local area network (LAN) card to provide data communication connections to a compatible LAN (or a WAN component communicating with a WAN). Wireless links can also be implemented. In any such implementation, communication interface 918 sends and receives electrical, electromagnetic, or optical signals that carry digital data streams representing various types of information.

[0093] A network link typically provides data communication to other data devices through one or more networks. For example, a network link can provide a connection to a host computer or to data equipment operated by an Internet Service Provider (ISP) through a local area network. The ISP, in turn, provides data communication services through the global packet data communication network now commonly referred to as the "Internet." Both local area networks and the Internet use electrical, electromagnetic, or optical signals to transmit digital data streams. Signals through various networks, as well as signals on network links and signals through communication interface 618, are example forms of transmission media that carry digital data to and from computer system 900.

[0094] Computer system 900 can send messages including program code and receive data through the network, network link, and communication interface 918. In the Internet example, a server can send the requested code for an application program through the Internet, ISP, local area network, and communication interface 918.

[0095] The received code may be executed by processor 904 as it is received, and / or stored in storage device 910 or other non-volatile storage for later execution.

[0096] Each of the processes, methods, and algorithms described in the preceding sections may be embodied in code components executed by one or more computer systems or computer processors comprising computer hardware, and may be fully or partially automated by these code components. One or more computer systems or computer processors may also operate to support the performance of related operations in a "cloud computing" environment or as "software as a service" (SaaS). The processes and algorithms may be implemented in part or in whole in dedicated circuit devices. The various features and processes described above may be used independently of one another or may be combined in various ways. Different combinations and sub-combinations are intended to fall within the scope of this disclosure, and certain methods or processing blocks may be omitted in some implementations. The methods and processes described herein are also not limited to any particular sequence, and the blocks or states associated therewith may be executed in other appropriate sequences, or may be executed in parallel, or in some other manner. Blocks or states may be added to or removed from the disclosed exemplary embodiments. The performance of certain operations or processes may be distributed among computer systems or computer processors, not only residing within a single machine, but also deployed on multiple machines.

[0097] As used herein, any form of hardware, software or its combination can be utilized to realize circuit.For example, one or more processors, controllers, ASIC, PLA, PAL, CPLD, FPGA, logic components, software routines or other mechanisms can be realized to form circuit.In implementation, various circuits described herein can be implemented as discrete circuits, or described functions and features can be shared in part or in whole between one or more circuits.Even if various features or elements of function can be described or advocated as separate circuits separately, these features and functions can be shared between one or more common circuits, and this description does not require or imply the need for separate circuits to realize this feature or function.When using software to realize circuit in whole or in part, this software can be implemented as and can operate together with a computing or processing system (such as computer system 900) that performs the described function relative to it.

[0098] As used herein, the term "or" may be interpreted in an inclusive or exclusive sense. Furthermore, singular descriptions of resources, operations, or structures should not be construed to exclude the plural. Unless otherwise specifically stated or understood otherwise in the context of use, conditional language such as "can," "would," "might," or "may" is generally intended to convey that certain embodiments include certain features, elements, and / or steps while other embodiments do not.

[0099] The terms and phrases used in this document, and variations thereof, unless expressly stated otherwise, should be construed as open ended and not restrictive. Adjectives such as "traditional," "conventional," "normal," "standard," "known," and terms of similar meaning should not be construed to limit the items described to those available at a given time period or at a given time, but rather should be understood to include conventional, conventional, normal, or standard technology available or known at any time now or in the future. In certain cases, the presence of broadening words and phrases (such as "one or more," "at least," "but not limited to," or other similar phrases) should not be construed to mean that a narrower context is intended or required where such broadening words may not be present.

Claims

1. A computer system for performing a set of hardware-specific tuning tasks related to a machine learning (ML) model, the computer system comprising: a memory storing a scheduler circuit, wherein the scheduler circuit operates according to machine-readable instructions; as well as the scheduler circuitry communicating with a job manager at the agent circuitry, wherein the job manager receives a schedule request message from the scheduler circuitry and performs an auto-tuning phase of scheduling; as well as one or more processors configured to access the memory and execute the stored machine-readable instructions to: The scheduler circuit initiates the first part of the task using a graphics processing unit (GPU) or a central processing unit (CPU) based on a scheduling policy; scheduling, by the scheduler circuit, a second portion of the task with the GPU or the CPU based on the scheduling policy, wherein starting from the first portion of the task, the scheduling utilizes the GPU or the CPU when the GPU or the CPU is not being used; receiving, by the scheduler circuitry, updates to auto-tuning task metrics from the execution of the scheduled auto-tuning phase as the job manager executes the scheduled auto-tuning phase and measures execution time associated with the execution of the scheduled auto-tuning phase, wherein the scheduled auto-tuning phase comprises a selection phase, a build phase, a reduction phase, or an update phase; as well as updating, by the scheduler circuitry, the auto-tuning task metric based at least in part on the received update; The proxy circuit is configured to optimize the inference performance of the ML model for a specific hardware configuration.

2. The computer system of claim 1 , wherein the machine-readable instructions are further configured to: A registration request with an auto-tuning option is received by the scheduler circuit from the job manager.

3. The computer system of claim 1, wherein the scheduler circuit utilizes exclusive resource access to a particular resource for only one stage at a time.

4. The computer system of claim 1, wherein the scheduler circuit utilizes Shortest Job First (SJF) to prioritize a first phase requiring a short completion time over a second phase having a longer completion time.

5. The computer system of claim 1, wherein a multi-process service (MPS) is implemented to isolate computing resources.

6. The computer system according to claim 1, further comprising a task launcher circuit residing together with the scheduler circuit, wherein the task launcher circuit launches the task residing on the agent circuit on the GPU or CPU based on computing resource availability information of the GPU or CPU.

7. A computer-implemented method for performing a set of hardware-specific tuning tasks related to a machine learning (ML) model, the computer-implemented method comprising: Initiating a first portion of the task using a graphics processing unit (GPU) or a central processing unit (CPU) based on a scheduling policy by a scheduler circuit, the scheduler circuit communicating with a job manager at an agent circuit, wherein the job manager receives a scheduling request message from the scheduler circuit and performs an automatic tuning phase of scheduling; scheduling a second portion of the task with the GPU or the CPU based on the scheduling policy, wherein starting from the first portion of the task, the scheduling utilizes the GPU or the CPU when the GPU or the CPU is not being used; receiving, by the scheduler circuitry, updates to auto-tuning task metrics from the execution of the scheduled auto-tuning phase as the job manager executes the scheduled auto-tuning phase and measures execution time associated with the execution of the scheduled auto-tuning phase; as well as updating, by the scheduler circuitry, the auto-tuning task metric based at least in part on the received update; as well as Optimize the inference performance of the ML model for a specific hardware configuration.

8. The computer-implemented method of claim 7, further comprising: A registration request with an auto-tuning option is received by the scheduler circuit from the job manager.

9. The computer-implemented method of claim 7, wherein the scheduler circuit utilizes exclusive resource access to a particular resource for only one stage at a time.

10. The computer-implemented method of claim 7, wherein the scheduler circuit utilizes Shortest Job First (SJF) to prioritize a first phase requiring a short completion time over a second phase having a longer completion time.

11. The computer-implemented method of claim 7, wherein a multi-process service (MPS) is implemented to isolate computing resources. 12 . The computer-implemented method of claim 7 , wherein the task residing on the agent circuit is launched on the GPU or CPU based on computing resource availability information of the GPU or CPU.

13. A non-transitory computer-readable storage medium storing a plurality of instructions executable by one or more processors, the plurality of instructions, when executed by the one or more processors, causing the one or more processors to: Based on the scheduling policy, the scheduler circuit initiates a first part of the task using a graphics processing unit (GPU) or a central processing unit (CPU), the scheduler circuit communicates with a job manager at the agent circuit, wherein the job manager receives a scheduling request message from the scheduler circuit and performs an automatic tuning phase of the scheduling; scheduling a second portion of the task using the GPU or the CPU based on the scheduling policy, wherein starting from the first portion of the task, the scheduling utilizes the GPU or the CPU when the GPU or the CPU is not being used; receiving updates to auto-tuning task metrics from execution of the scheduled auto-tuning phase as the job manager executes the scheduled auto-tuning phase and measures execution time associated with execution of the scheduled auto-tuning phase; and updating the auto-tuning task metric based at least in part on the received update; The agent circuit is configured to optimize the inference performance of a machine learning (ML) model for a specific hardware configuration.

14. The non-transitory computer-readable storage medium of claim 13, wherein the one or more processors are further configured to receive a registration request with an auto-tuning option. 15 . The non-transitory computer-readable storage medium of claim 13 , wherein the scheduler circuit utilizes exclusive resource access to a particular resource for only one stage at a time.

16. The non-transitory computer-readable storage medium of claim 13, wherein the scheduler circuit utilizes Shortest Job First (SJF) to prioritize a first phase requiring a short completion time over a second phase having a longer completion time.

17. The non-transitory computer-readable storage medium of claim 13, wherein a multi-process service (MPS) is implemented to isolate computing resources.

Images