Automatic hardware resource optimization

Abstract

The automated hardware resource optimization system includes a computing platform having a hardware processor and having a system memory storing software code. The hardware processor is configured to execute the software code to perform the following: identifying computing hardware for hosting a neural network-based application; determining a batch size for performing data processing using the neural network-based application based on a first performance parameter of the computing hardware; and tuning a second performance parameter of the computing hardware using data batches having the determined batch size to enable substantially continuous loading of its hardware processor memory. The software code also optimizes a processing flow for performing data processing based on the determined batch size and the tuned second performance parameter, and generates a configuration file identifying the computing hardware, the neural network-based application, the determined batch size, the tuned second performance parameter, and the optimized processing flow.

Description

Background Art

[0001] An artificial neural network - also referred to simply as a neural network - is a machine learning framework in which patterns or learned representations of observed data are processed using highly connected computational layers that map relationships between inputs and outputs. In the context of deep learning, a "deep neural network" may refer to a neural network that utilizes multiple hidden layers between the input and output layers, which may allow learning based on features that are not explicitly defined in the raw data. Various forms of neural networks may be used to make predictions about new data based on past examples or "training data sets." In various embodiments, deep neural networks may be trained as classifiers and may be used to perform, for example, image processing or natural language processing.

[0002] Applications based on deep neural networks typically require powerful computing resources for training and runtime. In addition, such applications typically have pre-processing steps that consume resources. Therefore, it is important to properly determine and configure the computing hardware on which those neural network-based applications are to be trained or run. However, the hardware on which the neural network-based applications are hosted can be heterogeneous hardware resources with a variety of different hardware configurations. In addition, the hardware computing environment in which the neural network-based applications are executed can be shared with other applications, resulting in some variability in the hardware available for hosting the neural network-based applications at any particular time. Therefore, there is a need in the art for a solution for dynamically configuring available computing hardware to optimize the performance of the neural network-based applications hosted by the hardware. Summary of the invention

[0003] An automated hardware resource optimization system and method used by such a system is provided, substantially as shown and / or described in connection with at least one of the accompanying drawings, and as more fully set forth in the claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0004] Figure 1 A diagram illustrating an exemplary hardware resource optimization system according to one embodiment;

[0005] Figure 2 According to one embodiment, Figure 1 A more detailed view of hardware resource optimization system and hardware machine combination;

[0006] Figure 3A is a flowchart presenting an exemplary hardware resource optimization method according to one embodiment; and

[0007] Figure 3B It presents a Figure 3AAdditional optional actions performed as part of the exemplary method outlined in . DETAILED DESCRIPTION

[0008] The following description contains specific information related to the embodiments in the present disclosure. Those skilled in the art will recognize that the present disclosure can be implemented in a manner different from that specifically discussed herein. The drawings in this application and the detailed description attached thereto only relate to exemplary embodiments. Unless otherwise stated, the same or corresponding elements in the drawings may be indicated by the same or corresponding reference numerals. In addition, the drawings and illustrations in this application are generally not drawn to scale and are not intended to correspond to actual relative sizes.

[0009] As mentioned above, deep neural network based applications typically require powerful computing resources for training and runtime. In addition, such applications typically have resource consuming pre-processing steps. Therefore, it is important to properly determine and configure the computing hardware on which those neural network based applications are to be trained or run. However, the hardware on which the neural network based applications are hosted can be heterogeneous, and the hardware computing environment can be shared with other applications, resulting in some variability in the hardware available for hosting the neural network based applications at any particular time.

[0010] With respect to the term "heterogeneous" as applied to computing hardware, it should be noted that for the purposes of this application, heterogeneous hardware is defined as two or more hardware elements having different configurations. Examples of different hardware configurations include different types or numbers of central processing units (CPUs), different sizes or speeds of memory or disks, the presence or absence of a graphics processing unit (GPU), and differences in the type or number of graphics processing units when present, network connection speeds, and / or the presence or absence of other attached devices such as a tensor processing unit (TPU), to name a few.

[0011] The present application discloses an automated hardware resource optimization system and a method used by such a system, which overcomes the shortcomings and deficiencies in conventional techniques by dynamically configuring available computing hardware to be suitable for neural network-based applications that will be hosted by the hardware. It should be noted that, as used in the present application, the terms "automation", "automated" and "automating" refer to systems and methods that do not require the participation of human users such as network administrators. Although in some embodiments, a human operator or administrator can check the performance of an automated system that performs the automated method described herein, human participation is optional. Therefore, the method described in the present application can be performed under the control of the hardware processing components described herein.

[0012] In addition, as described above, a neural network is a machine learning framework in which patterns or learned representations of observed data are processed using highly connected computational layers that map the relationship between inputs and outputs. In the context of deep learning, a "deep neural network" may refer to a neural network that utilizes multiple hidden layers between the input layer and the output layer, which may allow learning based on features that are not clearly defined in the original data. As used in this application, what is described as a "neural network-based application" is characterized as an integrated software application that includes one or more embedded deep neural networks. Therefore, in some embodiments, a neural network-based application may include a single neural network, while in other embodiments, a neural network-based application may include multiple neural networks that may be configured to perform data processing tasks sequentially, in parallel, or in a combination of sequential and parallel operations.

[0013] Although the disclosure in this application focuses on hardware resource optimization for neural network-based applications, the emphasis is on the fact that these applications typically require powerful computing resources and resource-intensive preprocessing steps. However, the hardware resource optimization solution can be advantageously used for other applications that do not include embedded neural networks. Other types of applications that can benefit from the optimization solution include, for example, video shot segmentation applications and various applications for transforming video.

[0014] Figure 1 1 shows a diagram of an exemplary automated hardware resource optimization system 120 according to one embodiment. Figure 1 As shown, the automated hardware resource optimization system 120 may be used in a use environment that includes a local network 100 that includes local computing hardware 106, such as one or more of locally based machines 130a, machines 130b, and / or machines 130c that are communicatively coupled to the automated hardware resource optimization system 120. Additionally or alternatively, the use environment in which the automated hardware resource optimization system 120 is utilized may include cloud-based computing hardware 116, such as one or more of remote cloud-based machines 130d and / or machines 130e that are communicatively coupled to the automated hardware resource optimization system 120.

[0015] like Figure 1 As further shown in FIG. 1 , the automated hardware resource optimization system 120 is interactively linked to a neural network-based application database 102 that stores one or more neural network-based software applications. Figure 1 Depicted in FIG. 1 are neural network based applications 104 a and 104 b. Figure 1Also shown are test data 110, tuning data 112, and performance monitoring data 114 that are used by an automated hardware resource optimization system 120 to optimize the performance of one or more of the neural network-based application 104a and the application 104b when hosted by one or both of the local computing hardware 106 and the cloud-based computing hardware 116.

[0016] It should be noted that although Figure 1 Cloud-based computing hardware 116 is shown as being provided by a single cloud entity, but this representation is merely exemplary. In other embodiments, cloud-based computing hardware 116 may be provided by a plurality of different cloud computing services. Furthermore, the specific numbers of machines 130a, 130b, 130c, 130d, and 130e (hereinafter referred to as "machines 130a to 130e") and neural network-based applications 104a and 104b are merely exemplary. In other embodiments, the neural network-based application database 102 may store one neural network-based application, two neural network-based applications, or more than two neural network-based applications. Furthermore, the local computing hardware 106 included in the local network 100 may include a single machine, two machines, Figure 1 , or more than three machines, while the cloud-based computing hardware 116 may similarly include a single machine, Figure 1 The two machines shown in , or more than two machines.

[0017] Each of machines 130a-130e may be implemented as one or more computing platforms, such as, for example, computer servers. In one embodiment, for example, machines 130d and 130e may correspond to one or more web servers that may be accessed by the automated hardware resource optimization system 120 via a packet-switched network such as the Internet. In contrast, machines 130a, 130b, and 130c (hereinafter referred to as "machines 130a-130c") may correspond to one or more computer servers that may be accessed by the automated hardware resource optimization system 120 via a local area network (LAN) or another type of limited distributed or dedicated network.

[0018] It should also be noted that although machines 130a through 130e are depicted using the same image, this representation is merely exemplary. In other embodiments, two or more of machines 130a through 130e may be implemented as a heterogeneous combination of computing hardware, each including its own processor resources and memory resources. Figure 1The neural network-based application database 102 is depicted as being included as part of the local network 100, but this representation is also merely exemplary. In other embodiments, the neural network-based application database 102 can be remote from the automated hardware resource optimization system 120 and can be accessed by the automated hardware resource optimization system 120 via a public communication network such as the Internet.

[0019] Figure 2 A more detailed view of an automated hardware resource optimization system 220 in combination with a machine 230 is shown, according to one embodiment. Figure 2 Also shown in FIG. 2 are test data 210, tuning data 212, and performance monitoring data 214, which are used by an automated hardware resource optimization system 220 to optimize the performance of the hardware resource when hosted by a machine 230. Figure 1 The performance of one or more of the neural network-based applications 104a and 104b in the embodiment of the present invention is optimized. The test data 210, the tuning data 212, and the performance monitoring data 214 generally correspond to Figure 1 Thus, the test data 210, tuning data 212, and performance monitoring data 214 may share any characteristics attributed to the respective test data 110, tuning data 112, and performance monitoring data 114 by the present disclosure, and vice versa.

[0020] like Figure 2 As shown, the automated hardware resource optimization system 220 includes a computing platform 222 and a system memory 226, wherein the computing platform 222 has a hardware processor 224, and the system memory 226 is implemented as a non-transitory storage device storing software code 228 and a configuration file 240. The hardware processor 224 can be a CPU of the computing platform 222, for example, in the computing platform 222, the hardware processor 224 executes the software code 228 to generate the configuration file 240.

[0021] It should be noted that, although for the sake of conceptual clarity, the present application refers to the software code 228 and the configuration file 240 as being stored in the system memory 226, more generally, the system memory 226 can take the form of any computer-readable non-transitory storage medium. The expression "computer-readable non-transitory storage medium" as used in the present application refers to any medium that does not include a carrier or other temporary signal that provides instructions to the hardware processor 224 of the automated hardware resource optimization system 220. Therefore, the computer-readable non-transitory medium can correspond to various types of media, such as, for example, volatile media and non-volatile media. Volatile media can include dynamic memory such as dynamic random access memory (dynamic RAM), while non-volatile memory can include optical storage devices, magnetic storage devices, or electrostatic storage devices. Common forms of computer-readable non-transitory media include, for example, optical disks, RAM, programmable read-only memory (PROM), erasable PROM (EPROM), and flash memory.

[0022] Figure 2 The automated hardware resource optimization system 220 in the embodiment generally corresponds to Figure 1 , and those corresponding features may share any characteristics attributed to any corresponding feature by the present disclosure. That is, although Figure 1 Not shown, but the automated hardware resource optimization system 120 may include features corresponding to a computing platform 222 , a hardware processor 224 , and a system memory 226 , respectively, which stores software code 228 for generating a configuration file 240 .

[0023] like Figure 2 As further shown in , the machine 230 may include: one or more CPUs 232 each having a CPU memory 233, one or more GPUs 234 each having a GPU memory 235, one or more TPUs 236 each having a TPU memory 237, a main memory 238 implemented as a non-temporary storage device such as a hard drive or solid-state memory, and a memory cache 239. Figure 2 The machine 230 in may generally correspond to Figure 1 Thus, machines 130a to 130e may share any of the features attributed to machine 230 in the present disclosure, and vice versa. That is, although Figure 12, but machines 130a through 130e may include features corresponding to CPU 232, GPU 234, TPU 236, main memory 238, and cache 239, respectively. In addition, like one or more of machines 130a through 130e, machine 230 may be part of local computing hardware 106 included in local network 100, or may be included in cloud-based computing hardware 116.

[0024] The communication between the automated hardware resource optimization system 120 / 220 and the machines 130a to 130c / 230 may occur as wired communication or wireless communication. When the communication is performed wirelessly, the communication between the automated hardware resource optimization system 120 / 220 and the machines 130a to 130e / 230 may be performed using a fourth generation (4G) wireless communication protocol or using a 5G wireless communication protocol that meets the IMT-2020 requirements established by the International Telecommunication Union (ITU). Alternatively or in addition, the wireless communication between the automated hardware resource optimization system 120 / 220 and the machines 130a to 130e / 230 may occur via Wireless Fidelity (WiFi), Worldwide Interoperability for Microwave Access (WiMAX) communication, or any other suitable wireless communication protocol.

[0025] By reference Figure 3A The functionality of software code 228 is further described. Figure 3A A flowchart 350 presenting an exemplary hardware resource optimization method is shown according to one embodiment. Figure 3A It should be noted that certain details and features are omitted from flowchart 350 so as not to obscure the discussion of the inventive features of the present application.

[0026] Reference combination Figure 1 and Figure 2 of Figure 3A , the flowchart 350 begins by identifying computing hardware 106 / 116 for hosting a neural network-based application 104a or an application 104b (act 351). Figure 1 As described, the expression "neural network-based application" refers to an integrated software application that includes one or more embedded deep neural networks. Neural network-based applications 104a and 104b may include a single neural network or multiple neural networks that may be configured to perform data processing tasks sequentially, in parallel, or in a combination of sequential and parallel operations.

[0027] The neural network or neural networks included in the neural network-based applications 104a and 104b may be configured to, for example, inferentially perform facial recognition, object recognition, location recognition, and / or speech recognition. In some embodiments, the neural network-based applications 104a and 104b may be configured to perform data processing by analyzing images or media content. When used to analyze media content, such content may, for example, include one or more audio-video files in the form of movies, episode content including television programs or web-based programs, and video games, as well as one or more video files with audio omitted, or one or more audio files such as music, or electronic text such as digital books or other publications.

[0028] The computing hardware 106 / 116 identified in act 351 may include one or more of machines 130a to 130e / 230. That is, the computing hardware 106 / 116 identified in act 351 may include one or more of machines 130a to 130c / 230 included in the local computing hardware 106 and / or one or more of machines 130d / 230 and machines 130e / 230 included in the cloud-based computing hardware 116. The identification of computing hardware 106 / 116 for hosting one of the neural network-based applications 104a or 104b may be performed by software code 228, which is executed by the hardware processor 224 of the automated hardware resource optimization system 120 / 220. As described above, machine 230 corresponds to any one of machines 130a to 130e included in the computing hardware 106 / 116. Therefore, and for the sake of conceptual clarity, the features “computing hardware 106 / 116 ” and “machine 230 ” will be used synonymously in the following description of flowchart 350 .

[0029] The flowchart 350 continues by determining a batch size for performing data processing using the neural network-based application 104a or the application 104b based on the first performance parameter of the machine 230 (act 352). Act 352 may be performed by software code 228 executed by a hardware processor 224 of the automated hardware resource optimization system 120 / 220. It should be noted that the present hardware resource optimization solution may perform an initial performance assessment to benchmark each of the machines 230 identified in act 351. This benchmark will enable the software code 228 to optimize the process flow for performing data processing using the neural network-based application 104a or the application 104b hosted by one or more machines 230.

[0030] In some embodiments, the software code 228 may determine the maximum number of samples (i.e., batch size) that will be propagated simultaneously through the neural network-based application 104a or application 104b. To determine the batch size, the software code 228 may send the test data 110 / 210 in the form of synthetic data in groups of different sizes to model and monitor the performance of the machine 230. In some embodiments, it may be advantageous or desirable to use a GPU 234 or TPU 236 rather than a CPU 232 to perform data processing using the neural network-based application 104a or application 104b because faster neural network inference is typically performed using a GPU or TPU. In some of those embodiments, the first performance parameter for determining the batch size may be GPU or TPU usage, as measured by GPU or TPU memory load during processing of the test data 110 / 210 by the machine 230. For example, software code 228 may monitor the inference time for each batch included in test data 110 / 210, as well as the memory load of GPU 234 or TPU 236 during inference, and software code 228 may determine the batch size based on one or both of those performance GPU or TPU parameters. It should be noted that GPU or TPU memory load may be used as a particularly good proxy for determining batch size.

[0031] It should also be noted that while initial estimation of performance parameters of machines 230 associated with neural network-based applications 104a or deep learning architectures of one or more neural networks included in applications 104b may be done from scratch for each machine 230, as described above, in other embodiments, benchmarking may be performed based on pre-existing data. For example, performance parameters of machines 230 may be obtained from public benchmarks or as a result of previous benchmarking of similar neural network-based applications on similar computing hardware.

[0032] Flowchart 350 continues by tuning a second performance parameter of machine 230 using the batches of data having the batch size determined in act 352 to enable substantially continuous loading of the hardware processor memory of machine 230 (act 353). Once the appropriate batch size is determined in act 352, the hardware processor 224 of the automated hardware resource optimization system 120 / 220 may execute software code 228 to send the tuning data 112 / 212 having the now determined batch size to machine 230 and monitor GPU memory 235 or TPU memory 237 to substantially minimize the amount of time that GPU memory 235 or TPU memory 237 is empty (i.e., GPU or TPU memory load is intermittent). Thus, in some embodiments, the second performance parameter tuned in act 353 may be the memory load rate of GPU 234 or the memory load rate of TPU 236. Alternatively, in other embodiments, it may be advantageous or desirable to use the utilization rate of GPU 234 or the utilization rate of TPU 236.

[0033] GPU or TPU memory load intermittency is often the result of a slow data loading process. In response, software code 228 can increase the number of threads dedicated to loading data into GPU memory 235 or TPU memory 237 and the shared memory between those threads until the memory load is continuously in a state of substantially constant data flow. If GPU or TPU memory load continues to be intermittent, main memory 238 input / output (I / O) speed can be evaluated, and in the event of insufficient performance, software code 228 can recommend hard disk updates or adding some CPU threads to fill memory cache 239 faster.

[0034] In cases where the machine 230 does not include a TPU 236, and the performance of the GPU 234 is insufficient to host the neural network-based application 104a or application 104b, or the batch size determined based on the performance of the GPU 234 in act 352 is very small, the performance of the CPU 232 may also be evaluated. While a CPU is typically slower than a GPU or TPU for performing data processing using a neural network-based application, it may be an acceptable alternative when the neural network model used is not too demanding, or when processing cost is a more important criterion than processing time.

[0035] In the case where the CPU 232 is used to perform data processing using the neural network-based application 104a or the application 104b, in conjunction with the GPU 234 or as an alternative to the GPU 234, the actions 352 and 353 may be performed for the CPU 232. That is, in some embodiments, the first performance parameter used to determine the batch size in the action 352 may include the memory load of the CPU 232 during the processing of the test data 110 / 210. In addition, in those embodiments, the second performance parameter tuned in the action 353 may include the utilization rate of the CPU 232 or the memory load rate of the CPU 232. Therefore, the first performance parameter used in the action 352 may be the memory load of the CPU 232 and / or the memory load of the GPU 234 and / or the memory load rate of the TPU 236 during the processing of the test data 110 / 210 by the machine 230. In addition, the second performance parameter tuned in action 353 may be the usage rate or memory load rate of the CPU 232 and / or the usage rate or memory load rate of the GPU 234 and / or the usage rate or memory load rate of the TPU 236 .

[0036] Flowchart 350 continues by optimizing a process flow for performing data processing using the neural network-based application 104a or the application 104b (act 354) based on the batch size determined in act 352 and the second performance parameter tuned in act 353. The optimization of the process flow for performing data processing using the neural network-based application 104a or the application 104b may be performed by software code 228 executed by a hardware processor 224 of the automated hardware resource optimization system 120 / 220.

[0037] For example, the software code 228 can group together data processing tasks sent to the same neural network included in the neural network-based application 104a or application 104b to utilize the optimal batch size and maximize TPU / GPU / CPU usage, thereby greatly reducing the overhead of launching each task independently. This can be particularly advantageous when applied to short tasks such as image labeling. When the computing hardware 106 / 116 identified in action 351 includes more than one machine 230, the software code 228 can use the information obtained during the initial performance benchmark to prioritize certain tasks on certain machines. Since not all neural network models have the same hardware requirements, one neural network model may perform better on a machine with a CPU, another neural network model may perform well with a small GPU, i.e., a relatively slow GPU and / or a GPU with relatively small GPU memory, while another neural network model may, for example, require a larger GPU with larger GPU memory and / or capable of higher processing speeds in order to work quickly.

[0038] Tasks involving the same input but different neural network models, such as face recognition and location recognition, may benefit from sharing common preprocessing techniques, such as, for example, lens detection in a video. In addition, in some embodiments, more than one neural network may share some elements of a common architecture, such as, for example, one or more initial neural network layers and weights. In any of these cases, the software code 228 may be configured to start tasks associated with the same input on the same machine so as to reuse the time preprocessing material calculated by the first task in subsequent tasks. In some embodiments, optimizing the processing flow in action 354 may also include determining when two or more data processing tasks can be executed simultaneously, i.e., in parallel, on the same machine. For example, if two or more tasks do not use all hardware resources according to the benchmark study, this may be possible, thereby advantageously reducing the total time required to execute two or more tasks compared to executing two or more tasks sequentially. As a specific example, intensive GPU or TPU-specific tasks may be processed in parallel with intensive CPU-specific tasks. Therefore, data processing performed using a neural network-based application 104a or application 104b may include multiple tasks, and the processing flow optimized in action 354 may facilitate the simultaneous execution of at least some of these tasks.

[0039] In some embodiments, the exemplary flowchart 350 may end with generating a configuration file 240 that identifies the computing hardware 116 / 216, the neural network-based application 104a or application 104b, the batch size determined in act 352, the second performance parameter tuned in act 353, and the process flow optimized in act 354 (act 355). The configuration file 240 may be used to optimize hardware resource usage when hosting the neural network-based application 104a or application 104b using one or more machines 230. Act 355 may be performed by software code 228 that is executed by a hardware processor 224 of the automated hardware resource optimization system 120 / 220.

[0040] Configuration file 240 may be stored in system memory 238 so it can be reused in future use cases where the same neural network-based application is to be hosted by the same computing hardware. Additionally, configuration file 240 may be used to facilitate future benchmarking of similar computing hardware when used to host the same or similar neural network-based applications, and / or configuration file 240 may be used to predict optimal configurations for similar computing hardware when used to host the same or similar neural network-based applications.

[0041] Figure 3B It presents a Figure 3A 355. Thus, while in some embodiments the method outlined by flowchart 350 may end with action 355, in other embodiments flowchart 350 may continue, i.e., optionally during data processing using a neural network-based application 104a or application 104b hosted by the computing hardware 106 / 116 identified in action 351 to monitor one or both of the first performance parameter and the second performance parameter (action 356). The performance monitoring performed in action 356 may include monitoring of some performance parameters and / or substantially continuous monitoring of those same or other performance parameters. For example, action 356 may include: continuously monitoring the time taken to complete the data processing task, while monitoring TPU / GPU / CPU usage or memory load if and when those times begin to degrade or when a hardware change is detected. In some embodiments, the data collected during action 356 may be persistently stored and may be used to detect performance degradation over time.

[0042] Monitoring the performance of computing hardware during data processing provides the advantage of identifying processing bottlenecks in real time as they develop. These processing bottlenecks may develop due to poor initial benchmarks, resulting in inaccurate batch determinations in action 352, detuning in action 353, and / or suboptimal processing flows in action 354. Alternatively, bottlenecks may arise due to degradation in the performance of one or more machines 230 included in the computing hardware 106 / 116. Action 356 may be performed by software code 228 based on performance monitoring data 114 / 214 received from one or more machines 230 by the automated hardware resource automation system 120 / 220, the software code 228 being executed by the hardware processor 224 of the automated hardware resource optimization system 120 / 220.

[0043] In some embodiments, flowchart 350 may continue and end by modifying at least one of the batch size determined in act 352, the tuning of the second performance parameter tuned in act 353, or the processing flow optimized in act 354, optionally based on changes in one or both of the first performance parameter or the second performance parameter during data processing (act 357). Act 357 facilitates modification of configuration file 240 to account for the actual performance of machine 230 included in computing hardware 106 / 116 identified in act 351 during data processing using neural network-based application 104a or application 104b. It should be noted that although flowchart 350 shows act 356 as preceding act 357, in some embodiments, act 356 and act 357 may be performed substantially simultaneously.

[0044] In some use cases, changes in one or both of the first performance parameter and the second performance parameter may lead to the development of a processing bottleneck, as described above. In those embodiments, action 357 may be performed to resolve the bottleneck. Action 357 may be performed by software code 228, which is executed by the hardware processor 224 of the automated hardware resource optimization system 120 / 220.

[0045] Alternatively or in addition, when one or more machines 230 receive software or hardware updates, performance monitoring data 114 / 214 may also be used to determine whether configuration files 240 need to be updated. For example, a new software version may change the processing efficiency of a machine 230. By monitoring queued tasks, for example, a reduction in GPU memory 235 may be detected, and benchmarking of the machine 230 may be repeated in anticipation of an increase in batch size.

[0046] Therefore, the present application discloses an automated hardware resource optimization system and a method used by such a system that overcomes the shortcomings and deficiencies in conventional techniques. Compared to conventional methods of optimizing the performance of neural network-based applications by optimizing the neural network architecture for the computing hardware hosted thereon, the present method advantageously makes the neural network-based application effectively agnostic to the hardware hosting it by optimizing the configuration of the available hardware and the process flow utilizing the hardware. That is, compared to conventional methods, the present solution configures the available computing hardware to optimize multiple different neural network operations, which are described in the present invention. Therefore, the present hardware resource optimization solution enables the use of heterogeneous hardware resources, including those hardware resources available in a local computing environment, those hardware resources available as cloud-based resources, and those hardware resources used in a hybrid computing environment including a mix of local and cloud-based computing resources. In addition, the present hardware optimization solution advantageously enables the identification of real-time processing bottlenecks, as well as enabling changes to configuration files to address and resolve bottlenecks, and utilizing enhanced hardware efficiency caused by software or hardware updates.

[0047] A significant advantage of conventional methods for optimizing the performance of neural network-based applications is that the present hardware resource optimization solution enables exclusive reliance on local computing hardware to process critical or sensitive data, thereby avoiding sending such data to the cloud. That is, the present hardware resource solution achieves the advantages of scalability provided by using a hybrid local / cloud computing environment while selectively limiting some data processing tasks to local local computing hardware. In addition, the present solution identifies the best local hardware resource elements for performing each data processing task, thereby advantageously taking full advantage of the potential performance of the local hardware environment.

[0048] As can be seen from the above description, it is apparent that various techniques can be used to implement these concepts without departing from the scope of the concepts described in this application. In addition, although concepts have been described with specific reference to certain embodiments, those of ordinary skill in the art will recognize that changes can be made in form and detail without departing from the scope of these concepts. Therefore, the described embodiments are considered to be illustrative and non-restrictive in all respects. It should also be understood that the application is not limited to the specific embodiments described herein, but many rearrangements, modifications and replacements are possible without departing from the scope of the present disclosure.

Claims

1. An automated hardware resource optimization system for dynamically configuring available computing hardware to optimize the performance of neural network-based applications, comprising: A computing platform comprising a hardware processor and a system memory storing software code: The hardware processor is configured to execute the software code to: identifying computing hardware for hosting the neural network-based application, wherein the identifying determines that the computing hardware is usable for hosting the neural network-based application; determining a batch size for performing a plurality of data processing tasks using the neural network-based application based on a first performance parameter of the computing hardware, the first performance parameter comprising at least one of a tensor processing unit (TPU) memory load, a graphics processing unit (GPU) memory load, or a central processing unit (CPU) memory load during processing of test data; tuning a second performance parameter of the computing hardware using the data batches having the determined batch size, wherein the tuned second performance parameter results in continuously loading a hardware processor memory of the computing hardware, wherein the second performance parameter of the computing hardware comprises at least one of a usage rate of the TPU, the GPU, or the CPU, or a memory loading rate of the TPU, the GPU, or the CPU during processing of the data batches having the determined batch size; optimizing a process for performing the plurality of data processing tasks using the neural network-based application based on the determined batch size and the tuned second performance parameter, wherein optimizing the process reduces an amount of time a memory of the TPU, the GPU, or the CPU is empty; and A configuration file is generated that identifies the computing hardware, the neural network-based application, the determined batch size, the tuned second performance parameter, and the optimized process.

2. The automated hardware resource optimization system according to claim 1, wherein: The optimized processing causes at least some of the plurality of tasks to be performed simultaneously.

3. The automated hardware resource optimization system according to claim 1, wherein: The hardware processor is further configured to execute the software code to: monitoring at least one of the first performance parameter and the second performance parameter during execution of the data processing task; as well as At least one of the determined batch size, the tuned second performance parameter, and the optimized process is modified based on a change in at least one of the first performance parameter or the second performance parameter during execution of the data processing task.

4. The automated hardware resource optimization system according to claim 1, wherein: The neural network-based application includes a plurality of neural networks.

5. The automated hardware resource optimization system according to claim 1, wherein: The computing hardware includes a plurality of machines.

6. The automated hardware resource optimization system according to claim 5, wherein: At least one of the plurality of machines is a cloud-based machine, and at least another of the plurality of machines is part of a same local network as the automated hardware resource optimization system.

7. The automated hardware resource optimization system according to claim 1, wherein: The computing hardware includes at least one machine, and wherein the automated hardware resource optimization system and the at least one machine are part of a same local network.

8. The automated hardware resource optimization system according to claim 1, wherein: The computing hardware includes at least one cloud-based machine.

9. A method for use by an automated hardware resource optimization system for dynamically configuring available computing hardware to optimize the performance of a neural network-based application, the automated hardware resource optimization system comprising a computing platform having a hardware processor and having a system memory storing software code, the method comprising: identifying, by the software code executed by the hardware processor, computing hardware for hosting the neural network-based application, wherein the identifying determines that the computing hardware is usable for hosting the neural network-based application; determining, by the software code executed by the hardware processor and based on a first performance parameter of the computing hardware, a batch size for performing a plurality of data processing tasks using the neural network-based application, the first performance parameter comprising at least one of a tensor processing unit (TPU) memory load, a graphics processing unit (GPU) memory load, or a central processing unit (CPU) memory load during processing of test data; tuning, by the software code executed by the hardware processor and using the data batches having the determined batch size, a second performance parameter of the computing hardware, wherein the tuned second performance parameter results in continuously loading a hardware processor memory of the computing hardware, wherein the second performance parameter of the computing hardware comprises at least one of a usage rate of the TPU, the GPU, or the CPU, or a memory loading rate of the TPU, the GPU, or the CPU during processing of the data batches having the determined batch size; optimizing, by the software code executed by the hardware processor and based on the determined batch size and the tuned second performance parameter, a process of performing the plurality of data processing tasks using the neural network-based application, wherein optimizing the process reduces an amount of time a memory of the TPU, the GPU, or the CPU is empty; and A profile identifying the computing hardware, the neural network-based application, the determined batch size, the tuned second performance parameter, and the optimized process is generated by the software code executed by the hardware processor.

10. The method according to claim 9, wherein: The optimized processing causes at least some of the plurality of tasks to be performed simultaneously.

11. The method according to claim 9, further comprising: During execution of the data processing task, monitoring at least one of the first performance parameter or the second performance parameter by the software code executed by the hardware processor; as well as The software code executed by the hardware processor modifies at least one of the determined batch size, the tuned second performance parameter, or the optimized processing based on a change in at least one of the first performance parameter or the second performance parameter during execution of the data processing task.

12. The method according to claim 9, wherein: The neural network-based application includes a plurality of neural networks.

13. The method according to claim 9, wherein: The computing hardware includes a plurality of machines.

14. The method according to claim 13, wherein: At least one of the plurality of machines is a cloud-based machine, and at least another of the plurality of machines is part of a same local network as the automated hardware resource optimization system.

15. The method according to claim 9, wherein: The computing hardware includes at least one machine, and wherein the automated hardware resource optimization system and the at least one machine are part of a same local network.

16. The method according to claim 9, wherein: The computing hardware includes at least one cloud-based machine.

Images