Managing machine learning environments

Abstract

The present invention relates to managing a machine learning environment. A method and a computing device for managing a machine learning environment (ML) are disclosed herein, the method comprising the following steps: receiving, by an ML controller, ML model information from an ML application, the ML model information comprising an ML model definition and ML model metadata, the ML model metadata comprising information specifying an ML runtime for executing the ML model; and generating, by the ML controller, a model runner instance in an abstraction layer at the ML controller using the received ML model information, the model runner instance being configured to interact with the specified ML runtime hosted by a target ML platform so that the ML runtime executes the ML model.

Description

Technical Field

[0001] This description relates to managing a machine learning environment. Background Art

[0002] In recent years, machine learning (ML) algorithms have been accepted as a solution to a wide range of problems in multiple application domains. As a result, electronic devices are now often required to support numerous applications that use multiple different ML models that need to be executed simultaneously on a single system.

[0003] In a first existing approach, the Android Neural Network API (NNAPI) has been developed. This is a low-level backend for running machine learning based algorithms and applications on Android devices. NNAPI includes a low-level declarative API that defines the machine learning algorithm structure and describes the associated meta information (i.e., information related to memory management, execution priority, and mapping to hardware units) and a runtime system that executes the machine learning algorithm by utilizing available hardware units via a hardware abstraction layer (HAL).

[0004] NNAPI requires users to reimplement machine learning algorithms using the provided low-level APIs or to use machine learning libraries that can leverage NNAPI. NNAPI also requires hardware vendors to design, implement, and maintain NNAPI-specific HALs.

[0005] Another existing approach involves a machine learning compiler stack. For example, the TVM compiler stack includes: an intermediate representation (IR); a compiler front end that converts a machine learning algorithm given in an external format (e.g., Tensorflow Protobuf, ONNX, PyTorch) into an internal TVM IR; a compiler back end that implements IR-level optimizations and emits executable code; a runtime system that executes the generated code and manages memory; and other resources. However, this approach requires hardware vendors to design, implement, and maintain a TVM-specific compiler back end. Summary of the invention

[0006] According to a first aspect of the present disclosure, a method for managing a machine learning (ML) environment is provided, the method comprising the following steps: receiving, by an ML controller, ML model information from an ML application, the ML model information comprising an ML model definition and ML model metadata, the ML model metadata comprising information specifying an ML runtime that executes the ML model; and generating, by the ML controller, a model runner instance in an abstraction layer at the ML controller using the received ML model information, the model runner instance being configured to interact with the specified ML runtime hosted by a target ML platform so that the ML runtime executes the ML model.

[0007] The method may further include the following steps: receiving an ML execution request from the ML application requesting execution of the ML model, the ML execution request specifying a model runner instance corresponding to the ML runtime to be used, the ML execution request also including input data to be input to the ML model; during execution of the ML model, sending the input data from the specified model runner instance to the corresponding ML runtime to be input to the ML model; receiving output data from the corresponding ML runtime by the model runner instance, wherein the output data has been output from the ML model; and providing the output data to the ML application.

[0008] The ML model metadata may also include information specifying the type of processing resources used to execute the ML model.

[0009] The method may also include the steps of executing one or more subgraphs of the ML model to determine one or more execution time measurements for the ML model; and providing the one or more execution time measurements within the ML model metadata.

[0010] The method may also include monitoring system performance parameters during execution of the ML model.

[0011] The method may also include using the monitored system performance parameters to schedule execution of the ML model.

[0012] The ML model metadata may further include scheduling information for the ML model, and the method may further include scheduling execution of the ML model using the scheduling information.

[0013] The scheduling information may include at least one of an ML model priority, a memory requirement of the ML model, a memory budget of a computing device, a target frame rate, and a power requirement of the ML model.

[0014] According to a second aspect of the present disclosure, there is provided a computer program product comprising computer-readable instructions, which, when executed by a computer, cause the computer to perform the method of the first aspect.

[0015] According to a third aspect of the present disclosure, a computing device for managing a machine learning (ML) environment is provided, the computing device comprising: a memory storing computer-readable instructions; and an ML controller, the ML controller being configured to receive ML model information from an ML application, the ML model information comprising an ML model definition and ML model metadata, the ML model metadata comprising information specifying an ML runtime that executes the ML model, wherein the ML controller comprises a model runner instance generator, the model runner instance generator being configured to generate a model runner instance in an abstraction layer at the ML controller using the received ML model information, the model runner instance being configured to interact with a specified ML runtime hosted by a target ML platform so that the ML runtime executes the ML model.

[0016] The ML controller may be further configured to: receive an ML execution request from the ML application requesting execution of the ML model, the ML execution request specifying a model runner instance corresponding to the ML runtime to be used, the ML execution request also including input data to be input to the ML model; during execution of the ML model, send the input data from the specified model runner instance to the corresponding ML runtime to be input to the ML model; receive output data from the corresponding ML runtime by the model runner instance, wherein the output data has been output from the ML model; and provide the output data to the ML application.

[0017] The ML controller may further include a ML workload monitor configured to monitor one or more system performance parameters during execution of the ML model.

[0018] The ML controller may further include an ML workload scheduler configured to schedule execution of the ML model.

[0019] The ML workload scheduler may be configured to schedule execution of the ML model using information received from the ML workload monitor.

[0020] The ML application and the target ML platform may be embedded within the computing device. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Example embodiments will now be described, by way of example only, with reference to the accompanying drawings, in which:

[0022] Figure 1 is a schematic block diagram of a computing device;

[0023] Figure 2is a schematic block diagram illustrating further details of a computing device;

[0024] Figure 3 is a flow diagram illustrating example operations performed during generation of a machine learning runner instance;

[0025] Figure 4 is a flowchart illustrating example operations performed during execution of a machine learning model; and

[0026] Figure 5 is a diagram illustrating allocation of processing resources. DETAILED DESCRIPTION

[0027] Applicants have devised a method for managing a machine learning (ML) environment in which a complex ecosystem of ML models can be housed, supporting a highly heterogeneous system in terms of different ML frameworks (i.e., ML runtimes) and in terms of the different processing resources required to execute these ML frameworks.

[0028] To exploit the full potential of these highly heterogeneous devices, a system is needed to effectively manage the execution and use of ML resources. Applicants address the previously described challenges with a runtime architecture that efficiently abstracts and coordinates the use of ML resources while ensuring that ML applications meet strict timing and resource usage requirements.

[0029] Applicants address the challenge of efficiently executing a full ecosystem of ML applications on highly heterogeneous embedded systems by designing a runtime architecture that: (1) abstracts interactions with ML resources to an ML controller; (2) monitors ML workloads being executed on the system; and 3) coordinates the execution of ML models across available resources.

[0030] In particular, the examples described herein allow for supporting different applications using different ML models. Since the functionality of the application programming interface (API) is abstracted to an intermediate controller located between the application layer and the ML runtime software, the approach described herein simplifies the interaction between the application and the ML framework (or runtime) from the application's perspective.

[0031] Figure 1 1 is a schematic block diagram illustrating components of a computing device 100 according to an example of the present disclosure. The computing device 100 may be any type of computing device, including, for example, a desktop computer, a laptop computer, a tablet computing device, or a smartphone. The computing device 100 may be part of a car control system.

[0032] Computing device 100 includes memory 110 and one or more processors 115 .

[0033] Memory 110 may include one or more non-transitory computer-readable storage media such as RAM, ROM, EEPROM, EPROM, flash memory devices, disks, etc. and combinations thereof. Memory 110 may store data and instructions executed by processing resources 115 to cause computing device 100 to perform operations.

[0034] Memory 110 also includes one or more buffers used by components of computing device 100 to temporarily store data.

[0035] Computing device 100 may also include a network interface 130 that enables communication over one or more networks (eg, the Internet).

[0036] The computing device 100 includes one or more ML runtimes 140. An ML runtime (also referred to herein as an ML framework) is software installed on the computing device 100 to implement ML functionality. One or more ML runtimes 140 act as a software layer that can interact with hardware such as processing resources to execute ML models in operation. Example ML runtimes include Tensorflow, TFLite, TVM, SNPE, and TIDL. However, other ML frameworks may be implemented in addition to or as an alternative to any of these example ML frameworks. Each ML runtime may execute one or more specific ML models in operation.

[0037] Each ML runtime or framework 140 uses one or more ML processing resources 120 when executing a machine learning model, as will be described in more detail below. The processing resources 120 may include one or more processing cores, such as a GPU, CPU, FPGA, DSP, or NPU. However, other processing resources that are apparent to those skilled in the art may be provided in addition to or as one or more alternatives to the processing resources described herein. The processor 115 and the ML processing resources 120 are shown as separate functional elements because the processor 115 is for general processing tasks, while the ML processing resources are those resources dedicated to performing the ML functions described herein. However, in some embodiments, the processor 115 and the ML processing resources may be integrated.

[0038] Processing resources 120 and ML runtime 140 together form target platform 150 .

[0039] One or more applications that utilize machine learning models may also be installed on the computing device 100. Such applications are referred to herein as ML applications 160. Different ML applications 160 may use different ML models 161, depending on the requirements of the ML application in question. Each specific ML model 161 may be associated with and executed by a corresponding ML runtime 140. The ML model may also specify a specific processing resource 120 that should be used to execute the ML model.

[0040] Where the computing device 100 is part of a car control system, then the ML application 160 running therein may relate to the functionality of the car control system, such as lane guidance for an autonomous vehicle or monitoring of interior sensors to determine if the driver is no longer able to control the vehicle.

[0041] In examples of the present disclosure, each ML application 160 interacts with the ML runtime 140 via an ML controller, referred to herein as an ML model runner controller (MRC) 170. As will be described in more detail below, the MRC acts as an intermediary between the ML runtime 140 and the ML applications 160.

[0042] The direct interaction between the ML runtime 140 and the ML application 160, which in the prior art would be provided through an application programming interface (API) of the ML application 160, is instead provided by the MRC 170, which provides an intermediate abstraction layer between the application and the ML runtime. In this way, the programming of the ML application is simplified because it is not necessary to provide an API between the ML application 160 and the ML runtime 140 to execute the ML model on behalf of the ML application 160.

[0043] Figure 2 is a schematic block diagram illustrating components of computing device 100 in greater detail.

[0044] MRC 170 may be operative to receive input from one or more ML applications 160. Each ML application may include one or more ML models 161.

[0045] ML model 161 includes model definition 162, which may be in the form of a graph definition file containing the model architecture and parameters (ie, trained weights).

[0046] Additionally, ML model 161 includes metadata 163 that provides relevant information to MRC 170 to enable the generation of an ML runner instance and subsequent execution of ML model 161. Metadata 163 includes, but is not limited to, the following details about ML model 161:

[0047] The path to the model definition file;

[0048] The name of the specific ML runtime to be used to execute the ML model;

[0049] The type of processor core on which the ML model will be executed;

[0050] Profiling information for ML models for one or more processor core types;

[0051] The memory requirements of the ML model, i.e., memory for model parameters and runtime memory;

[0052] ML model prioritization; and

[0053] The power requirements or energy requirements of the ML model.

[0054] MRC 170 includes core modules including a model runner instance generator 171 , an ML workload monitor 172 , and an ML model scheduler 173 .

[0055] The model runner instance generator 171 is configured to create a model runner instance 175 to interact with the ML runtime 140 specified by the ML application 160. The model runner instance generator 171 can access one or more centralized API libraries 180, which contain API information for the ML runtime specified by the ML application 160. The API library 180 can be stored in the memory 110 of the device 100. The API library can be provided for each of the supported ML runtimes 140. The model runner instance 175 acts as an interface between the application 160 and the ML runtime 140. The model runner instance 175 occupies the API abstraction layer 190 located at the MRC 170.

[0056] Using the API information, the ML runner instance 175 can interact with the ML runtime 140 according to the API of the ML runtime. In other words, the runner instance 175 acts as an interface between the application 160 and the ML runtime 140.

[0057] This arrangement is beneficial because the coding of ML applications is simplified because the management of compliance with the ML runtime API is abstracted to the MRC 170. This is because the ML application only needs to provide a generic input buffer for buffering input and output data and specify the model definition using metadata. The runner instance is created at the MRC 170 using the received information and information obtained from the centralized one or more API libraries.

[0058] This is in contrast to prior art approaches, in which the ML runtime interacts directly with the ML application according to the API of the runtime 140. In prior systems, applications must be programmed to use the ML runtime API, increasing programming complexity.

[0059] The ML workload monitor 172 is a module responsible for tracking the use of hardware platform resources by ML models. The monitoring performed by the ML workload monitor 172 may involve monitoring the use of execution cores (ML processing resources 120) and / or memory use. Other system parameters may also be monitored, such as power consumption or bandwidth consumption. However, it should be understood that any system parameter that indicates the use of resources by the system may be monitored. The information obtained by the ML workload monitor 172 may be output to the ML model scheduler 173 to assist in scheduling decisions. In addition, this information may be output to a user via a user interface (not shown) so that developers can understand the workload being deployed.

[0060] The ML model scheduler 173 is a module responsible for coordinating the use of platform resources by the ML models 161. Scheduling decisions taken by the ML model scheduler 173 may be based on metadata 163 of the ML models 161 provided to the MRC 170 by the ML applications 160. Examples of metadata used in making scheduling decisions include, among others, profiling information, memory requirements, model priorities, power requirements, etc. As will be described in more detail below, when multiple ML applications require the execution of ML models simultaneously, it is important to schedule the use of processing resources by different ML applications and ML models. As described above, the ML model scheduler may also use information received from the ML workload monitor to schedule the execution of ML models.

[0061] As long as the memory budget allows, there can be as many ML runner instances as required by the applications 160 running on the computing device 100. As such, the ML workload monitor 172 is configured to track current memory usage. If the memory budget is fully used, the ML model scheduler 173 can deallocate existing runner instances (e.g., idle or lower priority runner instances) to accommodate other runner instances.

[0062] In some embodiments, the traditional ML application 160 may interact directly with the ML runtime 140, rather than interacting with the ML runtime via the runner instance 175 in the abstraction layer 190 of the MRC 170. In this case, the application uses appropriate prior art methods to access the API of the ML runtime 140. However, in this case, the ML application 160 can be configured to report to the MRC 170 via the appropriate API when ML models begin and end execution on a given core. In this way, the MRC 170 is still aware of the execution and resource usage of those ML models so that the ML model scheduler 173 can make appropriate scheduling decisions.

[0063] As described above, the MRC 170 is configured to target a highly heterogeneous platform 150 with multiple ML runtime and execution core types. In general, the MRC 170 and its abstraction layer 190 allow ML applications 160 to work with any ML runtime 140, including open source (such as, but not limited to, Tensorflow, TFLite, TVM) or proprietary solutions (such as, but not limited to, SNPE from Qualcomm or TIDL from Texas Instruments). The open source and proprietary ML runtimes mentioned are only examples, and embodiments provide interfaces with ML runtimes in a generic manner with the aid of the abstraction layer 190. In other words, the system is not bound to any specific ML runtime system. In terms of processing cores, the MRC 170 is not limited to a specific type. The MRC 170 can target processing resources including CPUs, GPUs, DSPs, FPGAs, and / or NPUs or other core types.

[0064] Relevant information used by the ML model scheduler 173 of the MRC 170 to make scheduling decisions can be collected before creating a final runtime environment including multiple runner instances. Various types of static code analysis and profiling steps can be taken to create model metadata including typical execution times for specific network parts to execute on one or more ML runtimes.

[0065] Static information creation

[0066] As the ML model is executed, performance information may be obtained therefrom and analyzed by the ML workload monitor 172 and the ML model scheduler 173. Information about the memory requirements of various nodes in the ML model may be analyzed during compile time within its ML runtime ecosystem. The ML runtime 140 may list the peak memory required locally (including all relevant memory used during runtime and by input and output buffers), as well as the parameters (e.g., weights) of each node. This information may then be output to the MRC 170. This information may then be represented in a list as a preparation for scheduling decisions at runtime, and output within metadata 163 when a runner instance is created.

[0067] Evaluation information creation

[0068] In order to make the actual execution time of the scheduling decisions available before execution, empirically collected information can be obtained. The creation of this information will now be explained. Before executing the ML model, a benchmarking process can occur, where an automated routine independently executes a subgraph of the entire ML model on all relevant target ML runtimes 140, which can be as small as a single operator. In addition, execution benchmarks of the full graph can be performed and the results collected. In addition, initialization time can be measured.

[0069] The execution time measurements may be combined in a list along with additional information (eg, pre-calculation of average runtime, worst case runtime, and best case runtime) and output to metadata 163 .

[0070] interface

[0071] Corresponding interfaces are provided between the MRC 170 and the user-level ML application 160 on the one hand, and between the MRC 170 and the target platform 150 on the other hand.

[0072] To communicate with the MRC 170, each user-level ML application 160 uses a corresponding MRCI interface. The MRCI interface is implemented by the MRC 170. To generate an ML runner instance, the following interactions occur between the ML application 160 and the MRC 170.

[0073] ML runner instance generation

[0074] The process Figure 3 The flowchart is illustrated in FIG.

[0075] At step 3.1, a registration request is made by user-level application 160, which passes configuration parameters of the desired runner instance to MRC 170. As described above, the configuration parameters may be specified by metadata 163. At step 3.2, MRC 170 receives the configuration parameters.

[0076] Based on these parameters, at step 3.3, MRC 170 may send a request to ML runtime 140 to request the creation of runner instance 175. The request may contain parameters obtained from the metadata, such as ML model definition or expected memory usage.

[0077] However, metadata for scheduling does not need to be sent to the ML runtime 140, since scheduling is managed at the MRC 170. At step 3.4, a request to initialize a runner instance 175 is received at the ML runtime 140. If the request is successful and the ML runtime 140 allows the creation of a runner instance at the MRC 170, then at step 3.5, the model runner instance generator 171 creates the runner instance 175 in the abstraction layer 190 of the MRC 170. A corresponding runner instance reference is also created by the MRC 170. Then, at step 3.6, the runner instance reference is sent to the ML application 160 for storage.

[0078] ML Model Execution

[0079] The user-level application 160 that initiated the registration of the runner instance 175 can then request processing of input data into the ML model that specifies the runner instance to be used. Figure 4 The flowchart is illustrated in FIG.

[0080] User-level ML application 160 may send a request to MRC 170 for execution of an ML model by ML runtime 140 , which may be accessed via runner instance 175 .

[0081] The process begins in response to the ML application 160 determining that the ML model is to be executed by the ML runtime 140. At step 4.1, the ML application 160 prepares to execute the ML model by obtaining the runner reference created in the above registration process. In addition, the ML application 160 obtains input data to be input into the ML model. At step 4.2, this information is sent to the MRC 170. At step 4.3, the runner instance 175 corresponding to the runner reference receives the input data from the ML application 160.

[0082] At step 4.4, the input data of the ML model is forwarded by the runner instance 175 to the target ML runtime 140. At step 4.5, the target ML runtime 140 executes the ML model using the processing resources specified for the runner instance. The execution of the ML model by the ML runtime is performed according to the rules of the ML model and any general rules of the ML runtime itself.

[0083] The execution of step 4.4 and step 4.5 may be controlled by the ML model scheduler 173 so that the execution timing of the ML model is according to rules defined by one or more scheduling algorithms. In addition, the ML workload monitor 172 may monitor the computing resources used when executing the ML model.

[0084] At step 4.6, the ML runtime 140 provides the output data from the executed ML model to the MRC 170. The output data is then forwarded by the MRC 170 to the ML application 160 at step 4.7.

[0085] The runner instance 175 may remain active and wait for further input data from the ML application 160, or may be deinitialized by the ML scheduler 173. At step 4.8, the ML application 160 processes the received output data according to the requirements of the application.

[0086] ML Workload Monitor

[0087] It is advantageous for the MRC 170 to understand the resource usage of the target platform 150 (e.g., in terms of details such as execution time, processing core usage, memory requirements, and power consumption) by the various ML models being executed. This information can be collected by the ML workload monitor 172 during ML model execution. This information can be provided to the ML model scheduler 173 to make effective scheduling decisions. This is advantageous in situations where multiple ML applications seek to execute corresponding ML models simultaneously.

[0088] In addition, the information obtained by the ML workload monitor 172 can be output via a user interface (not shown), because this information helps developers better understand the behavior and requirements of the workload being deployed. In this way, informed design decisions can be made about how and where to deploy and optimize ML models on the target system.

[0089] The ML workload monitor 172 can provide developers with textual and visual reports of resource usage in a convenient manner. For example, a Gantt chart can be generated, such as Figure 5 As shown. The Gantt chart shows the type of cores that execute ML models on the y-axis and a timeline on the x-axis. The execution of a particular ML model on a given core is depicted as a box in the Gantt chart. The chart helps visualize the time frame in which the model runs on a given core and which models run in parallel. Such a chart can be displayed to a user via a display device (not shown).

[0090] exist Figure 5 The information shown in the figure can be output to the ML model scheduler 173 so that scheduling decisions can be automatically made using one or more scheduling algorithms.

[0091] ML Model Scheduler

[0092] The ML model scheduler 173 of the MRC 170 can use the metadata received from the ML application 160 to decide when the ML model should be executed. The ML model scheduler 173 can support both static scheduling scenarios and dynamic scheduling scenarios. Scheduling decisions can be based on various goals, such as but not limited to, model priority, system memory budget, target frame rate or performance to be achieved, power consumption, etc. Therefore, in the MRC 170, multiple scheduling algorithms can be provided according to the requirements of the system. These features make the ML model scheduler 173 flexible and scalable.

[0093] As previously described, if the memory budget is fully used, the ML model scheduler 173 may deallocate existing runner instances (e.g., idle or lower priority runner instances) to accommodate other runner instances. In this way, multiple ML applications may be accommodated simultaneously, and resources may be managed according to the respective priorities and operating conditions of the computing device 100.

[0094] As previously described, in the case where a traditional application triggers an ML model by interacting directly with the ML runtime (i.e., without using a runner instance), the application can still report this execution to the MRC 170 via a given API call. In this way, the execution of all models, whether executed via the MRC 170 or directly between the application 160 and the ML runtime 140, can be monitored and coordinated.

[0095] From the foregoing description, a number of benefits and advantages of the examples of the present disclosure will be apparent to those skilled in the art:

[0096] MRC works at an abstraction level that allows targeting highly heterogeneous platforms with multiple ML runtimes and execution core types.

[0097] The examples described herein enable high portability of ML applications by abstracting the API implementation details of a specific ML runtime from the application to the MRC.

[0098] MRC monitors ML resource usage in terms of execution cores and memory requirements as well as other parameters such as (but not limited to) bandwidth consumption or power consumption. This is useful during the development phase to better understand the ML workload being deployed. Furthermore, this information is key for the ML scheduler module to take optimal resource management decisions.

[0099] • Multiple scheduling schemes can be enabled using the model's meta-information as input in MRC, including but not limited to priority, measurement information, memory requirements, power consumption, etc.

[0100] Since the details of the ML runtime API are abstracted in a shared library accessible to MRC, the size of the application code becomes lighter.

[0101] MRC allows direct integration of different applications (provided by different application vendors) with different ML runtimes provided by different ML runtime vendors.

[0102] In the foregoing description, example aspects are described with reference to a number of example embodiments. Therefore, the description should be considered illustrative rather than restrictive. Similarly, the figures illustrated in the accompanying drawings that highlight the features and advantages of the example embodiments are presented for illustrative purposes only. The architecture of the example embodiments is sufficiently flexible and configurable so that it can be utilized in a manner different from that shown in the accompanying drawings.

[0103] In an exemplary embodiment, the software implementation of the examples presented herein may be provided as a computer program or software, such as, one or more programs with instructions or instruction sequences contained or stored in an article such as a machine-accessible medium or a machine-readable medium, an instruction storage unit, or a computer-readable storage device (each of which may be non-transitory). The program or instructions on the non-transitory machine-accessible medium, machine-readable medium, instruction storage unit, or computer-readable storage device may be used to program a computer system or other electronic device. The machine or computer-readable medium, instruction storage unit, and storage device may include, but are not limited to, a floppy disk, an optical disk, and a magneto-optical disk or other types of media / machine-readable media / instruction storage unit / storage device suitable for storing or transmitting electronic instructions. The technology described herein is not limited to any specific software configuration. They can find applicability in any computing environment or processing environment. The terms "computer-readable medium", "machine-accessible medium", "machine-readable medium", "instruction storage unit", and "computer-readable storage device" used herein shall include any medium capable of storing, encoding, or transmitting instructions or instruction sequences for execution by a machine, a computer, or a computer processor and causing the machine / computer / computer processor to perform any of the methods described herein. Furthermore, it is common in the art to refer to software in one form or another (e.g., program, process, procedure, application, module, unit, logic, etc.) as taking an action or causing a result. Such expressions are merely a shorthand way of stating that execution of the software by a processing system causes the processor to perform an action to produce a result.

[0104] Some embodiments may also be implemented by the preparation of application specific integrated circuits, field programmable gate arrays, or by interconnecting an appropriate network of conventional component circuits.

[0105] Some embodiments include computer program products. A computer program product may be a storage medium or multiple storage media, an instruction storage unit or multiple instruction storage units, or a storage device or multiple storage devices storing instructions that can be used to control or cause a computer or computer processor to perform any of the programs of the example embodiments described herein. The storage medium / instruction storage unit / storage device may include, for example, but not limited to, an optical disk, a ROM, a RAM, an EPROM, an EEPROM, a DRAM, a VRAM, a flash memory, a flash card, a magnetic card, an optical card, a nanosystem, a molecular memory integrated circuit, a RAID, a remote data storage unit / archive / warehousing, and / or any other type of device suitable for storing instructions and / or data.

[0106] Some implementations stored on a computer readable medium or multiple computer readable media, an instruction store or multiple instruction stores, or a storage device or multiple storage devices include hardware for controlling the system and software for enabling the system or microprocessor to interact with a human user or other mechanism using the results of the example implementations described herein. Such software may include, but is not limited to, device drivers, operating systems, and user applications. Finally, such computer readable media or storage devices also include software for performing the example aspects of the present disclosure as described above.

[0107] Included in the programming and / or software of the system are software modules for implementing the processes described herein. In some example embodiments herein, the modules include software, while in other example embodiments herein, the modules include hardware or a combination of hardware and software.

[0108] Although various example embodiments of the present disclosure have been described above, it should be understood that they are presented by way of example and not by way of limitation. It will be apparent to those skilled in the relevant art that various changes in form and detail may be made therein. Therefore, the present disclosure should not be limited by any of the above example embodiments, but should only be limited according to the attached claims and their equivalents.

[0109] In addition, the purpose of the abstract is to enable patent offices and the public, and especially scientists, engineers, and practitioners in the field who are not familiar with patent or legal terms or wording, to quickly determine the nature and substance of the technical disclosure of the present application from a cursory inspection. The abstract is not intended to limit the scope of the example embodiments presented herein in any way. It should also be understood that any process recited in the claims need not be performed in the order presented.

[0110] Although this specification contains many specific implementation details, these details should not be interpreted as limitations on the scope of the claimed protection, but as descriptions of features specific to the specific implementations described herein. Certain features described in the context of separate implementations in this specification may also be implemented in combination in a single implementation. On the contrary, the various features described in the context of a single implementation may also be implemented individually or in any suitable sub-combination in multiple implementations. In addition, although features may be described above as working in certain combinations and even initially claimed as such, one or more features from the claimed combination may be deleted from the combination in some cases, and the claimed combination may be directed to a sub-combination or a variation of the sub-combination.

[0111] In some cases, multitasking and parallel processing may be advantageous. In addition, the separation of various components in the above-mentioned embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0112] Now that some exemplary embodiments and multiple embodiments have been described, it is apparent that the foregoing is illustrative and not restrictive, and has been presented by way of example. In particular, although many of the examples presented herein relate to specific combinations of devices or software elements, these elements may be combined in other ways to achieve the same purpose. The actions, elements, and features discussed only in conjunction with one embodiment are not intended to exclude similar effects in other embodiments.

[0113] Without departing from the features of the devices described herein, these devices may be implemented in other specific forms. The above embodiments are illustrative rather than limiting of the described systems and methods. Therefore, the scope of the devices described herein is indicated by the appended claims rather than the preceding description, and changes falling within the meaning and range of equivalents of the claims are included therein.

Claims

1. A method for managing a machine learning environment (ML environment), the method comprising the following steps: Receiving, by the ML controller, ML model information from the ML application, the ML model information comprising an ML model definition and ML model metadata, the ML model metadata comprising information specifying an ML runtime that executes the ML model; generating, by a model runner instance generator of the ML controller, a model runner instance in an abstraction layer at the ML controller using the received ML model information and API information for a specified ML runtime accessed by the model runner instance generator from a centralized API library, the model runner instance being configured to interact with the specified ML runtime hosted by a target ML platform so that the ML runtime executes the ML model; monitoring, by a ML workload monitor of the ML controller, one or more system performance parameters during execution of the ML model; and The monitored one or more system performance parameters are used by an ML workload scheduler of the ML controller to schedule execution of the ML model.

2. The method according to claim 1, further comprising the steps of: receiving an ML execution request from the ML application requesting execution of the ML model, the ML execution request specifying a model runner instance corresponding to the ML runtime to be used, the ML execution request also including input data to be input to the ML model; During execution of the ML model, sending the input data from the designated model runner instance to the corresponding ML runtime for input to the ML model; receiving, by the model runner instance, output data from the corresponding ML runtime, wherein the output data has been output from the ML model; as well as The output data is provided to the ML application.

3. The method according to claim 1 or 2, wherein: The ML model metadata also includes information specifying the type of processing resources used to execute the ML model.

4. The method according to claim 1, further comprising the steps of: executing one or more subgraphs of the ML model to determine one or more execution time measurements for the ML model; as well as The one or more execution time measurements are provided within the ML model metadata.

5. The method according to claim 1, wherein: The ML model metadata also includes scheduling information for the ML model, and the method also includes scheduling execution of the ML model using the scheduling information.

6. The method according to claim 5, wherein: The scheduling information includes at least one of an ML model priority, a memory requirement of the ML model, a memory budget of a computing device, a target frame rate, and a power requirement of the ML model.

7. A computer program product comprising computer readable instructions which, when executed by a computer, cause the computer to perform the method according to any one of claims 1 to 6.

8. A computing device for managing a machine learning environment (ML environment), the computing device comprising: a memory storing computer-readable instructions; as well as an ML controller configured to receive ML model information from an ML application, the ML model information comprising an ML model definition and ML model metadata, the ML model metadata comprising information specifying an ML runtime that executes the ML model, Wherein, the ML controller comprises: a model runner instance generator configured to generate a model runner instance in an abstraction layer at the ML controller using the received ML model information and API information for a specified ML runtime accessed by the model runner instance generator from a centralized API library, the model runner instance configured to interact with the specified ML runtime hosted by a target ML platform so that the ML runtime executes the ML model; an ML workload monitor configured to monitor one or more system performance parameters during execution of the ML model; and A ML workload scheduler configured to schedule execution of the ML model using the monitored one or more system performance parameters.

9. The computing device of claim 8, wherein: The ML controller is also configured to: receiving an ML execution request from the ML application requesting execution of the ML model, the ML execution request specifying a model runner instance corresponding to the ML runtime to be used, the ML execution request also including input data to be input to the ML model; During execution of the ML model, sending the input data from the designated model runner instance to the corresponding ML runtime for input to the ML model; receiving, by the model runner instance, output data from the corresponding ML runtime, wherein the output data has been output from the ML model; as well as The output data is provided to the ML application.

10. The computing device of claim 8, wherein: The ML workload scheduler is configured to schedule execution of the ML model using information received from the ML workload monitor.

11. The computing device according to any one of claims 8 to 10, wherein: The ML application and the target ML platform are embedded within the computing device.

Images