Fair and simultaneous comparison of parallel machine learning models

Abstract

A method of comparing performance of a plurality of algorithms using a computing device, the method comprising receiving, by a computing device, a plurality of algorithms to be evaluated. The computing device further receives a total amount of resources to be allocated to the plurality of algorithms. The computing device additionally assigns a fair share of the total amount of resources to each of the plurality of algorithms. The computing device still further executes each of the plurality of algorithms using the assigned fair share of the total amount of resources. The computing device additionally compares the performance of each of the plurality based on at least one of a plurality of hardware relative utility metrics describing a hardware relative utility of any given resource allocation for each of the plurality of algorithms.

Description

[Technical Field]

[0001] The field of embodiments of the present invention relates to the fair comparison of parallel machine learning (ML) algorithms or models.

[0002] There are many parallel ML algorithms and even more implementations and variations. The term "parallel" with respect to a model and / or algorithm refers to distributed or federated execution on a grid, cluster, cloud, etc. to enhance performance. These ML algorithms and / or models differ in terms of required computation, memory, networking bandwidth, network latency, disk storage requirements, and specialized hardware (e.g., GPU, ASIC, etc.). For example, support vector machines (SVMs) have high computational complexity and relatively low memory requirements. Convolutional neural networks (CNNs) have medium computational complexity and medium memory requirements; K-nearest neighbor (KNN) neural networks (NNs) have high computational complexity and high memory requirements. Another example may involve federated learning versus distributed learning, each with different network utilization patterns and, therefore, different bandwidth and latency requirements. Further examples are the ratio of node-local resource requirements to total resource requirements and ideal distribution characteristics (e.g., uniform distribution). Thus, different parallel ML algorithms, and even different implementations of the same algorithm, will perform differently: depending on the particular ML problem (i.e., data) to which they are applied; and depending on different hardware configurations and architectures. Summary of the Invention

[0003] Embodiments relate to fair comparison of parallel machine learning models. One embodiment provides a method for comparing the performance of multiple algorithms using a computing device. The method includes receiving, by the computing device, multiple machines to be evaluated. The computing device further receives a total amount of resources to be allocated to the multiple algorithms. The computing device additionally allocates a fair share of the total amount of resources to each of the multiple algorithms. The computing device still further executes each of the multiple algorithms using the allocated fair share of the total amount of resources. The computing device additionally compares the performance of each of the multiple algorithms based on at least one of multiple hardware relative utility metrics that describe the relative hardware utility of any given resource allocation for each of the multiple algorithms. Embodiments significantly improve benchmarking and / or comparing different algorithms simultaneously in a fair and distributed manner. As a result, embodiments significantly improve when dividing limited distributed computing resources across different (parallel) algorithms (such as algorithms or models including, but not limited to, machine learning (ML), simulation, or search) such that biases resulting from different ideal hardware requirements / preferences are eliminated or mitigated. Some features contribute to the advantage of considering the (negative) effects of higher order distributions and resource heterogeneity in the optimized ML model, improving "fairness." Some other features contribute to the advantage that ML algorithms or models can be compared simultaneously with respect to their performance (e.g., accuracy) using the same single pool of resources.

[0004] One or more of the following features may be included: In some embodiments, the method may further include the plurality of algorithms comprising machine learning algorithms, simulation algorithms, or search algorithms, and the performance of each of the algorithms is compared in terms of execution time, accuracy, a metric, or a combination thereof.

[0005] In some embodiments, the method may further include the input for each of the plurality of algorithms including a resource bundle containing a set of resource types and required hardware resources.

[0006] In one or more embodiments, the method may further include: the input for each of the algorithms further includes a resource preference profile including the plurality of hardware relative utility metrics; bias resulting from different hardware requirements or preferences for the plurality of algorithms is removed or mitigated; and the plurality of hardware relative utility metrics are based on a resource allocation index, a resource distribution index, and a resource dispersion index.

[0007] In one or more embodiments, the method may further include calculating an optimal overall resource allocation as a Pareto-optimal, equitable or relaxed-equitable distribution of relative hardware utilities across the plurality of algorithms according to the plurality of hardware relative utility metrics.

[0008] In some embodiments, the method may further comprise determining an optimal allocation of hardware resources for each of the plurality of algorithms in a resource bundle multiplicity.

[0009] These and other features, aspects, and advantages of the present embodiments will become apparent with reference to the following description, appended claims, and accompanying drawings. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 1 is a block diagram of a process for fair comparison of different models or algorithms running in parallel, according to one embodiment.

[0011] [Figure 2] FIG. 1 illustrates an example of a pool of resources, a set of resource bundles, different ML models or algorithms, and resource preference profiles, according to one embodiment.

[0012] [Figure 3] FIG. 2 illustrates an example of a resource preference profile according to one embodiment.

[0013] [Figure 4] According to one embodiment, the relative utility parameter (

number

[0014] [Figure 5] FIG. 1 is a block diagram of a process for determining relative fairness between different resource allocations for different algorithms and / or models to be compared, according to one embodiment.

[0015] [Figure 6] 1 illustrates a cloud computing environment according to one embodiment.

[0016] [Figure 7] FIG. 2 illustrates a set of abstraction model layers according to one embodiment.

[0017] [Figure 8] FIG. 1 illustrates a network architecture of a system for determining relative fairness between different resource allocations for different ML algorithms and / or models to be compared, according to one embodiment.

[0018] [Figure 9] FIG. 7 illustrates a representative hardware environment that may be associated with the server and / or client of FIG. 6, according to one embodiment.

[0019] [Figure 10] FIG. 1 is a block diagram illustrating a distributed system for determining relative fairness between different resource allocations for different ML algorithms and / or models to be compared, according to one embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0020] The descriptions of various embodiments are presented for illustrative purposes and are not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope of the described embodiments. The terminology used herein has been selected to best explain the principles of the embodiments, practical applications, or technical improvements over commercially available technologies, or to enable others skilled in the art to understand the embodiments disclosed herein.

[0021] Embodiments relate to fair comparison of parallel algorithms. One embodiment provides a method for comparing performance of multiple algorithms using a computing device. The method includes receiving, by the computing device, multiple algorithms to be evaluated. The computing device further receives a total amount of resources to allocate to the multiple algorithms. The computing device additionally assigns a fair share of the total amount of resources to each of the multiple algorithms. The computing device still further executes each of the multiple algorithms using the assigned fair share of the total amount of resources. The computing device additionally compares performance of each of the multiple algorithms based on at least one of a plurality of relative hardware utility metrics that describe the relative hardware utility of any given resource allocation for each of the multiple algorithms.

[0022] One or more of the following features may be included: In some embodiments, the method may further include the plurality of algorithms comprising a machine learning (ML) algorithm (or model), a simulation algorithm (or model), or a search algorithm (or model), and the performance of each of these algorithms is compared in terms of execution time, accuracy, a metric, or a combination thereof.

[0023] In some embodiments, the method may further include the input for each of the plurality of algorithms including a resource bundle containing a set of resource types and required hardware resources.

[0024] In one or more embodiments, the method may further include: the input for each of the plurality of algorithms further includes a resource preference profile including a plurality of hardware relative utility metrics; bias resulting from different hardware requirements or preferences for the plurality of algorithms is removed or mitigated; and the plurality of hardware relative utility metrics are based on a resource allocation index, a resource distribution index, and a resource dispersion index.

[0025] In one or more embodiments, the method may further comprise calculating the optimal overall resource allocation as a Pareto-optimal, fair, or relaxed-fair distribution of relative hardware utilities across the multiple algorithms according to the multiple hardware relative utility metrics.

[0026] In some embodiments, the method may further comprise determining an optimal allocation of hardware resources to each of the plurality of algorithms in the resource bundle multiplicity.

[0027] One or more embodiments relate to an ML model or algorithm that utilizes one or more artificial intelligence (AI) models or algorithms. The AI ​​models may include trained ML models (e.g., models such as NNs, CNNs, recurrent NNs (RNNs), long short-term memory (LSTM)-based NNs, gated recurrent unit (GRU)-based RNNs, tree-based CNNs, KNNs, self-attention networks (e.g., NNs that utilize attention mechanisms as a basic building block; self-attention networks have been shown to be effective for sequence modeling tasks without recursion or convolution), BiLSTMs (bidirectional LSTMs), etc.). An artificial NN is an interconnected group of nodes or neurons.

[0028] One or more embodiments define a fairness metric when comparing different parallel algorithms or models with respect to their performance to mitigate the bias toward a subset of algorithms or models that typically results from utilizing uniform resource bundling (including a uniform degree of distribution) across algorithms or models. In some embodiments, a (Pareto) optimal, fair (i.e., just) partition / mapping of different computing resources across a set of different parallel algorithms or models is determined from a limited pool of resources at a pre-set, distinct degree of resource bundling, and while considering the relative impact of distribution overhead. Thus, in one or more embodiments, algorithms or models may be compared simultaneously with respect to their performance (e.g., accuracy) using the same, single pool of resources.

[0029] The approach of the embodiments significantly improves benchmarking and / or comparing different algorithms simultaneously in a fair and distributed manner. As a result, the embodiments significantly improve when dividing limited distributed computing resources across different (parallel) algorithms in a way that eliminates or mitigates biases resulting from different ideal hardware requirements / preferences. Moreover, the embodiments consider the (negative) impact of higher degrees of distribution and resource heterogeneity in the optimized ML model to further improve "fairness."

[0030] Traditional interfaces display a comparison of different ML algorithms or models run in succession. Model / algorithm performance comparison is an essential activity in ModelOps (i.e., the process of operationalizing ML models, focusing on governing and managing the lifecycle of AI and decision models (ML, learning graphs, rules, optimization agents, and linguistics-based models)) to select the best model for a manufacturing use case. Typically, different algorithms / implementations can be compared by some performance metric (e.g., receiver operating characteristic (ROC) area under the curve (AUC) score) plotted over elapsed training time and / or until some predefined timeout occurs. Automated ML (the process of automating the repetitive tasks of ML model development) techniques can help set up those experiments in the ModelOps process. Current solutions for comparing parallelized ML algorithms run different ML algorithms on a single node that is not parallelized, or on a generalized distributed infrastructure (e.g., open source software for reliable, scalable, distributed computing, open source frameworks focused on interactive query, ML, and real-time workloads): sequentially train and test one algorithm at a time, finally collect the results, and compare performance results, which results in wasted data scientists' time in ModelOps compared to one or more embodiments that utilize parallel training and testing.

[0031] Without differentiation for different hardware requirements, conventional techniques all run on the same infrastructure (e.g., open source software for reliable, scalable, distributed computing, focusing on interactive queries, ML, and real-time workloads, excluding central processing units (CPUs), graphics processing units (GPUs), tensor processing units (TPUs), TrueNorth, edge devices, etc., with different memory requirements, different network bandwidth requirements, and different algorithms / models). This leads to unfair (i.e., biased) comparisons of ML algorithms or model implementations, as model training / prediction runs on abstract hardware generalizations that may be preferable for some algorithms over others, which is favoritism.

[0032] Furthermore, traditional systems may utilize resources for test runs of ML algorithms, which can be provisioned in an ad-hoc manner. In a typical enterprise cloud, there is a total amount (or budget) of CPU, GPU, memory, network latency, network bandwidth, and disk that can be allocated for a particular ModelOps use case. Users can provision virtual machines (VMs) with different pre-built configurations or configure themselves with random access memory (RAM), disk, virtual CPU (vCPU), GPU, network, etc. This leads to a non-deterministic polynomial-time hard (NP-hard) VM packing problem.

[0033] Depending on the implementation, parallel ML algorithms may utilize model parallelism, data parallelism, or both (hybrid). The utilized degree of data parallelism and model parallelism typically both inversely correlate with solution quality (i.e., out-of-sample prediction error for the total model level given a limited training time budget), depending on myriad factors, such as batch size (either too small or too large), the model and model hyperparameters, and the characteristics of the data itself. (However, under conventional techniques, it appears relatively unclear how data features correlate with solution quality under varying degrees of parallelism.) Conventional techniques note that for NNs, increasing batch size is a straightforward way to generate worthwhile speedups across a range of workloads, but the benefits decline within the limits of current hardware. This suggests that ML solution quality may often have some inverse correlation with the degree of distribution (e.g., the number of nodes and node heterogeneity). This may not be true or equally true for all ML algorithms, but in one embodiment it is assumed that this is a general property and that more nodes have a negative impact.

[0034] Another question is: does it matter whether resources are divided equally (e.g., even distribution across only a single type of VM flavor versus a high variance of VM configurations across nodes)? It probably does matter, more so for some algorithms and less so for others; moreover, efficient scheduling across heterogeneous nodes is a practical challenge; therefore, in one embodiment, it is generally assumed that there is a negative impact when nodes are heterogeneous.

[0035] Traditional approaches to automated ML include automatically building ML pipelines and automatically selecting algorithms and hyperparameters, potentially with a human in the loop. Many traditional techniques focus specifically on deep learning (DL). Some non-DL traditional approaches consider other algorithms but do not consider fairness in terms of parallel processing and resource allocation. Some typical approaches include building a search space across algorithm / hyperparameter combinations; using an overall "time budget" for the entire model / algorithm search / optimization procedure; and timing out evaluation of individual models / algorithms within the search space (or aborting for iteratively trained models). However, the issue of "fair" comparison of parallel ML algorithms against "biases" in the underlying hardware configuration in one or more embodiments appears to be understudied in the current literature.

[0036] 1 shows a block diagram of a process 200 for fair comparison of different models or algorithms executed in parallel, according to one embodiment. In some embodiments, the inputs for process 200 are: a set of parallel ML algorithms / models I 330 (FIG. 3) to be compared, a set of available resource types T 310 (FIG. 2) and the corresponding total available pool of resources, pre-configured resource bundle definitions or "bundlings" B 320 (i.e., allowed combinations of different resource types and corresponding resource amounts when provisioning resources), and resource preference profiles (RPPs) for the ML algorithms (e.g., stored on a storage device, cloud storage, repository, etc.). i ) 210, and the parameter λ that reflects the expected negative impact of resource distribution and resource variance on ML model / algorithm performance, respectively. dist and λ var Includes.

[0037] FIG. 2 illustrates a diagram of a resource type T 310, a set of resource bundling B 320, different ML algorithms / models I 330, and an RPP according to one embodiment. i 210. In one exemplary embodiment, resource types T 310 include resource types such as CPU, RAM, network, disk, etc. In one exemplary embodiment, resource bundling B 320 includes resource bundles each including various resource types and the number of resources per type. In one exemplary embodiment, ML algorithms / models I 330 include different ML algorithms / models to be compared in parallel. RPP i 210 calculates a relative resource utility parameter ( ) with a value between 0 and 1 (i.e., a percentage) for each ML algorithm / model I 330.

number

number

[0038] FIG. 3 illustrates an exemplary RPP according to one embodiment. i 210. An exemplary RPP i In 210, the resource types / ML algorithms or models 405 include various types of ML algorithms / models I 330, such as CNN distributed 440, CNN federated 450, kNN parallel 460, and SVM parallel 470. i 210 may include other types of ML algorithms / models 1330. In an exemplary embodiment, RPP iThe top row of 210 contains various utility percentages as follows: CPU Utility %410, Memory Utility %415, Node Web Kit (NW) Node Bandwidth Utility %420, Disk Utility %425, GPU Utility %430 and Total %435, which is 100% per row. RPP i The 210 utility percentage is based on the type of ML algorithm / model I330, which inherently varies in resource type utility usage.

[0039] Returning to FIG. 1, in one or more embodiments, the output for process 200 includes: the allocation of resources in the form of multiplicities of resource bundling B320 allocated to the individual parallel ML algorithms / models I330 to be compared, and a metric determining the relative "fairness" between the resource allocations described above.

[0040] In some embodiments, process 200 uses RPPs 210 of different ML algorithms, created from empirical evidence and / or theoretical considerations, in combination with a mathematical optimization procedure / process to enable fair simultaneous comparison of ML models / algorithms I330 when dividing limited distributed computing resources across different (parallel) ML algorithms such that biases arising from different ideal hardware requirements / preferences are eliminated or mitigated, and further takes into account the (negative) impact of higher degrees of distribution and resource heterogeneity of the optimization models described above to further improve "fairness."

[0041] In one or more embodiments, processing for block 220 of process 200 initializes a pool of resources 300 (FIG. 2) that includes a set of resource types T 310, pre-configured resource bundling B 320, and ML algorithms or models I 330 to be compared. In one embodiment, processing for block 220 includes the following: Let T be the set of different resource types T 310 (e.g., CPU, GPU, application specific integrated circuit (ASIC), RAM, network bandwidth, etc.) in the resource allocation pool 300, where t=|T|. The total available amount of each resource of type k (1≦k≦t) is c k Let B be the set of possible different discrete combinations of resource types (resource bundling B320, e.g., VM flavors), where m=|B|, e.g., B={B1, B2,..., B m} Bundling B j (1≦j≦m) is the number of resources of type k (1≦k≦t).

number

number

number

number

[0042] FIG. 4 illustrates a diagram of a resource preference profile (RPP) and its respective resource preference profile (RPP) according to one embodiment. i )210 relative utility parameter (

number

[0043] 1, in one embodiment, in block 230, process 200 calculates the optimal allocation of resources to ML algorithms or models in the multiplicity of resource bundling B320 (FIG. 2). In one embodiment, the allocation of resources for algorithm i∈I is denoted as S i Let, i.e., the multiset

number

number

number

number

number

[0044] In some embodiments, block 230 of process 200 comprises determining the optimal parameters.

number

number

number

number

[0045] In one or more embodiments, the resource allocation index 520 refers to how much resources are allocated to a particular ML algorithm or model compared to others. The resource allocation index 520 has a range of 0...1, where higher is better, 0 being the lowest and 1 being the highest relative resource allocation:

number

[0046] In one or more embodiments, the resource distribution index 530 indicates how distributed the i-th workload is across a number of instances of resource bundling B 320 (FIG. 2) compared to the least distributed workload of all. 。

number

number

[0047] In some embodiments, the resource variance index 540 refers to how much variance there is in the resource configuration of the instance of resource bundling B 320 assigned to a particular workload compared to the minimum variance in the set of all workloads. The resource variance index 540 ranges from 0...1, where lower is better, 0 being the minimum and 1 being the maximum relative variance. :

number

[0048] In some embodiments, various use cases for process 200 (FIG. 1) are as follows: The interface may be utilized for a product that compares ML algorithms or models. Process 200 may be utilized for a product in which a metric for fairness regarding bias when comparing different ML algorithms or models is used on a uniform hardware architecture. Process 200 may additionally be used for a product for benchmarking ML algorithms and models. Process 200 may be used for AI (e.g., a “robot data scientist”) to automatically select the correct ML techniques for automatically designing a specific ML solution by comparing different ML algorithms or models. Process 200 may further be used for a product to automatically estimate the amount of resources required in an enterprise cloud to reach a specific goal in terms of required learning performance (i.e., accuracy) given an ML algorithm or model and a desired time frame for training. Process 200 may also be used to transform what is currently a completely ad hoc and manual process—the selection and provisioning of hardware resources / configurations for a specific enterprise ML solution—into an automated, deterministic, and well-governed process.

[0049] FIG. 5 illustrates a block diagram of a process 600 for determining relative fairness between different resource allocations for different algorithms and / or models to be compared, according to one embodiment. In one embodiment, at block 610, process 600 receives, by a computing device (e.g., from computing node 10, FIG. 6, hardware and software layer 60, FIG. 7, processing system 900, FIG. 8, system 1000, FIG. 9, system 1100, FIG. 10, etc.), multiple machine learning algorithms to be evaluated (e.g., algorithm or model I 330, FIG. 2). At block 620, process 600 further provides for receiving, by the computing device, a total amount of resources to allocate to the multiple algorithms. At block 630, process 600 receives, by the computing device, a resource allocation (e.g., RRP) for each of the multiple algorithms. i 2-3 using the method described above. At block 640, process 600 additionally provides for the computing device executing each of the plurality of algorithms using the allocated fair share of the total amount of resources. At block 650, process 600 further provides for the computing device comparing performance of each of the plurality of algorithms based on at least one of a plurality of relative hardware utility metrics describing the relative hardware utility of any given resource allocation for each of the plurality of algorithms.

[0050] In one embodiment, the process 600 may further include the feature that the plurality of algorithms includes machine learning algorithms, simulation algorithms, or search algorithms, and the performance of each of the algorithms is compared in terms of execution time, accuracy, a metric, or a combination thereof.

[0051] In one embodiment, process 600 may additionally include a feature in which the input for each of the plurality of algorithms includes a set of resource types (e.g., resource type T 310, FIG. 2) and a resource bundle (e.g., resource bundling B 320, FIG. 2) that includes the required hardware resources.

[0052] In one embodiment, the process 600 further comprises: an input for each of the plurality of algorithms being a resource preference profile (e.g., RRP) that includes a large number of hardware relative utility metrics; i 2-3 using 210), where bias resulting from different hardware requirements or preferences for multiple algorithms is eliminated or mitigated, and this bulk hardware relative utility metric may include a feature based on a resource allocation index (e.g., resource allocation index 520, FIG. 4), a resource distribution index (e.g., resource distribution index 530, FIG. 4), and a resource variance index (e.g., resource variance index 540, FIG. 4).

[0053] In one embodiment, process 600 may further include the feature of computing, by a computing device, the optimal overall resource allocation as a Pareto-optimal, fair, or relaxed-fair distribution of relative hardware utilities across multiple algorithms according to a plethora of hardware relative utility metrics.

[0054] In one embodiment, the process 600 may still further include the feature of determining, by the computing device, an optimal allocation of hardware resources to each of the plurality of algorithms in the multiplicity of the resource bundle.

[0055] Although this disclosure includes detailed descriptions of cloud computing, it is understood in advance that implementation of the teachings described herein is not limited to cloud computing environments. Rather, embodiments of the present invention can be implemented in conjunction with any other type of computing environment now known or later developed.

[0056] Cloud computing is a model of service delivery that enables convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, network bandwidth, servers, processing, memory, storage, applications, virtual machines (VMs), and services) that can be rapidly provisioned and released with minimal administrative effort or interaction with a service provider. The cloud model may include at least five characteristics, at least three service models, and at least four deployment models.

[0057] The characteristics are as follows:

[0058] On-Demand Self-Service: Cloud consumers can unilaterally provision computing capacity, such as server time and network storage, automatically as needed, without requiring human interaction with the service provider.

[0059] Wide network access: This capability is available over the network and accessed through standard mechanisms that facilitate use by heterogeneous thin or thick client platforms (eg, cell phones, laptops, and PDAs).

[0060] Resource Pooling: A provider's computing resources are pooled to serve multiple consumers using a multi-tenant model, with different physical and virtual resources dynamically allocated and reallocated according to demand. There is location independence in that consumers generally have no control or knowledge over the exact location of the provided resources, but may be able to specify location at a higher level of abstraction (e.g., country, state, or data center).

[0061] Rapid Elasticity: This capacity can be rapidly and elastically provisioned, in some cases automatically, to rapidly scale out, and rapidly released to rapidly scale in. To the consumer, the capacity available for provisioning often appears unlimited, and can be purchased in any quantity at any point in time.

[0062] Metered Services: Cloud systems automatically control and optimize resource usage by leveraging metering capabilities at a level of abstraction appropriate to the type of service (e.g., storage, processing, bandwidth, and active consumer accounts). Resource usage can be monitored, controlled, and reported, providing transparency to both providers and consumers of the services utilized.

[0063] The service model is as follows:

[0064] Software as a Service (SaaS): The consumer is offered the ability to use a provider's applications running on a cloud infrastructure. The applications are accessible from a variety of client devices through a thin-client interface such as a web browser (e.g., web-based email). The consumer does not manage or control the underlying cloud infrastructure, including the network, servers, operating systems, storage, or even individual application capabilities, with the possible exception of limited consumer-specific application configuration settings.

[0065] Platform as a Service (PaaS): The ability offered to consumers is the ability to deploy applications they create or acquire, written using programming languages ​​and tools supported by the provider, onto a cloud infrastructure. The consumer does not manage or control the underlying cloud infrastructure, including networks, servers, operating systems, or storage, but does control the deployed applications and, in some cases, the application hosting environment configuration.

[0066] Infrastructure as a Service (IaaS): The ability offered to consumers is the ability to provision processing, storage, network, and other basic computing resources, on which the consumer can deploy and run any software, which may include operating systems and applications. The consumer does not manage or control the underlying cloud infrastructure, but does control the operating systems, storage, deployed applications, and possibly limited control over selected networking components (e.g., host firewalls).

[0067] The deployment model is as follows:

[0068] Private Cloud: This cloud infrastructure operates solely for an organization. It may be managed by the organization or a third party and may exist on-premise or off-premise.

[0069] Community Cloud: This cloud infrastructure is shared by several organizations and supports a specific community with shared interests (e.g., mission, security requirements, policies and compliance considerations). The community cloud may be managed by those organizations or a third party and may exist on-premises or off-premises.

[0070] Public Cloud: This cloud infrastructure is made available to the general public or large industry groups and is owned by an organization that sells cloud services.

[0071] Hybrid Cloud: This cloud infrastructure is a composite of two or more clouds (private, community, or public) that remain distinct entities but are bound together by standard or proprietary technologies that allow for data and application portability (e.g., cloud bursting for load balancing between clouds).

[0072] Cloud computing environments are service-oriented with a focus on statelessness, low coupling, modularity, and semantic interoperability. At the core of cloud computing is an infrastructure that includes a network of interconnected nodes.

[0073] Referring now to FIG. 6, an exemplary cloud computing environment 50 is shown. As shown, the cloud computing environment 50 comprises one or more cloud computing nodes 10 with which local computing devices used by cloud consumers, such as, for example, a personal digital assistant (PDA) or mobile phone 54A, a desktop computer 54B, a laptop computer 54C, and / or an automobile computer system 54N, may communicate. The nodes 10 may communicate with each other. They may be physically or virtually grouped in one or more networks (not shown), such as a private cloud, a community cloud, a public cloud, or a hybrid cloud, or combinations thereof, as described hereinabove. This enables the cloud computing environment 50 to provide infrastructure, platforms, and / or software as a service for which cloud consumers are not required to maintain resources on their local computing devices. The types of computing devices 54A-N shown in FIG. 6 are intended for illustrative purposes only, and it will be understood that the computing nodes 10 and the cloud computing environment 50 may communicate with any type of computerized device via any type of network and / or network-addressable connection (e.g., using a web browser).

[0074] Referring now to Figure 7, a set of functional abstraction layers provided by cloud computing environment 50 (Figure 6) is shown. It should be understood in advance that the components, layers, and functions shown in Figure 7 are intended to be merely exemplary, and embodiments are not limited thereto. As shown, the following layers and corresponding functions are provided:

[0075] Hardware and software layer 60 comprises hardware and software components. Examples of hardware components include: mainframe 61; RISC (reduced instruction set computer) architecture-based servers 62; servers 63; blade servers 64; storage devices 65; and network and networking components 66. In some embodiments, software components include network application server software 67 and database software 68.

[0076] The virtualization layer 70 provides an abstraction layer over which the following examples of virtual entities can be provided: virtual servers 71; virtual storage 72; virtual networks 73, including virtual private networks; virtual applications and operating systems 74; and virtual clients 75.

[0077] In one example, management layer 80 may provide the functions described below. Resource provisioning 81 provides dynamic procurement of computing and other resources used to execute tasks within the cloud computing environment. Metering and pricing 82 provides cost tracking as resources are utilized within the cloud computing environment and billing or invoicing for the consumption of these resources. In one example, these resources may include application software licenses. Security provides identity verification for cloud consumers and tasks, as well as protection for data and other resources. User portal 83 provides access to the cloud computing environment for consumers and system administrators. Service level management 84 provides cloud computing resource allocation and management so that required service levels are met. Service level agreement (SLA) planning and fulfillment 85 provides advance arrangements and procurement of cloud computing resources where future requirements are anticipated according to SLAs.

[0078] Workload layer 90 provides examples of functionality for which a cloud computing environment may be utilized. Examples of workloads and functionality that may be provided from this layer include: mapping and navigation 91; software development and lifecycle management 92; virtual classroom instructional delivery 93; data analysis processing 94; transaction processing 95; and relative fairness measurement processing 96 between different resource allocations for different ML algorithms and / or models to be compared (e.g., process 600, FIG. 5, system 900, FIG. 8, system 1000, FIG. 9, system 1100, FIG. 10, etc.). As mentioned above, all of the foregoing examples described with respect to FIG. 7 are merely illustrative, and embodiments are not limited to these examples.

[0079] Although this disclosure includes detailed descriptions of cloud computing, it is reiterated that implementation of the teachings described herein is not limited to cloud computing environments. Rather, embodiments can be implemented in conjunction with any other type of computing environment now known or later developed.

[0080] 8 illustrates a network architecture for a system 900 for determining relative fairness between different resource allocations for different ML algorithms and / or models to be compared, according to one embodiment. As shown in FIG. 8, multiple remote networks 902 are provided, including a first remote network 904 and a second remote network 906. A gateway 901 may be coupled between the remote network 902 and a proximate network 908. In the context of this network architecture 900, the networks 904, 906 may each take any form, including, but not limited to, a LAN, a WAN, such as the Internet, a public switched telephone network (PSTN), an internal telephone network, etc.

[0081] In use, gateway 901 acts as an entry point from remote network 902 to proximal network 908. Gateway 901 may therefore function as a router capable of directing a given packet of data arriving at gateway 901, and as a switch that provides the actual path into and out of gateway 901 for a given packet.

[0082] Also included is at least one data server 914 coupled to the proximate network 908, which is accessible from the remote network 902 via the gateway 901. Note that the data server 914 may include any type of computing device / groupware. Coupled to each data server 914 are multiple user devices 916. Such user devices 916 may include desktop computers, laptop computers, handheld computers, printers, and / or any other type of logic-containing device. Note that the user devices 916 may be directly coupled to any of the networks in some embodiments.

[0083] A peripheral 920 or a series of peripherals 920, such as a facsimile machine, a printer, a scanner, a hard disk drive, a network and / or local storage unit or system, etc., may be coupled to one or more of the networks 904, 906, 908. It should be noted that databases and / or additional components may be utilized with or integrated into any type of network element coupled to the networks 904, 906, 908. In the context of this description, a network element may refer to any component of a network.

[0084] According to some approaches, the methods and systems described herein may be implemented using and / or on virtual systems and / or systems that emulate one or more other systems, such as a UNIX system emulating an IBM® z / OS environment, a UNIX system virtually hosting a MICROSOFT® WINDOWS® environment, a MICROSOFT® WINDOWS® system emulating an IBM® z / OS environment, etc. Virtualization and / or emulation may be implemented in some embodiments through the use of VMWARE® software.

[0085] Figure 9 illustrates a representative hardware system 1000 environment associated with the user device 916 and / or server 914 of Figure 8, according to one embodiment. In one example, the hardware configuration comprises a workstation having a central processing unit 1010, such as a microprocessor, and several other units interconnected via a system bus 1012. The workstation illustrated in Figure 9 may also include an I / O adapter 1018 that connects peripheral devices, such as random access memory (RAM) 1014, read-only memory (ROM) 1016, and disk storage unit 1020, to the bus 1012; a user interface adapter 1022 that connects a keyboard 1024, a mouse 1026, speakers 1028, a microphone 1032, and / or other user interface devices, such as a touch screen, a digital camera (not shown), etc., to the bus 1012; a communications adapter 1034 that connects the workstation to a communications network 1035 (e.g., a data processing network); and a display adapter 1036 that connects the bus 1012 to a display device 1038.

[0086] In one example, the workstation may have resident an operating system such as the MICROSOFT® WINDOWS® Operating System (OS), MAC OS®, UNIX® OS, etc. In one embodiment, the system 1000 utilizes a POSIX®-based file system. It is understood that other examples may be implemented on platforms and operating systems other than those mentioned. Such other examples include operating systems written using JAVA®, XML, C, and / or C++, or other programming languages, along with object-oriented programming methodologies. Object-oriented programming (OOP), which is increasingly being used to develop complex applications, may also be used.

[0087] 10 is a block diagram illustrating a distributed system 1100 for determining relative fairness between different resource allocations for different ML algorithms and / or models to be compared, according to one embodiment. In one embodiment, system 1100 includes client devices 1110 (e.g., mobile devices, smart devices, computing systems, etc.), a cloud or resource sharing environment 1120 (e.g., a public cloud computing environment, a private cloud computing environment, a data center, etc.), and a server 1130. In one embodiment, client devices 1110 are provided cloud services by server 1130 through cloud or resource sharing environment 1120.

[0088] One or more embodiments may be systems, methods, and / or computer program products at any possible level of technical detail of integration. The computer program product may include a computer-readable storage medium (or media) having computer-readable program instructions for causing a processor to perform aspects of the embodiments.

[0089] A computer-readable storage medium may be a tangible device that can hold and store instructions for use by an instruction execution device. A computer-readable storage medium may be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of computer-readable storage media includes the following: portable computer diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disk read-only memory (CD-ROM), digital versatile disk (DVD), memory sticks, floppy disks, mechanically encoded devices such as punch cards or ridge structures in grooves that record instructions, and any suitable combination of the foregoing. Computer-readable storage medium, as used herein, should not be construed as a transitory signal per se, such as an electric wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse passing through a fiber optic cable), or an electrical signal transmitted through a wire.

[0090] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device or to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network may include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and transfers the computer-readable program instructions to a computer-readable storage medium in the respective computing / processing device for storage.

[0091] The computer-readable program instructions for carrying out the operations of the present embodiments may be either assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, configuration data for an integrated circuit, or source or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk®, C++, etc., and procedural programming languages ​​such as the “C” programming language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection may be to an external computer (e.g., via the Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example, a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA) can execute computer readable program instructions to personalize the electronic circuitry by utilizing state information of the computer readable program instructions to perform aspects of the present embodiments.

[0092] Aspects of the embodiments are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems) and computer program products. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0093] These computer-readable program instructions can be provided to a processor of a computer or other programmable data processing apparatus to produce a machine, whereby the instructions, executed by the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / acts specified in one or more blocks of the flowcharts and / or block diagrams. These computer-readable program instructions can also be stored on a computer-readable storage medium, whereby the instructions can instruct a computer, programmable data processing apparatus, and / or other device to function in a particular manner, such that the computer-readable storage medium having the instructions stored thereon comprises an article of manufacture including instructions that implement aspects of the functions / acts specified in one or more blocks of the flowcharts and / or block diagrams.

[0094] The computer-readable program instructions may also be loaded into a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be executed on the computer, other programmable apparatus, or other device to generate a computer-implemented process, whereby the instructions executing on the computer, other programmable apparatus, or other device implement the function(s) / act(s) specified in one or more blocks of the flowcharts and / or block diagrams.

[0095] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of instructions, including one or more executable instructions, that implement the specified logical function. In some alternative implementations, the functions noted in the blocks may occur out of the order noted in the figures. For example, two blocks shown in succession may actually be realized as a single step and may be executed simultaneously, substantially simultaneously, partially, or fully in a time-overlapping manner, or the blocks may possibly be executed in the reverse order, depending on the functionality involved. It should also be noted that each block of the block diagrams and / or flowchart diagrams, and combinations of blocks in the block diagrams and / or flowchart diagrams, may be implemented by a dedicated hardware-based system that performs the specified functions or operations or executes a combination of dedicated hardware and computer instructions.

[0096] References to elements in the claims in the singular are not intended to mean "one and only" unless expressly stated otherwise, but rather "one or more." All structural and functional equivalents to the elements of the exemplary embodiments described above that are now known or that later become known to those skilled in the art are intended to be encompassed by the claims. No claim element herein shall be construed under the provisions of 35 U.S.C. 112, sixth paragraph, unless the element is expressly recited using the phrase "means for" or "step for."

[0097] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the embodiments. As used herein, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and / or "comprising," when used herein, specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0098] In the following claims, corresponding structure, material, acts, and equivalents of all means-plus-function or step-plus-function elements are intended to include any structure, material, or acts for performing a function in combination with other claimed elements as specifically claimed. The description of the present embodiments has been presented for purposes of illustration and description, but is not intended to be exhaustive or limited to the embodiments in the form disclosed. Many modifications and variations will be apparent to those skilled in the art without departing from the scope of the embodiments. The embodiments were chosen and described to best explain the principles and practical applications of the embodiments and to enable others skilled in the art to understand the embodiments for various embodiments, with various modifications suitable for the particular use contemplated.

Claims

1. receiving, by a computing device, a plurality of machine learning (ML) algorithms; receiving, by the computing device, a total amount of hardware resources; receiving, by the computing device, a resource preference profile for each of the plurality of ML algorithms, the resource preference profile including a plurality of hardware relative utility metrics describing a hardware relative utility indicating a relative usefulness of each of a plurality of hardware resources; allocating, by the computing device, an allocation of the total amount of the hardware resources to each of the plurality of ML algorithms such that a difference between subjective utilities based on the resource preference profiles indicating usefulness of the allocation of the plurality of hardware resources for each of the plurality of ML algorithms is less than or equal to a predetermined value; and executing, by the computing device, each of the plurality of ML algorithms using the assigned share of the total amount of hardware resources; A method comprising:

2. The method of claim 1 , wherein the plurality of ML algorithms includes a simulation algorithm or a search algorithm.

3. The method of claim 2 , wherein the performance of each algorithm of the plurality of ML algorithms is measured in terms of execution time, accuracy, a metric, or a combination thereof.

4. A method as described in claim 2 or 3, wherein at the stage of receiving the resource preference profile, a resource bundle is received containing a set of resource types and required hardware resources.

5. The method described in claim 4, wherein the multiple hardware relative utility metrics are based on a resource allocation index, a resource distribution index, and a resource dispersion index.

6. The method described in claim 5, wherein the allocation of the total amount of hardware resources to each of the plurality of ML algorithms is based on a Pareto-optimal, fair or relaxed fair distribution of relative hardware utilities across the plurality of ML algorithms according to the plurality of hardware relative utility metrics.

7. A method according to any one of claims 1 to 6, wherein the allocation of the total amount of hardware resources to each of the plurality of ML algorithms is based on the multiplicity of resource bundles.

8. A computer program, the computer program causing a processor to: receiving, by the processor, a plurality of machine learning (ML) algorithms; receiving, by the processor, a total amount of hardware resources; receiving, by the processor, a resource preference profile for each of the plurality of ML algorithms, the resource preference profile including a plurality of hardware relative utility metrics describing a hardware relative utility indicating a relative usefulness of each of a plurality of hardware resources; allocating, by the processor, a share of the total amount of the hardware resources to each of the plurality of ML algorithms such that a difference between subjective utilities based on the resource preference profiles, which indicate the usefulness of the share of the plurality of hardware resources for each of the plurality of ML algorithms, is equal to or less than a predetermined value; and executing, by the processor, each of the plurality of ML algorithms using the assigned share of the total amount of hardware resources; A computer program for executing

9. The computer program product of claim 8 , wherein the plurality of ML algorithms include a simulation algorithm or a search algorithm.

10. The computer program product of claim 9 , wherein the performance of each of the ML algorithms is measured in terms of execution time, accuracy, a metric, or a combination thereof.

11. A computer program as described in claim 9 or 10, wherein in the step of receiving the resource preference profile, a resource bundle including a set of resource types and required hardware resources is received.

12. The computer program of claim 11, wherein the multiple hardware relative utility metrics are based on a resource allocation index, a resource distribution index, and a resource dispersion index.

13. The computer program of claim 12, wherein the allocation of the total amount of hardware resources to each of the plurality of ML algorithms is based on a Pareto-optimal, fair or relaxed fair distribution of relative hardware utilities across the plurality of ML algorithms according to the plurality of hardware relative utility metrics.

14. The allocation of the total amount of hardware resources to each of the plurality of ML algorithms is 14. A computer program according to any one of claims 9 to 13, based on the multiplicity of resource bundles.

15. a memory configured to store instructions; and Processor wherein the processor executes the following instructions: instructions for receiving a plurality of ML algorithms; instructions for receiving a total amount of hardware resources; instructions for receiving, for each of the plurality of ML algorithms, a resource preference profile including a plurality of hardware relative utility metrics describing a hardware relative utility indicating a relative usefulness of each of a plurality of hardware resources; instructions for allocating an allocation of the total amount of the hardware resources to each of the plurality of ML algorithms such that a difference between subjective utilities based on the resource preference profile, which indicates a usefulness of the allocation of the plurality of hardware resources for each of the plurality of ML algorithms, is less than or equal to a predetermined value; and instructions for executing each of the plurality of ML algorithms using the assigned share of the total amount of hardware resources; 20. An apparatus configured to:

16. The apparatus of claim 15 , wherein the plurality of ML algorithms include a simulation algorithm or a search algorithm.

17. The apparatus of claim 16 , wherein performance of each algorithm of the plurality of ML algorithms is measured in terms of execution time, accuracy, a metric, or a combination thereof.

18. The apparatus of claim 16 or 17, wherein the instructions for receiving the resource preference profile include instructions for receiving a resource bundle including a set of resource types and required hardware resources.

19. The apparatus of claim 18, wherein the plurality of hardware relative utility metrics are based on a resource allocation index, a resource distribution index, and a resource dispersion index.

20. The apparatus of claim 19, wherein the allocation of the total amount of hardware resources to each of the plurality of ML algorithms is based on a Pareto-optimal, fair or relaxed fair distribution of relative hardware utilities across the plurality of ML algorithms according to the plurality of hardware relative utility metrics.

Images