Fair and simultaneous comparison of parallel machine learning models

Abstract

A method of comparing performance of a plurality of algorithms using a computing device, the method comprising receiving, by the computing device, the plurality of algorithms for evaluation. The computing device also receives a total amount of resources to be allocated to the plurality of algorithms. The computing device also allocates a fair share of the total amount of resources to each of the plurality of algorithms. The computing device also executes each of the plurality of algorithms using the allocated fair share of the total amount of resources. The computing device also compares performance of each of the plurality of algorithms based on at least one of a plurality of hardware relative utility metrics that describe a hardware relative utility of any given resource allocation for each of the plurality of algorithms.

Description

Technical Field

[0001] The field of embodiments of the present invention relates to fair comparison of parallel machine learning (ML) algorithms or models. Background Technology

[0002] There are numerous parallel ML algorithms, and even more implementations and variations. The term parallelism in relation to models and / or algorithms involves running them in a distributed or collaborative manner on grids, clusters, clouds, etc., to improve performance. These ML algorithms and / or models differ in terms of required computation, memory, network bandwidth, network latency, disk storage requirements, and specialized hardware (e.g., GPUs, ASICs, etc.). For example, Support Vector Machines (SVMs) have high computational complexity and relatively low memory requirements. Convolutional Neural Networks (CNNs) have moderate computational complexity and moderate memory requirements; K-Nearest Neighbors Neural Networks (KNNs) have high computational complexity and high memory requirements. Another example might involve collaborative learning versus distributed learning, where each collaborative learning instance has a different network utilization pattern and therefore different bandwidth and latency requirements. Another example is the ratio of node-local resource requirements to overall resource requirements and ideal distribution characteristics (e.g., uniform distribution). Therefore, different parallel ML algorithms, and even different implementations of the same algorithm, perform differently: depending on the specific ML problem they are applied to (i.e., the data); and depending on different hardware configurations and architectures. Summary of the Invention

[0003] The embodiments relate to fair comparisons of parallel machine learning models. One embodiment provides a method for comparing the performance of multiple algorithms using a computing device. The method includes receiving multiple machines for evaluation by the computing device. The computing device also receives a total amount of resources to be allocated to the multiple algorithms. The computing device also allocates a fair share of the total resources to each of the multiple algorithms. The computing device also executes each of the multiple algorithms using the fair share of the allocated total resources. The computing device also compares the performance of each of the multiple algorithms based on at least one of a plurality of hardware relative utility metrics describing the hardware relative utility of any given resource allocation for each of the multiple algorithms. The embodiments significantly improve benchmarking and / or comparison of different algorithms in a fair and distributed manner. As a result, the embodiments significantly provide the ability to eliminate or mitigate biases arising from different ideal hardware requirements / preferences when dividing limited distributed computing resources across different (parallel) algorithms, such as algorithms or models including but not limited to machine learning (ML), simulation, or search. Some features take into account the (negative) effects of higher distribution and resource heterogeneity in optimizing ML models and contribute to the benefits of improved "fairness". Several other features help to compare the performance (e.g., accuracy) advantages of ML algorithms or models simultaneously using the same single resource pool.

[0004] It may include one or more of the following features. In some embodiments, the method may also include multiple algorithms, including machine learning algorithms, simulation algorithms, or search algorithms. The performance of each algorithm is compared based on execution time, accuracy, metrics, or a combination thereof.

[0005] In some embodiments, the method may further include: the input for each of a plurality of algorithms includes a set of resource types and a resource bundle including the required hardware resources.

[0006] In one or more embodiments, the method may further include that the input of each algorithm further includes a resource optimization profile, which includes multiple hardware relative utility metrics to eliminate or mitigate biases caused by different hardware requirements or preferences of the multiple algorithms, and the multiple hardware relative utility metrics are based on a resource allocation index, a resource distribution index, and a resource variance index.

[0007] In one or more embodiments, the method may include calculating the optimal overall resource allocation as a Pareto optimal, fair, or loosely fair distribution of relative hardware utility across multiple algorithms based on multiple hardware relative utility metrics.

[0008] In some embodiments, the method may further include determining the optimal allocation of hardware resources to each of the plurality of algorithms using a plurality of resource bundles.

[0009] These and other features, aspects, and advantages of this embodiment will be understood with reference to the following description, the appended claims, and the accompanying drawings. Attached Figure Description

[0010] Figure 1 A block diagram is shown, according to one embodiment, of a process for fairly comparing different models or algorithms that are executed in parallel;

[0011] Figure 2 Examples of resource pools, a set of resource bundles, different ML models or algorithms, and resource preference profiles according to one embodiment are shown;

[0012] Figure 3 An example of a resource preference profile according to one embodiment is shown;

[0013] Figure 4 This illustrates a subjective utility equation according to one embodiment, the subjective utility being calculated by a machine learning (ML) algorithm (S). i According to the relative utility parameter in its corresponding resource preference profile (i) Resource Providing (RPP) i ) specific allocation;

[0014] Figure 5A block diagram is shown, according to one embodiment, of a process for determining the relative fairness between different resource allocations for different algorithms and / or models to be compared;

[0015] Figure 6 A cloud computing environment according to an embodiment is described;

[0016] Figure 7 A set of abstract model layers according to an embodiment is shown;

[0017] Figure 8 This is a network architecture of a system according to an embodiment for determining the relative fairness between different resource allocations for different ML algorithms and / or models to be compared;

[0018] Figure 9 The embodiments shown can be used with Figure 6 The representative hardware environment associated with the server and / or client; and

[0019] Figure 10 This is a block diagram illustrating a distributed system according to one embodiment for determining the relative fairness between different resource allocations for different ML algorithms and / or models to be compared. Detailed Implementation

[0020] Various embodiments have been described for illustrative purposes, but are not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope of the described embodiments. The terminology used herein has been chosen to best explain the principles of the embodiments, their practical application, or improvements to existing technologies in the market, or to enable those skilled in the art to understand the embodiments disclosed herein.

[0021] The embodiments relate to fair comparisons for parallel algorithms. One embodiment provides a method for comparing the performance of multiple algorithms using a computing device. The method includes receiving multiple algorithms for evaluation by the computing device. The computing device also receives a total amount of resources to be allocated to the multiple algorithms. The computing device also allocates a fair share of the total resources to each of the multiple algorithms. The computing device also uses the allocated fair share of the total resources to execute each of the multiple algorithms. The computing device further compares the performance of each of the multiple algorithms based on at least one of a plurality of hardware relative utility metrics describing the hardware relative utility of any given resource allocation for each of the multiple algorithms.

[0022] It may include one or more of the following features. In some embodiments, the method may also include multiple algorithms, including machine learning (ML) algorithms (or models), simulation algorithms (or models), or search algorithms (or models). The performance of each algorithm is compared based on execution time, accuracy, metrics, or a combination thereof.

[0023] In some embodiments, the method may further include: the input for each of a plurality of algorithms includes a set of resource types and a resource bundle including the required hardware resources.

[0024] In one or more embodiments, the method may further include that the input for each of the plurality of algorithms further includes a resource preference profile, the resource preference profile including a plurality of hardware relative utility metrics that eliminate or mitigate biases caused by different hardware requirements or preferences of the plurality of algorithms, and the plurality of hardware relative utility metrics are based on a resource allocation index, a resource distribution index, and a resource change index.

[0025] In one or more embodiments, the method may include calculating the optimal overall resource allocation as a Pareto optimal, fair, or loosely fair distribution of relative hardware utility across multiple algorithms based on multiple hardware relative utility metrics.

[0026] In some embodiments, the method may further include determining the optimal allocation of hardware resources to each of the plurality of algorithms using a plurality of resource bundles.

[0027] One or more embodiments relate to ML models or algorithms employing one or more artificial intelligence (AI) models or algorithms. AI models may include trained ML models (e.g., models such as NN, CNN, recurrent NN (RNN), long-term memory (LSTM) based NN, gated recurrent unit (GRU) based RNN, tree-based CNN, KNN, self-attention networks (e.g., NNs that utilize attention mechanisms as basic building blocks; self-attention networks have been shown to be effective for sequence modeling tasks without recursion or convolution), BiLSTM (bidirectional LSTM), etc. An artificial NN is an interconnected group of nodes or neurons.

[0028] One or more embodiments define a fairness metric when comparing the performance of different parallel algorithms or models to mitigate bias towards subsets of algorithms or models that typically result from indiscriminate resource bundling (including the degree of indiscriminate distribution) across algorithms or models. In some embodiments, a (Pareto-optimal) fair (i.e., equitable) partitioning / mapping of different computational resources across a set of different parallel algorithms or models is determined: from a finite resource pool, in multiple pre-defined discrete resource bundles, while simultaneously considering the relative impact of distributed overhead. Thus, in one or more embodiments, the same single resource pool can be used to compare the performance (e.g., accuracy) of algorithms or models simultaneously.

[0029] The method of the embodiments significantly improves benchmarking and / or comparison of different algorithms in a fair and distributed manner. As a result, the embodiments significantly provide the elimination or mitigation of biases arising from different ideal hardware requirements / preferences when dividing limited distributed computing resources across different (parallel) algorithms. Furthermore, the embodiments consider the (negative) effects of higher distribution and resource heterogeneity in the ML model to further improve "fairness".

[0030] Traditional interfaces display comparisons of different ML algorithms or models executed sequentially. Model / algorithm performance comparison is a key activity in ModelOps (i.e., the process of operating ML models, focusing on the management and administration of the lifecycle of AI and decision models (ML-based, learning graphs, rule-based, optimization agent, and linguistic models) to select the best model for production use. Typically, different algorithms / implementations can be compared over elapsed training time and / or up to a predetermined timeout using a performance metric (e.g., receiver operating characteristic (ROC) - area under the curve (AUC) score). Automated ML (the process of automating iterative tasks in ML model development) approaches can help establish those experiments within the ModelOps process. Current solutions for comparing parallelized ML algorithms run different ML algorithms that are either non-parallelized on a single node or on a general distributed infrastructure (e.g., pen-source software for reliable, scalable, distributed computing, open-source frameworks focused on interactive queries, ML, and real-time workloads): sequentially—where they train and test one algorithm at a time, then collect the results and compare performance at the end—resulting in wasted time for data scientists in ModelOps compared to one or more implementations that employ parallel training and testing.

[0031] Without differentiation for different hardware requirements, conventional technologies all run on the same infrastructure (e.g., open-source software for reliable, scalable, distributed computing or open-source frameworks, focusing on interactive queries, ML, and abstracting different algorithms / models with varying memory requirements, network bandwidth requirements, CPUs, GPUs, Tensor Processing Units (TPUs), TrueNorth, edge devices, etc., and real-time workloads). This leads to unfair (i.e., biased) comparisons of ML algorithm or model implementations because model training / prediction is performed on an abstract hardware generalization that is advantageous to some algorithms than others.

[0032] Furthermore, conventional systems can utilize resources for test runs of ML algorithms, which can be provided in a self-organizing manner. In a typical enterprise cloud, there exists a total amount (or budget) of CPUs, GPUs, memory, network latency, network bandwidth, and disks that can be allocated for specific ModelOps use cases. Users can provision different pre-built configurations to virtual machines (VMs) or configure random access memory (RAM), disks, virtual CPUs (vCPUs), GPUs, networks, etc., themselves. This leads to the nondeterministic polynomial-time hard (NP-hard) VM packaging problem.

[0033] Depending on the implementation, parallel ML algorithms can employ model parallelism, data parallelism, or a hybrid of both. Both the degree of data parallelism and model parallelism employed are generally negatively correlated with solution quality (i.e., the total out-of-sample prediction error at the model level given a limited training time budget), depending on numerous factors such as: batch size—too small or too large; the model and its hyperparameters; and the characteristics of the data itself (however, in conventional techniques, how data characteristics relate to solution quality at different degrees of parallelism is relatively unclear). Conventional techniques mention that for neural networks, increasing the batch size is a straightforward way to generate valuable speed increases within the workload, but the benefits diminish within the limitations of current hardware. This suggests that ML solution quality may often have some inverse correlation with the degree of distribution (e.g., the number of nodes and node heterogeneity). This may not be true, or even true for all ML algorithms, but in one embodiment, it is assumed that this is a general property, and that more nodes have a negative impact.

[0034] Another question is: is it important if resources are evenly distributed (e.g., uniform distribution across only a single type of VM style against high variability in VM configurations across nodes)? It may do more for some algorithms and less for others; moreover, efficient scheduling across heterogeneous nodes is a practical challenge; therefore, in one embodiment, it is assumed that there is generally a negative impact if the nodes are heterogeneous.

[0035] Conventional automated ML methods include automatically building ML pipelines and automatically selecting algorithms and hyperparameters, possibly with human intervention in the loop. Many traditional techniques focus specifically on deep learning (DL). Some non-DL conventional methods consider other algorithms but do not consider fairness in parallelization and resource allocation. Some typical approaches include: establishing a search space on algorithm / hyperparameter combinations; using an overall "time budget" for the overall model / algorithm search / optimization process; and evaluating individual models / algorithms in the search space with timeouts (or shortening of iterative training of the model). However, it appears that the current literature has not yet investigated the issue of "fair" comparisons of parallel ML algorithms relative to the "bias" in the underlying hardware configuration in one or more embodiments.

[0036] Figure 1 A block diagram of a process 200 for fairly comparing different models or algorithms executed in parallel, according to one embodiment, is shown. In some embodiments, the inputs to process 200 include: a set of parallel ML algorithms / models I330 for comparison. Figure 3 A set of available resource types T310 ( Figure 2 The resource pool includes the total available resource pool, the pre-defined resource bundles or "bundles" B320 (i.e., the combination of different resource types and corresponding resource quantities allowed when resources are provided), and the resource preference profile (RPP) for ML algorithms. i )210 (e.g., stored in storage devices, cloud storage, repositories, etc.) and parameters λ that respectively reflect the expected negative impact of resource distribution and resource changes on the performance of ML models / algorithms. dist and λ var .

[0037] Figure 2 An example of resource pool 300 is shown, which includes, according to one embodiment, resource type T310, resource set B320, different ML algorithms / models I330, and RPP. i 210. In one example embodiment, resource type T310 includes resource types such as CPU, RAM, network, disk, etc. In one example embodiment, resource bundle B320 includes resource bundles, each resource bundle including various resource types and the number of resources of each type. In one example embodiment, ML algorithm / model I 330 includes different ML algorithms / models that will be compared in parallel. i 210 includes relative resource utility parameters ( i ) i The relative resource utility parameter has values ​​(i.e., percentages) between 0 and 1 for each ML algorithm / model I 330, and includes all parameters for a specific ML algorithm / model. The sum of is always equal to 1 (i.e., 100%).

[0038] Figure 3 An exemplary RPPi 210 according to one embodiment is shown. In example RPPi 210, resource type / ML algorithm or model 405 includes various types of ML algorithms / models I 330 such as CNN distributed 440, CNN joint 450, kNN parallel 460, and SVM parallel 470. Other RPPs i s210 may include other types of ML algorithms / models I 330. In this example embodiment, RPP i The top row of 210 includes the following various utility percentages: CPU utility % 410, memory utility % 415, node network packet (NW) node bandwidth utility % 420, disk utility % 425, GPU utility % 430, and total % 435, with the total percentage for each row being 100%. The RPPi 210 utility percentages are based on the type of ML algorithm / model I 330, which inherently varies in resource type utility usage.

[0039] Back Figure 1 In one or more embodiments, the output of process 200 includes: resource allocation in the form of multiples of resource bundles B320 allocated to the respective parallel ML algorithms / models I 330 to be compared, and a metric for determining the relative “fairness” between the aforementioned resource allocations.

[0040] In some embodiments, process 200 uses RPP 210 of different ML algorithms created from empirical evidence and / or theoretical considerations, combined with mathematical optimization processes / procedures, to allow fair simultaneous comparison of ML models / algorithms I 330 when partitioning limited distributed computing resources across different (parallel) ML algorithms, thereby eliminating or mitigating biases arising from different ideal hardware requirements / preferences; and further considers the (negative) effects of higher distribution and resource heterogeneity in the above-described optimization model to further improve "fairness".

[0041] In one or more embodiments, the processing initialization of block 220 of process 200 includes resource pool 300 of resource type T310. Figure 2 ), a pre-set resource bundle B320 and a set of ML algorithms or models I 330 to be compared. In one embodiment, the processing of block 220 includes the following steps. Let T be a set of different resource types T310 in resource allocation pool 300 (e.g., CPU, GPU, ASIC, RAM, network bandwidth, etc.) and t = |T|. Each type k (1 ≤ k ≤ t) is c k Let B be the resource type (resource bundle B320, e.g., VM style) and m = |B|, for example, B = {B1, B2, ..., B}. mThe set of possible different discrete combinations of}, bound to B j (1≤j≤m) Consumption For a quantity of type k resources (1≤k≤t), let I have the following properties for resource i∈I. i RPP in form i 210, of which Indicates resource type (k to algorithm 1≤k≤t), where i represents the relative utility)

[0042] Figure 4 Equation 500 is shown according to one embodiment for use by ML algorithm (u i (S i According to its corresponding resource preference profile (S) i The relative utility parameter (i) in 210 gives resources Subjective utility of specific allocation RPP i In Equation 500, reference 510 represents the resource type, reference 520 represents the resource allocation (where the index of the resource allocation is relative to the worst allocation (0..1)), reference 530 represents the resource distribution (where the index of the resource distribution is relative to the best allocation (0..1)), and reference 540 represents the resource variance (where the index of the resource variance is relative to the best allocation (0..1)).

[0043] Back Figure 1 In one embodiment, in block 230, process 200 uses resource bundle B320 ( Figure 2 The multiplicity computation of S is used for the optimal allocation of resources in ML algorithms or models. In one embodiment, let S... i Allocate resources for the algorithm i∈I, i.e., multisets. Where parameters This indicates that some resources are bundled with B. j (1≤j≤m) are assigned to the ML algorithm or model i. The multiplicity of the allocation instances (which can be zero) is given by S, where S represents all multisets S i The set of u i (S i ) is a machine learning algorithm or model S i Based on the parameters in their respective i Given an ML algorithm or model RPP for equation 500 i Subjective utility of a specific allocation of 210

[0044]

[0045] In some embodiments, block 230 of process 200 searches for the parameter ∑ i∈I ui (S i multiset Maximize the set S (until the fair allocation of the difference of ∈), and (Relative to the total capacity of the resource pool). Note the parameter λ. dist and λ var It can be estimated or learned, and ranges from 0 (no effect) to 1 (maximum effect).

[0046] In one or more embodiments, resource allocation index 520 refers to how many resources are allocated to a particular ML algorithm or model compared to others. Resource allocation index 520 has a range starting from 0…1, with higher being better, 0 being the minimum, and 1 being the highest relative resource allocation.

[0047]

[0048]

[0049] In one or more embodiments, resource distribution index 530 refers to the number of resource grid instances B320 distributed relative to the minimum distributed workload of all resource grids. Figure 2 ) is the i-th workload. It should be noted that... Resource allocation S was specified i The multiplicity of resource bundle B320 in the index 530 ranges from 0…1 to 1, with lower values ​​being better, 0 being the minimum, and 1 being the maximum.

[0050]

[0051] In some embodiments, the resource variance index 540 refers to how much variation exists in the resource configuration of an instance of resource grid B320 allocated to a specific workload compared to the minimum variation across the set of all workloads. The resource variance index 540 ranges from 0…1-1, with lower being better, 0 being the minimum, and 1 being the maximum relative variance.

[0052]

[0053]

[0054] In some embodiments, process 200 ( Figure 1The various use cases for Process 200 are as follows: It can be used for products that compare ML algorithms or models. Process 200 can be used in products where a fairness metric relative to bias is used on a uniform hardware architecture when comparing different ML algorithms or models. Process 200 can also be used in products that benchmark ML algorithms and models. Process 200 can be used by AI (e.g., a “robot data scientist”) to automatically select the right ML technology to automatically design specific ML solutions by comparing different ML algorithms or models. Process 200 can also be used in products that automatically estimate the amount of resources required in an enterprise cloud to achieve a specific goal of desired learning performance (i.e., accuracy) for a given ML algorithm or model and the expected time frame used for training. Process 200 can also be used to transform the currently fully self-organized and manual process of selecting and supplying hardware resources / configurations for a specific enterprise ML solution into an automated, deterministic, and well-managed process.

[0055] Figure 5 A block diagram of a process 600 for determining the relative fairness between different resource allocations for different algorithms and / or models to be compared, according to one embodiment, is shown. In one embodiment, in block 610, process 600 is performed by a computing device (from...) Figure 6 Computing node 10, hardware and software layer 60, Figure 7 Processing system 900 Figure 8 System 1000 Figure 9 System 1100 Figure 10 (etc.) receive multiple machine learning algorithms for evaluation (e.g., Figure 2 The algorithm or model (I 330). In block 620, process 600 further provides the computing device receiving the total amount of resources to be allocated to multiple algorithms. In block 630, process 600 also provides the computing device allocating a fair share of the total resources (e.g., using...). Figure 2-3 RPP i 210) Assignment to each of the multiple algorithms. In block 640, process 600 further provides that each of the multiple algorithms is executed by the computing device using a fair share of the total amount of resources. In block 650, process 600 further provides that the computing device compares the performance of each of the multiple algorithms based on at least one of a plurality of hardware relative utility metrics describing the hardware relative utility of any given resource allocation for each of the multiple algorithms.

[0056] In one embodiment, process 600 may further include features of multiple algorithms, including machine learning algorithms, simulation algorithms, or search algorithms. The performance of each algorithm is compared based on execution time, accuracy, metrics, or a combination thereof.

[0057] In one embodiment, process 600 may further include the feature that the input for each of the plurality of algorithms includes a set of resource types (e.g., resource type T300, ...). Figure 2 ) and resource bundles that include the required hardware resources (e.g., resource bundle B320, Figure 2 ).

[0058] In one embodiment, process 600 may further include the following feature: the input to each of the plurality of algorithms also includes a resource preference profile (e.g., using RPP). i 210, Figures 2-3 This resource preference profile includes multiple hardware relative utility measures, eliminating or mitigating biases caused by different hardware requirements or preferences from multiple algorithms, and these multiple hardware relative utility measures are based on a resource allocation index (e.g., resource allocation index 520). Figure 4 ), resource distribution index (e.g., resource distribution index 530, Figure 4 ) and resource change index (e.g., resource change index 540, Figure 4 ).

[0059] In one embodiment, process 600 may further include the computing device calculating the optimal overall resource allocation as a Pareto optimal distribution, fair distribution, or loosely fair distribution of relative hardware utility across multiple algorithms based on the surplus of the hardware relative utility metric.

[0060] In one embodiment, process 600 may further include a feature of determining, by the computing device, the optimal allocation of hardware resources in a plurality of resource bundles to each of a plurality of algorithms.

[0061] It should be understood in advance that although this disclosure includes a detailed description of cloud computing, the implementation of the teachings described herein is not limited to a cloud computing environment. Rather, embodiments of the invention can be implemented in conjunction with any other type of computing environment now known or developed hereafter.

[0062] Cloud computing is a service delivery model that enables convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, network bandwidth, servers, processing, storage, applications, virtual machines (VMs), and services) that can be rapidly provisioned and released with minimal management effort or interaction with service providers. This cloud model may include at least five features, at least three service models, and at least four deployment models.

[0063] The characteristics are as follows:

[0064] On-demand self-service: Cloud consumers can automatically and unilaterally supply computing power, such as server time and network storage, as needed, without requiring human interaction with the service provider.

[0065] Wide Area Network (WAN) Access: Capabilities are available on the network and accessed through standard mechanisms that facilitate use by heterogeneous, thin or thick client platforms (e.g., mobile phones, laptops, and PDAs).

[0066] Resource pooling: A provider's computing resources are pooled to serve multiple consumers using a multi-tenant model, where different physical and virtual resources are dynamically allocated and reallocated based on demand. Location independence has significance because consumers typically do not control or know the exact location of the resources provided, but can specify the location at a higher level of abstraction (e.g., country, state, or data center).

[0067] Rapid and flexible: Capabilities can be provided quickly and flexibly, and in some cases, can be automatically and rapidly scaled outwards and rapidly released inwards. For consumers, the capacity available for provision often appears unlimited and can be purchased at any time and in any quantity.

[0068] Measurement services: Cloud systems automatically control and optimize resource usage by leveraging metering capabilities at a level of abstraction appropriate to the service type (e.g., storage, processing, bandwidth, and active consumer accounts). Resource usage can be monitored, controlled, and reported, providing transparency to both service providers and consumers.

[0069] The service model is as follows:

[0070] Software as a Service (SaaS): The capability offered to consumers is the ability to use the provider's applications running on cloud infrastructure. Applications can be accessed from various client devices via thin client interfaces, such as web browsers (e.g., web-based email). Consumers do not manage or control the underlying cloud infrastructure, including the network, servers, operating system, storage, or even individual application capabilities, with the possible exception of limited consumer-specific application configuration settings.

[0071] Platform as a Service (PaaS): This provides consumers with the ability to deploy consumer-created or acquired applications onto cloud infrastructure using programming languages ​​and tools supported by the provider. Consumers do not manage or control the underlying cloud infrastructure, including networks, servers, operating systems, or storage, but they have control over the deployed applications and the configuration of possible application hosting environments.

[0072] Infrastructure as a Service (IaaS): This provides consumers with the ability to provide processing, storage, networking, and other basic computing resources, whereby consumers can deploy and run arbitrary software, which may include operating systems and applications. Consumers do not manage or control the underlying cloud infrastructure, but have control over the operating system, storage, deployed applications, and possibly limited control over selected networking components (e.g., host firewalls).

[0073] The deployment model is as follows:

[0074] Private cloud: Cloud infrastructure operated solely by an organization. It can be managed by the organization or a third party and can exist inside or outside a building.

[0075] Community cloud: Cloud infrastructure shared by several organizations and supporting a specific community with shared concerns (e.g., tasks, security requirements, policies, and compliance considerations). It can be managed by an organization or a third party and can exist on-site or off-site.

[0076] Public cloud: Cloud infrastructure available to the general public or large industrial groups and owned by organizations that sell cloud services.

[0077] Hybrid cloud: A cloud infrastructure is a combination of two or more clouds (private, community, or public) that remain a single entity but are bound together by standardized or proprietary technologies that enable data and application portability (e.g., cloud bursts for load balancing between clouds).

[0078] Cloud computing environments are designed for services that are stateless, loosely coupled, modular, and semantically interoperable. At the heart of cloud computing is the infrastructure of a network of interconnected nodes.

[0079] Now for reference Figure 6 The diagram illustrates an illustrative cloud computing environment 50. As shown, the cloud computing environment 50 includes one or more cloud computing nodes 10 to which local computing devices used by cloud consumers can communicate, such as personal digital assistants (PDAs) or cellular phones 54A, desktop computers 54B, laptop computers 54C, and / or automotive computer systems 54N. The nodes 10 can communicate with each other. They can be physically or virtually grouped (not shown) in one or more networks, such as private clouds, community clouds, public clouds, or hybrid clouds, or combinations thereof, as described above. This allows the cloud computing environment 50 to provide infrastructure, platform, and / or software as a service, without requiring cloud consumers to maintain resources on their local computing devices. It should be understood that... Figure 6The types of computing devices 54A-N shown are for illustrative purposes only, and computing node 10 and cloud computing environment 50 can communicate with any type of computerized device on any type of network and / or network-addressable connection (e.g., using a web browser).

[0080] Now for reference Figure 7 This demonstrates a cloud computing environment of 50 ( Figure 6 This provides a set of functional abstractions. It should be understood beforehand that... Figure 7 The components, layers, and functions shown are for illustrative purposes only, and the embodiments are not limited thereto. As depicted, the following layers and corresponding functions are provided:

[0081] The hardware and software layer 60 includes hardware and software components. Examples of hardware components include: a host 61; a server 62 based on a RISC (Reduced Instruction Set Computer) architecture; a server 63; a blade server 64; a storage device 65; and a network and network components 66. In some embodiments, software components include network application server software 67 and database software 68.

[0082] The virtualization layer 70 provides an abstraction layer from which the following examples of virtual entities can be provided: virtual server 71; virtual storage 72; virtual network 73, including virtual private network; virtual application and operating system 74; and virtual client 75.

[0083] In one example, management layer 80 can provide the following functionalities: Resource Provisioning 81 provides dynamic procurement of computing resources and other resources used to perform tasks within the cloud computing environment. Metering and Pricing 82 provides cost tracking when utilizing resources in the cloud computing environment and billing or invoicing for consuming these resources. In one example, these resources may include application software licenses. Security provides authentication for cloud consumers and tasks, as well as protection for data and other resources. User Portal 83 provides access to the cloud computing environment for consumers and system administrators. Service Level Management 84 provides cloud resource allocation and management to ensure that required service levels are met. Service Level Agreement (SLA) Planning and Fulfillment 85 provides pre-scheduling and procurement of cloud resources, where future needs are anticipated according to the SLA.

[0084] Workload tier 90 provides examples of functionalities that can be leveraged in a cloud computing environment. Examples of workloads and functionalities that can be provided from this tier include: graphing and navigation 91; software development and lifecycle management 92; virtual classroom education delivery 93; data analytics processing 94; transaction processing 95; and processing 96 for measuring the relative fairness between different resource allocations of different ML algorithms and / or models for comparison (see, for example, process 600). Figure 5 System 900 Figure 8System 1000 Figure 9 System 1100 Figure 10 (etc.). As mentioned above, regarding Figure 7 All the foregoing examples described are illustrative only, and the embodiments are not limited to these examples.

[0085] To reiterate, although this disclosure includes a detailed description of cloud computing, the implementation of the teachings set forth herein is not limited to a cloud computing environment. Rather, embodiments can be implemented using any type of cluster computing environment now known or developed hereafter.

[0086] Figure 8 This is a network architecture of system 900, according to an embodiment, for determining the relative fairness between different resource allocations for different ML algorithms and / or models to be compared. (e.g.) Figure 8 As shown, multiple remote networks 902 are provided, including a first remote network 904 and a second remote network 906. A gateway 901 may be coupled between the remote network 902 and a neighboring network 908. In the context of this network architecture 900, networks 904 and 906 may each take any form, including but not limited to LANs, WANs such as the Internet, public switched telephone networks (PSTN), internal telephone networks, etc.

[0087] In use, gateway 901 serves as the entry point from remote network 902 to neighboring network 908. Thus, gateway 901 can function as a router capable of routing a given data packet to gateway 901, and as a switch providing the actual path for a given packet to and from gateway 901.

[0088] The system further includes at least one data server 914 coupled to a neighboring network 908, accessible from a remote network 902 via a gateway 901. It should be noted that the data server 914 may include any type of computing device / component. Multiple user devices 916 are coupled to each data server 914. Such user devices 916 may include desktop computers, laptop computers, handheld computers, printers, and / or any other type of logically-containing device. It should be noted that in some embodiments, the user device 916 may also be directly coupled to any other device in the network.

[0089] Peripheral device 920, or a series of peripheral devices 920, such as fax machines, printers, scanners, hard drives, networking and / or local storage units or systems, may be coupled to one or more of networks 904, 906, and 908. It should be noted that databases and / or additional components may be used with or integrated into any type of network element coupled to networks 904, 906, and 908. In the context of this specification, a network element may refer to any component of a network.

[0090] According to some methods, the methods and systems described herein can be implemented using and / or on virtual systems and / or systems that emulate one or more other systems, such as simulation systems. / OS environment System, virtual hosting Environment System, simulation / OS environment Systems, etc. In some embodiments, such virtualization and / or emulation can be achieved by using... It is implemented using software.

[0091] Figure 9 An illustration is shown with respect to one embodiment. Figure 8 The representative hardware system 1000 environment associated with user equipment 916 and / or server 914. In one example, the hardware configuration includes a workstation with a central processing unit 1010 such as a microprocessor, and a plurality of other units interconnected via a system bus 1012. Figure 9 The workstation shown may include random access memory (RAM) 1014, read-only memory (ROM) 416, I / O adapter 1018 for connecting peripheral devices such as disk storage unit 1020 to bus 1012, user interface adapter 1022 for connecting keyboard 1024, mouse 1026, speaker 1028, microphone 1032 and / or other user interface devices such as touch screen, digital camera (not shown) to bus 1012, communication adapter 1034 for connecting the workstation to communication network 1035 (e.g., data processing network), and display adapter 1036 for connecting bus 1012 to display device 1038.

[0092] In one example, a workstation may have an operating system residing on it, such as... Operating System (OS) OS, etc. In one embodiment, system 1000 adopts a system based on... The file system. It should be understood that, in addition to those mentioned, other examples can also be implemented on different platforms and operating systems. Such other examples could include using... Operating systems written in XML, C and / or C++ or other programming languages, and object-oriented programming methods. Object-oriented programming (OOP), which is increasingly used for developing complex applications, can also be used.

[0093] Figure 10This is a block diagram illustrating a distributed system 1100 for determining the relative fairness between different resource allocations for different ML algorithms and / or models to be compared, according to one embodiment. In one embodiment, system 1100 includes client devices 1110 (e.g., mobile devices, smart devices, computing systems, etc.), a cloud or resource-sharing environment 1120 (e.g., public cloud computing environment, private cloud computing environment, data center, etc.), and a server 1130. In one embodiment, cloud services are provided from server 1130 to client devices 1110 via cloud or resource-sharing environment 1120.

[0094] One or more embodiments may be systems, methods, and / or computer program products at any possible level of technical detail integration. A computer program product may include a computer-readable storage medium (or media) having computer-readable program instructions thereon for causing a processor to perform aspects of this embodiment.

[0095] A computer-readable storage medium can be a tangible device capable of retaining and storing instructions used by an instruction execution device. A computer-readable storage medium can be, for example, but is not limited to, electronic storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of computer-readable storage media includes: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable optical disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices such as punch cards or recessed structures with instructions recorded thereon, and any suitable combination of the above computer-readable storage media. As used herein, a computer-readable storage medium should not be construed as a transient signal itself, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0096] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to a suitable computing / processing device, or via a network, such as the Internet, a local area network (LAN), a wide area network (WAN), and / or a wireless network, to an external computer or external storage device. The network may include copper cables, optical fibers, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to a computer-readable storage medium within the respective computing / processing device.

[0097] Computer-readable program instructions used to perform the operations of the embodiments may be assembly instructions, instruction set architecture (ISA) instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, integrated circuit configuration data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and procedural programming languages ​​such as the "C" programming language or similar programming languages. The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter case, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry including, for example, programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs) may execute the computer-readable program instructions by utilizing state information from the computer-readable program instructions to personalize the electronic circuitry in order to perform aspects of this embodiment.

[0098] This document describes aspects of embodiments with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products. It will be understood that each block of the flowcharts and / or block diagrams, and combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer-readable program instructions.

[0099] These computer-readable program instructions may be provided to a processor of a computer or other programmable data processing apparatus to produce a machine, such that the instructions, which are executed via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0100] These computer-readable program instructions may also be stored in a computer-readable storage medium that is capable of directing a computer, a programmable data processing device, and / or other equipment to operate in a particular manner, such that the computer-readable storage medium in which the instructions are stored includes an article of writing that includes instructions for implementing aspects of functions / actions specified in one or more blocks of a flowchart and / or block diagram.

[0101] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer-implemented process, such that the instructions, which execute on the computer, other programmable apparatus or other device, perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0102] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of instructions comprising one or more executable instructions for implementing a specified logical function. In some alternative embodiments, the functions indicated in the blocks may occur in a different order than indicated in the figures. For example, two blocks shown consecutively may actually be implemented as a single step, executed simultaneously, substantially simultaneously, with partial or complete time overlap, or these blocks may sometimes be executed in reverse order, depending on the functions involved. It will also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented by a dedicated hardware-based system that performs the specified function or action or executes a combination of dedicated hardware and computer instructions.

[0103] References to singular elements in the claims do not mean "one and only one," but rather "one or more," unless expressly stated otherwise. All structural and functional equivalents of the elements of the above exemplary embodiments, now known or hereafter known to those skilled in the art, are intended to be included in these claims. Elements of the claims herein should not be construed under paragraph 6 of Section 112 of 35 U.S. Section 6 unless such element is expressly stated using the phrase "for a method of...".

[0104] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the embodiments described. As used herein, the singular forms “a,” “an,” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that the terms “comprising” and / or “including” as used in this specification specify the presence of the stated features, integers, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0105] All the means or steps plus functional elements in the following claims are intended to include any structure, material, action, and equivalent for performing a function in combination with other claimed elements as specifically claimed. Descriptions of embodiments of the invention have been presented for purposes of illustration and description, but are not intended to be exhaustive or limited to the embodiments disclosed. Many modifications and variations will be apparent to those skilled in the art without departing from the scope of the embodiments. The embodiments were chosen and described in order to best explain the principles and practical application of the embodiments and to enable those skilled in the art to understand that embodiments with various modifications are suitable for the particular intended use.

Claims

1. A method for comparing the performance of multiple algorithms using a computing device, the method comprising: The computing device receives multiple algorithms for evaluation, and the input of each of the multiple algorithms includes a resource preference profile; The computing device receives the total amount of resources to be allocated to the plurality of algorithms; The computing device allocates a fair share of the total resources to each of the plurality of algorithms according to the resource preference profile; Each of the plurality of algorithms is executed by the computing device using the fair share allocated to the total amount of resources; as well as The computing device compares the performance of each of the plurality of algorithms based on at least one of a plurality of hardware relative utility metrics, which describe the hardware resource utility for any given resource allocation for each of the plurality of algorithms. Wherein: the resource preference profile includes the multiple hardware relative utility metrics; biases caused by different hardware requirements or preferences of multiple algorithms are eliminated or mitigated; and The aforementioned hardware relative utility metrics are based on the resource allocation index, resource distribution index, and resource variance index. The algorithms mentioned above include machine learning algorithms, simulation algorithms, or search algorithms.

2. The method of claim 1, wherein the performance of each of the plurality of algorithms is compared in terms of execution time, accuracy, metrics, or a combination thereof.

3. The method of claim 1, wherein the input of each of the plurality of algorithms further includes a set of resource types and a resource bundle including the required hardware resources.

4. The method according to claim 1, further comprising: The computing device calculates the optimal overall resource allocation as a Pareto optimal distribution, fair distribution, or loosely fair distribution of hardware resource utility across the multiple algorithms based on the multiple hardware relative utility metrics.

5. The method according to claim 1, further comprising: The computing device determines the optimal allocation of hardware resources from multiple resource bundles to each of the multiple algorithms.

6. A computer program product for comparing the performance of multiple algorithms, the computer program product comprising a computer-readable storage medium having program instructions embodied therein, the program instructions being executable by a processor to cause the processor to: The processor receives multiple algorithms for evaluation, each of which receives a resource preference profile as input. The processor receives the total amount of resources to be allocated to the plurality of algorithms; The processor allocates a fair share of the total resources to each of the plurality of algorithms according to the resource preference profile; The processor uses the allocated fair share of the total resources to execute each of the plurality of algorithms; as well as The processor compares the performance of each of the plurality of algorithms based on at least one of a plurality of hardware relative utility metrics, which describe the hardware resource utility for any given resource allocation for each of the plurality of algorithms. Wherein: the resource preference profile includes the plurality of hardware relative utility measures; biases caused by different hardware requirements or preferences of the plurality of algorithms are eliminated or mitigated; and the plurality of hardware relative utility measures are based on resource allocation index, resource distribution index, and resource variance index. The algorithms mentioned above include machine learning algorithms, simulation algorithms, or search algorithms.

7. The computer program product of claim 6, wherein the performance of each machine learning algorithm in the machine learning algorithms is compared in terms of execution time, accuracy, metric, or a combination thereof.

8. The computer program product of claim 6, wherein the input of each of the plurality of algorithms further includes a set of resource types and a resource bundle containing the required hardware resources.

9. The computer program product of claim 6, wherein the program instructions executable by the processor further cause the processor to: The processor calculates the optimal overall resource allocation as a Pareto optimal distribution, fair distribution, or loosely fair distribution of hardware resource utility across the multiple algorithms based on the multiple hardware relative utility metrics.

10. The computer program product of claim 6, wherein the program instructions executable by the processor further cause the processor to: The processor determines the optimal allocation of hardware resources from multiple resource bundles to each of the multiple algorithms.

11. An apparatus comprising: The memory is configured to store instructions; as well as The processor is configured to execute the instructions to: Multiple algorithms are received for evaluation, and the input of each of the multiple algorithms includes a resource preference profile; Receive the total amount of resources to be allocated to the plurality of algorithms; The total amount of resources is allocated fairly to each of the plurality of algorithms according to the resource preference profile; Each of the plurality of algorithms is executed using the allocated fair share of the total resources; and The performance of each of the plurality of algorithms is compared based on at least one of a plurality of hardware relative utility metrics, which describe the hardware resource utility of each of the plurality of algorithms for any given resource allocation. Wherein: the resource preference profile includes multiple hardware relative utility measures; biases caused by different hardware requirements or preferences of the multiple algorithms are eliminated or mitigated; and the multiple hardware relative utility measures are based on resource allocation index, resource distribution index, and resource variance index. The algorithms mentioned above include machine learning algorithms, simulation algorithms, or search algorithms.

12. The apparatus of claim 11, wherein the performance of each algorithm in the algorithms is compared in terms of execution time, accuracy, metric, or a combination thereof.

13. The apparatus of claim 11, wherein the input to each of the plurality of algorithms further includes a set of resource types and a resource bundle including the required hardware resources.

14. The apparatus of claim 11, wherein the instructions executable by the processor further cause the processor to: Based on the multiple hardware relative utility metrics, the optimal overall resource allocation is calculated as a Pareto optimal distribution, a fair, or a loosely fair distribution of hardware resource utility across the multiple algorithms; and Determine the optimal allocation of hardware resources to each of the plurality of algorithms in a plurality of resource bundles.

Images