Energy-efficient machine learning models

Abstract

The present invention relates to energy-efficient machine learning (ML) models, and more particularly to energy-efficient ML model generation for energy-efficient inference. According to one aspect of the present invention, a computer-implemented method for operating an ML training system to train an ML model is provided. The method includes receiving an energy consumption (EC) target threshold for training, training the ML model based on a training dataset for an epoch, estimating EC of the ML training system while training the ML model over the epoch, determining whether the cumulative EC estimate is less than the EC target threshold, updating the training dataset for a new epoch in response to determining that the cumulative EC estimate is less than the EC target threshold, and repeating the training, estimation, and generation steps using the updated training dataset with the new epoch until an ML model is effectively trained. In response to determining that the cumulative EC meets or exceeds the EC target threshold, terminating training of the ML model, and outputting the effectively trained ML model and associated EC metrics. Further aspects of the invention include a computer-implemented method of operating an ML training / inference system to perform EC benchmarking of trained ML models for inference, and an ML training and inference system for performing the method.

Description

[Technical Field]

[0001] The present invention relates to energy-efficient machine learning (ML) models, and in particular to energy-efficient ML model generation for energy-efficient inference. [Background technology]

[0002] Current state-of-the-art deep learning methods require increasingly large amounts of training data and long training cycles to develop more complex AI or ML models for inference. In 2018, global emissions from all human sources were equivalent to 51.8 billion tons of carbon dioxide (CO2). CO2 is the primary greenhouse gas heating the Earth, responsible for 75% of global warming. As AI or ML models become increasingly complex, they are also becoming increasingly energy-intensive to train and use for inference.

[0003] Net-zero can be defined as achieving a state in which activities have no net impact on the climate from greenhouse gas emissions. This presents a conundrum in terms of how complex AI / ML solutions can be delivered while enabling countries and / or organizations to meet their "net-zero" commitments. This challenge is exacerbated as the size of state-of-the-art AI / ML models increases, which also exponentially increases the energy demands of these models. At the same time, end users or customers are demanding increasingly complex solutions against the backdrop of countries, companies, and organizations striving to achieve "net-zero" goals. There is a desire to develop sustainable AI / ML frameworks / methods that can assist in achieving "net-zero" commitments. Summary of the Invention

[0004] According to a first aspect of the present invention, there is provided a computer-implemented method of operating a machine learning (ML) training system for training an ML model, the method comprising: receiving an energy expenditure (EC) target threshold for training; training the ML model based on a training dataset for an epoch; estimating EC of the ML training system while training the ML model over the epoch; determining whether the cumulative EC estimate is less than the EC target threshold; in response to determining that the cumulative EC estimate is less than the EC target threshold, updating the training dataset for a new epoch and repeating the training, estimation, and generation steps using the updated training dataset with the new epoch until an ML model is effectively trained; in response to determining that the cumulative EC meets or exceeds the EC target threshold, terminating training of the ML model; and outputting the effectively trained ML model and associated EC metrics.

[0005] The method of the first aspect may further comprise: estimating the EC of the ML training system further comprising receiving real-time sensor and performance data from hardware of the ML training system during training of the ML model over the epoch; and estimating the EC of the ML training system based on the real-time sensor and performance data received over the epoch.

[0006] The hardware of the ML training system may comprise one or more central processing units (CPUs), one or more graphical processing units (GPUs), one or more tensor processing units (TPUs), one or more memory or storage units, one or more communication or network resources, and / or any other group of hardware of the ML training system associated with training the ML model and influencing the EC of the ML training system.

[0007] The real-time sensor and performance data may comprise one or more of: electrical usage of hardware of an ML training system during training of an ML model; temperature sensor data associated with hardware used during training of an ML model; clock cycles associated with one or more CPUs, GPUs, and / or TPUs of the ML training system; power usage data associated with one or more CPUs, GPUs, and / or TPUs and / or memory units of the ML training system; and the number of floating-point operations (FLOPS) performed by one or more CPUs, GPUs, and / or TPUs and / or memory units associated with the ML training system when training the ML model.

[0008] The method of the first aspect may further comprise: estimating the EC of the ML training system further comprising invoking an EC routine configured to monitor the performance of the EC of a corresponding portion of the hardware ML training system.

[0009] The method of the first aspect may further comprise generating a cumulative EC estimate based on the estimated EC of the current epoch and any previous epochs.

[0010] EC estimates can be measured in terms of carbon or CO2 emissions.

[0011] The EC estimate may be an EC estimate for each training sample.

[0012] The method of the first aspect may further comprise determining whether the ML model has been effectively trained based on whether the average accuracy / loss of the ML model output achieves a required accuracy / loss threshold, and terminating training of the ML model in response to determining that the ML model has been effectively trained.

[0013] The method of the first aspect may further comprise terminating training of the ML model in response to determining that the accumulated EC meets or exceeds the EC target threshold, further comprising determining whether the ML model is considered to be effectively trained based on whether the average accuracy / loss of the ML model (108a) output achieves a required accuracy / loss threshold.

[0014] The method of the first aspect may further comprise terminating training of the ML model in response to determining that the accuracy / loss of the ML model diverges from a required accuracy / loss threshold over multiple consecutive epochs.

[0015] The method of the first aspect may further comprise terminating training of the ML model in response to determining that the convergence rate of the accuracy / loss of the ML model towards the required accuracy / loss for the current epoch is below a convergence rate threshold for multiple consecutive epochs.

[0016] The method of the first aspect may further comprise performing EC benchmarking of each of the effectively trained ML models for inference based on the steps of: for each effectively trained ML model (108a), receiving an EC target threshold for inference using the each trained ML model; receiving a batch of input data for input to each trained ML model for inference; performing ML model inference using the trained ML model on the batch of input data; estimating an EC per inference of the trained ML model when performing ML model inference on the batch of input data; determining whether the per-inference EC estimate is greater than the EC target threshold for inference; in response to determining that the per-inference EC estimate is less than or equal to the EC target threshold for inference, outputting an EC ML model including the trained ML model and data representing the per-inference EC estimate; in response to determining that the per-inference EC estimate is greater than the EC target threshold for inference, performing ML model optimization on the trained ML model to reduce the per-inference EC estimate; and iterating EC benchmarking of the resulting optimized ML models.

[0017] The method of the first aspect may further comprise: estimating the EC per inference comprises estimating a total EC for processing a batch of input data; and estimating an average EC per inference based on dividing the total EC by a number of input data samples in the batch of input samples.

[0018] The method of the first aspect may further comprise: performing optimization of the trained ML model further comprising modifying the trained ML model based on at least one of a quantization technique and a weight pruning technique to form an optimized ML model; and retraining the optimized ML model to meet the EC target threshold for training or when the optimized ML model has been effectively trained.

[0019] According to a second aspect of the present invention, there is provided a computer-implemented method of operating an ML training / inference system to perform EC benchmarking of trained ML models for inference, the method comprising: receiving an EC target threshold for inference with each of the trained ML models; receiving a batch of input data for input to each of the trained ML models for inference; performing ML model inference using the trained ML models on the batch of input data; estimating an EC per inference of the trained ML model when performing ML model inference on the batch of input data; determining whether the per-inference EC estimate is greater than the EC target threshold for inference; in response to determining that the per-inference EC estimate is less than or equal to the EC target threshold for inference, outputting an EC ML model including data representing the trained ML model and the per-inference EC estimate; in response to determining that the per-inference EC estimate is greater than the EC target threshold for inference, performing ML model optimization on the trained ML models to reduce the per-inference EC estimate; and iterating EC benchmarking of the resulting optimized ML models.

[0020] The method of the second aspect may further comprise: estimating the EC per inference comprises estimating a total EC for processing a batch of input data; and estimating an average EC per inference based on dividing the total EC by the number of input data samples in the batch of input samples.

[0021] The method of the second aspect may further comprise: performing optimization of the trained ML model further comprising modifying the trained ML model based on at least one of a quantization technique and a weight pruning technique to form an optimized ML model; and retraining the optimized ML model to meet the EC target threshold for training or when the optimized ML model has been effectively trained.

[0022] The method of the second aspect may further comprise the trained ML model being trained or retrained based on receiving an EC target threshold for training; training the ML model based on a training dataset for an epoch; estimating EC of an ML training system while training the ML model over the epoch; determining whether the cumulative EC estimate is less than the EC target threshold; updating the training dataset for a new epoch in response to determining that the cumulative EC estimate is less than the EC target threshold and repeating the training, estimation, and generation steps using the updated training dataset with the new epoch until an ML model is validly trained; terminating training of the ML model in response to determining that the cumulative EC meets or exceeds the EC target threshold; and outputting the validly trained ML model and associated EC metrics.

[0023] According to a third aspect of the present invention there is provided a computer program product comprising computer readable instructions which, when executed by a computer, cause the computer to perform a method according to any one or more of the computer-implemented methods of the first and / or second aspects and / or methods described herein.

[0024] According to a fourth aspect of the present invention there is provided an apparatus comprising one or more processors and a memory, the memory storing computer readable instructions which, when executed by the one or more processors, cause the apparatus to perform a method according to the computer-implemented method of the first and / or second aspects and / or any one or more of the methods described herein.

[0025] According to a fifth aspect of the present invention, there is provided a system comprising one or more processors and a memory, the memory storing computer readable instructions which, when executed by the one or more processors, cause the apparatus to perform a method according to the computer-implemented method of the first and / or second aspect and / or any one or more of the methods described herein.

[0026] According to a sixth aspect of the present invention, there is provided a machine learning training and inference system comprising: an ML training unit configured to perform the computer-implemented method according to the first aspect for training one or more ML models; an ML inference and optimization unit configured to perform the computer-implemented method according to the second aspect for benchmarking one or more effectively trained ML models; and an energy consumption training and optimization control unit configured to estimate an EC of the ML training unit that trains each ML model of the one or more ML models, estimate an EC of the ML inference and optimization unit when benchmarking each effectively trained ML model, and control the ML training unit when training the ML models or the ML inference and optimization unit when performing ML model inference based on the estimated EC.

[0027] A machine learning training and inference system, wherein the energy consumption training and optimization control unit may be configured to select effectively trained and / or optimized ML models that satisfy corresponding EC target thresholds for training and EC target thresholds for inference, and to store the selected effectively trained and / or optimized ML models together with their corresponding EC target thresholds for training and EC target thresholds for inference for use by one or more users for inference in an energy-efficient manner, and wherein each of the one or more users may perform carbon offsetting based on the corresponding EC target thresholds for training and EC target thresholds for inference of the effectively trained or optimized ML models used by each of the one or more users.

[0028] Embodiments of the present invention will now be described, by way of example only, with reference to the following figures. [Brief explanation of the drawings]

[0029] [Figure 1a]Schematic of a training / inference system for generating energy-efficient ML models that meet a required energy consumption (EC) target threshold. [Figure 1b] Schematic of the training / inference framework for the training / inference system in Figure 1a. [Figure 1c] FIG. 1b is a schematic diagram of various exemplary hardware components used by the training / inference system of FIG. 1a. [Figure 2a] FIG. 1 illustrates an exemplary method for training an ML model to meet an EC target threshold. [Figure 2b] FIG. 1 illustrates an example method for benchmarking a trained ML model for inference and / or optimization of the trained ML model. [Figure 3] FIG. 1 illustrates another exemplary method and training / inference framework for training an ML model and / or benchmarking a trained ML model to meet an EC target threshold. [Figure 4] Schematic representation of a table of test results for training an active learning baseline ML model using the method / framework in Fig. 1a through Fig. 3. [Figure 5] Schematic showing test results for training a transfer learning model using the method / framework in Figs. 1a to 3. [Figure 6] Schematic diagram of a computing system. [Figure 7] 1 is a schematic diagram of a computer-readable medium. DETAILED DESCRIPTION OF THE INVENTION

[0030] This specification describes systems, apparatus, and methods for quantitatively measuring energy efficiency (or energy consumption) as a metric of ML model performance for use in training / inference frameworks over the ML model lifecycle, and for optimizing the training and deployment of complex ML models in an energy-efficient manner. The training / inference frameworks can be used to develop energy-efficient standards for benchmarking and / or evaluating various ML algorithms / techniques for developing and training ML models based on the energy consumption of the hardware used to implement the training and / or inference of the ML models. The systems, apparatus, and methods described herein can be used and applied in the pursuit of sustainable AI / ML training / inference systems.

[0031] The framework may use configuration of one or more ML model application programming interfaces (APIs) to measure the energy consumption of hardware (e.g., central processing units (CPUs), graphical processing units (GPUs), tensor processing units (TPUs), and / or memory units, etc.) used during training / inference of one or more ML models. Energy consumption may be based on several quantitative energy efficiency metrics (e.g., carbon cost, FLOPS, model size, time, power, etc.). Energy consumption may be measured and / or calculated in real time from sensors and / or performance parameters associated with the hardware during runtime training / inference of the ML models. This may be further used to evaluate the measured energy consumption and output of one or more ML models during both the training and inference processes. Energy consumption efficiency metrics may be captured at key points during training and / or inference (e.g., per batch / per epoch / per K-fold) and used to determine whether to continue training, optimize the resulting ML models, and / or infer the ML models in an energy-efficient manner.

[0032] Machine learning (ML) algorithms are computational methods that can be used to devise complex ML models that can help recognize patterns from input training datasets, classify data, and / or solve complex problems, and once trained, can be used to infer previously unseen input data for inference, prediction, and analysis of patterns / classifications / complex processes, etc. ML models can learn historical relationships and trends contained in associated input training datasets and are configured to model one or more aspects or portions of complex processes and / or patterns within the input training data. ML algorithms may comprise or represent one or more or combinations of computational methods, which can be used to generate analytical ML models. Such models are suitable for solving complex problems and / or recognizing patterns of interest within training datasets, including, by way of example only, but not limited to, predicting and analyzing complex processes and / or recognizing various patterns when provided with new input data not previously seen by the ML model. ML algorithms may iteratively train ML models based on processing training samples from a training dataset. Multiple passes through the training dataset may be performed, with each pass through the training dataset being referred to as an epoch. An epoch is when all training data samples in the input training dataset are processed, and is defined as the total number of iterations of all training data in one cycle to train an ML model.

[0033] Examples of ML algorithms that may be used herein may include, by way of example only, and not by way of limitation, any ML algorithm that can be trained on labeled and / or unlabeled training datasets to generate an ML model associated with the labeled and / or unlabeled training datasets, or may be based on one or more supervised ML algorithms, semi-supervised ML algorithms, unsupervised ML algorithms, linear and / or non-linear ML algorithms, ML algorithms associated with classification, ML algorithms associated with regression, etc., and / or combinations thereof. Some examples of ML algorithms may include or may be based on one or more of, by way of example only and not by way of limitation, active learning, multi-task learning, transfer learning, neural message parsing, one-shot learning, dimensionality reduction, decision tree learning, association rule learning, similarity learning, data mining algorithms / methods, artificial neural networks (NNs), deep NNs, deep learning, deep learning ANNs, inductive logic programming, support vector machines (SVMs), sparse dictionary learning, clustering, Bayesian networks, reinforcement learning, representation learning, similarity and metric learning, sparse dictionary learning, genetic algorithms, rule-based machine learning, learning classifier systems, and / or one or more combinations thereof, etc.

[0034] While ML model performance can be evaluated based on loss, accuracy, confusion matrix, precision, and recall, energy efficiency is typically not associated with the performance characteristics of ML models. Various methods for measuring the energy efficiency (i.e., energy consumption (EC)) of an ML model may include one or more of the following: the amount of electricity consumed to train or run the ML model for inference; carbon emissions, such as the amount of CO2, emitted while training or running the ML model for inference; floating-point operations (FLOPS), which provide an estimate for the amount of work performed by a computational process, such as training or running an ML model for inference; the cost of fetching data from memory while training or running the ML model for inference; the number of ML parameters and / or hyperparameters used by the ML model, which may be correlated to the amount of work; and the total elapsed real-time execution time for training or running the ML model. The methods / frameworks, apparatus, and systems described herein provide energy-efficient methods / frameworks for EC ML model training and inference that can be applied to develop sustainable AI / ML models to achieve "net-zero" commitments of users, companies, countries, etc.

[0035] 1a shows a schematic diagram of a training / inference system 100 for training and optimizing ML models for inference in an energy-efficient manner. The training / inference system 100 may be based on a training / inference framework for use in developing sustainable ML models. The training / inference system 100 includes a user interface 102, an ML model ingestion unit 104, an EC training, optimization, and selection controller (TOSC) 106, an EC ML model training unit 108, an EC ML model optimization unit 110, and an ML model output store and / or unit 112. The ML model ingestion unit 104, the EC ML model training unit 108, the EC ML model optimization unit 110, and the ML model output unit 112 are connected via the EC TOSC unit 106.

[0036] The model ingestion unit 104 receives ML model data 104a-104m from one or more users via the user interface 102, the ML model data including necessary ML model information needed for the EC ML model training unit 108 to train one or more ML models 108a-108m using an ML model algorithm. The ML model information may include, by way of example only and not limitation, computer-implemented instructions for generating the ML model, ML model hyperparameters, the type of ML model algorithm / structure, an ML training dataset for use in training the ML model, and / or other ML model parameters. Each of the ML model data 104a-104m may further include an EC target threshold, which may be set by a user to indicate a maximum EC that may be used by the training / inference system 100 to train an ML model based on the ML model data; this EC target threshold may also be used by the training / inference system 100 for optimization or ML model inference in connection with the resulting trained ML model. The ML model data 104a may further include an additional accuracy / loss target to indicate when the corresponding trained ML model 108a has been effectively trained, and the accuracy / loss target may include data representing an output error loss / loss or accuracy threshold that the ML model 108a output should achieve during training.

[0037] Alternatively or additionally, the ML model data may further include data representing an EC training target threshold (or an EC target threshold for training) and an EC optimization / inference target threshold (or an EC target threshold for inference), which are used by the EC TOSC unit 106 to respectively control: a) training the ML model 108a based on the ML model data 104a via the EC ML model training unit 108; and b) optimizing / inferring the trained ML model 108a via the EC ML model optimization / inference unit 110, resulting in an optimized trained ML model 110a. The EC target threshold for training may be different from or the same as the EC target threshold for inference. If the training of the ML model 108a and the optimization / inference of the trained ML model 110a satisfy the EC target thresholds (or the corresponding EC training target threshold and EC optimization / inference target threshold), the ML model 110a may be selected for output to the ML model output unit 112 for storage and / or output to a corresponding user, etc. The multiple ML EC models 112a-112l may be stored by the ML model output unit 112 for use in inference in an energy-efficient manner, and a user is guaranteed that the resulting EC ML model will meet a given EC target threshold during inference and a given EC target threshold during training. Thus, a user can select an appropriate EC ML model from the stored EC ML models 112a-112l based on an EC training target threshold and / or an EC optimization / inference target threshold that meets the user's requirements and / or promotes sustainable AI. Similarly, the user can then offset the user's EC for the ML model using one or more carbon offset programs, etc., thereby meeting "net-zero" carbon emissions for the selected EC ML model 112a being used for inference.

[0038] In this example, the EC TOSC unit 106 may receive each of the ML model data 104a-104m from each user and is configured to control the training and / or optimization / inference of the resulting corresponding ML model 108a-108m. For each received ML model data 104a, the EC TOSC unit 106 is configured to monitor the EC of the hardware used by the training / inference system 100 during training of the corresponding ML model 108a. The hardware used by the training / inference system 100 may include one or more CPUs, one or more GPUs, one or more TPUs, and / or memory units, from which the EC may be measured and / or calculated based on real-time sensor data and / or real-time hardware parameters / data associated with the CPUs, GPUs, TPUs, and / or memory or storage units (e.g., volatile or non-volatile memory), and / or communication resources (e.g., energy consumption associated with data transferred over a network, etc.) used during training / inference. The EC TOSC unit 106 may use the measured or calculated EC corresponding to the training of the ML model 104a to control the training of the ML model 104a. For example, the EC TOSC unit 106 may estimate the EC for training the ML model 104 over epochs of the training dataset and then compare the cumulative EC used so far to train the ML model 104 with the corresponding EC target threshold (or EC training target threshold) for that ML model 108a. From the comparison, the EC TOSC unit decides whether to continue training the ML model 108a or whether to terminate and / or stop training and provides corresponding instructions to the EC ML model training unit 108.

[0039] For example, the EC TOSC 106 may determine whether to continue training the ML model 108a and / or determine whether the ML model 108a is effectively trained (e.g., achieves a minimum error loss or meets an accuracy target) based on the accumulated EC of the ML model 108a after each epoch during training. If the accumulated EC is sufficiently below an EC target threshold (e.g., a maximum EC specified by a user for training the ML model 108a), the EC TOSC unit 106 may allow further training of the ML model 108a over another epoch of the training dataset to proceed. Otherwise, the EC TOSC unit 106 may instruct the EC ML model training unit 108 to terminate training of the ML model 108a. Upon terminating training of the ML model 108a, the EC TOSC may also request the EC ML model training unit 108 to check whether the ML model 108a has been effectively trained. The ML model 108a may be effectively trained when the output error loss / loss or accuracy of the ML model 108a output meets a specific accuracy / loss target. The accuracy / loss target may be set by a user as part of the ML model data 104a provided by the user or may be based on a default accuracy / loss target. In response to the ML model 108a meeting a specific accuracy / loss target, the EC TOSC unit 106 may report various efficiency metrics, such as cumulative EC, EC per training item, etc., to the user via the user interface 102.

[0040] Although a validly trained ML model 108a may be output from the EC ML model training unit 108, this does not necessarily mean that the trained ML model 108a meets the EC target threshold for inference or the EC target per inference. With this in mind, the EC TOSC unit 106 is further configured to benchmark and / or evaluate whether the trained ML model 108a meets the EC target threshold at inference or whether it meets the EC inference target threshold at inference. The EC TOSC unit 106 passes the trained ML model 108a (that meets the EC target threshold or the EC training target threshold set by the user) to the EC ML model optimization / inference unit 110. The EC ML model optimization / inference unit 110 is configured to determine whether the trained ML model 108a meets the EC target threshold or the EC target threshold per inference. This may be performed using an input dataset that may include, for example, but is not limited to, multiple training data samples from a training dataset or multiple training data samples from a reserved portion of the training dataset used to train the trained ML model 108a. Alternatively or additionally, the input dataset for input to the trained ML model 108a may include real-world input data samples on which the trained ML model 108a may perform inference. To determine the EC for each inference of the trained ML model 108a, a batch of input data samples may be formed for input to the trained ML model 108a. The EC ML model optimization / inference unit 110 is configured to use the trained ML model 108a to process the batch of input data samples.

[0041] During inference benchmarking of the ML model 108a, the EC TOSC unit 106 is configured to monitor the EC of the hardware used by the training / inference system 100 during inference of the ML model 108a. As described, the hardware used by the training / inference system 100 may include one or more CPUs, one or more GPUs, one or more TPUs, memory units, and / or communication resources, and the EC TOSC unit 106 may query and receive real-time sensor data and / or real-time hardware parameters / data associated with the CPUs, GPUs, TPUs, memory / data storage units, and / or communication resources to measure and / or calculate estimated EC while the ML model 108a performs inference on batches of input data samples. The EC TOSC unit 106 may use the measured or calculated EC corresponding to the inferences of the ML model 108a to determine an EC estimate for each inference of the ML model 108a. For example, the EC TOSC unit 106 may estimate the EC for inferring the ML model 108a over batches of input data samples. The EC may then be divided by the number of input data samples in the batch to determine a per-inference EC estimate for the ML model 108 a. This per-inference EC estimate may be run across multiple batches of input data samples to determine an average per-inference EC estimate for the ML model 108 a.

[0042] If the per-inference EC estimate meets the EC inference target threshold or the per-inference EC target threshold, the ML model 108a is considered for output to the EC ML model output unit 112. However, the EC ML model optimization / inference unit 110 may be configured to further optimize the trained ML model 108a to reduce the per-inference EC using various optimization methods, such as, but not limited to, quantization and / or weight pruning, which may require retraining the modified ML model 108 to form the optimized ML model 110a. This additional optimization process may also be controlled by the EC TOSC unit 106 until the EC optimization target threshold is reached and / or when further optimization is not possible while the optimized EC ML model 110a still meets or is within its specific error threshold, set by the user. Once optimized and the per-inference EC estimates of the optimized ML model 110a are determined, the optimized ML model 110a, along with efficiency metrics including, but not limited to, the per-inference EC estimates, the EC for training the ML model 108a, and the EC for optimizing the ML model 110a, may be sent to an EC ML output unit 112 for storing the optimized ML model 110a and efficiency metrics as an EC ML model 112a.

[0043] As described above, multiple ML EC models 112a-112l may be stored by the ML model output unit 112 for use in inference in an energy-efficient manner, and a user can ensure from the efficiency metric associated with each EC ML model 112a that the resulting EC ML model 112a meets a given EC target threshold or an EC target per inference during inference, and also meets a given EC target threshold during training and / or an EC optimization target threshold during optimization / inference. Thus, a user can select an appropriate EC ML model 112a from the stored EC ML models 112a-112l based on the efficiency metric of the EC ML model 112a, including the EC training target threshold and / or the EC optimization / inference target threshold. If these metrics meet the user's requirements, the user can select an EC ML model 112a knowing the EC that the EC ML model 112a meets, and can therefore offset the user's EC for using the selected EC ML model 112a, such as using one or more carbon offset programs, thus helping the user meet "net-zero" carbon emissions with respect to the selected EC ML model 112a being used for inference. The training / inference system 100 can be used to help develop and / or promote sustainable AI.

[0044] 1b shows a schematic diagram of an EC training / inference framework 120 that can be used by the training / inference system 100 to control the training, optimization, and / or inference of an ML model in an energy-efficient manner. The EC training / inference framework 120 includes a training and inference / optimization level framework 122 that can define how to configure the training and / or inference of an ML model, an EC estimation framework 124 that can define key points for receiving sensor data and estimating EC in connection with training and / or inference, and an ML model framework for defining the ML model for training. The EC training / inference framework 120 can include customizable training parameters for controlling the training / inference of the ML model, output efficiency metrics such as, but not limited to, EC estimates that can be measured from sensor data and / or hardware parameters of the training / inference system 100 to evaluate the energy efficiency of the resulting trained ML model and / or the ML model for inference, regardless of whether it is effectively trained, and termination / exit criteria for early termination of the training of the ML model, such as based on an EC target threshold for training and / or an EC target threshold for inference. The EC estimates of the corresponding ML models may be converted into carbon cost or CO2 estimates, such as based on analyzing the energy mix used by training / inference system 100 to train the corresponding ML models and / or perform inference using the corresponding ML models. For example, training / inference system 100 may use computational resources of a data center / or cloud-based platform, and the energy mix includes an electricity mix associated with electricity provided to the data center / cloud-based platform when training / inference system 100 uses the computational resources to process the corresponding ML models during training and / or inference. From the electrical amount of the energy mix used by training system 100 and the EC estimates of the corresponding ML models, the carbon cost or CO2 amount of the corresponding ML models may be estimated during training and / or inference.

[0045] 1c shows a schematic diagram of the underlying hardware 130 of the training / inference system 100. In this example, the EC TOSC unit 106 is connected to at least one or more hardware components, including, but not limited to, one or more processors 132a-132m (e.g., CPUs), one or more graphical processing units 134a-134n (GPUs), one or more tensor processing units 136a-136p (TPUs), and one or more memory / storage units 138a-138q. Each of these types of hardware, i.e., the CPU 132a, the GPU 134a, the TPU 136a, and / or the memory / storage unit 138a, can be used to measure and / or estimate EC for training an ML model and / or EC for performing inference on the trained ML model. The EC TOSC unit 106 may receive real-time sensor data readings and / or parameters associated with each EC from at least one of the CPUs 132a-132m, GPUs 134a-134n, TPUs 136a-136p, and / or memory / storage units 138a-138q, while these hardware components are training ML models and / or performing ML model inference based on the trained ML models.

[0046] Various methods for measuring the EC of an ML model may include one or more of: the amount of electricity consumed by the CPUs 132a-132m, GPUs 134a-134n, TPUs 136a-136p, and / or memory / storage units 138a-138q during training or running the ML model for inference; carbon emissions, such as the amount of CO2, emitted during training or running the ML model for inference; floating point operations (FLOPS) performed by the CPUs 132a-132m, GPUs 134a-134n, TPUs 136a-136p, and / or memory / storage units 138a-138q, which provides an estimate of the amount of work performed by the computational process training or running the ML model for inference; the cost of fetching data from the memory / storage units 138a-138q while training or running the ML model for inference; the number of ML parameters and / or hyperparameters used by the ML model, which may be correlated to the amount of work; and the total elapsed real-time execution time for training or running the ML model.

[0047] For example, the EC TOSC unit 106 may receive from one or more CPUs 132a-132m corresponding CPU power (watts) over a time period when the CPU performed training and / or inference, which may be used to calculate the energy used per CPU when performing training and / or inference (e.g., in terms of kWh). The EC TOSC unit 106 may receive from one or more GPUs 134a-134n corresponding GPU power (watts) over a time period when the GPU performed training and / or inference, which may be used to calculate the energy used per GPU when performing training and / or inference (e.g., in terms of kWh). The EC TOSC unit 106 may also receive from one or more TPUs 136a-136p corresponding TPU power (watts) over a time period when the TPU performed training and / or inference, which may be used to calculate the energy used per TPU when performing training and / or inference (e.g., in terms of kWh). The EC TOSC 106 may also receive from one or more memory / storage units 138a-138q corresponding memory / storage power (watts) over the time period the memory / storage units were used / accessed during training and / or inference, which may be used to calculate the energy used per memory storage unit when performing training and / or inference. The EC TOSC 106 may calculate an EC estimate when performing training and / or inference based on the sum of the corresponding energy per CPU, energy per GPU, energy per TPU, energy per memory storage unit, etc. The EC estimate may also be converted to a carbon cost (e.g., kg of CO2 / kWh) or an estimate of CO2 based on analyzing the electricity mix associated with supplying electricity to the training / inference system 100.It should be appreciated by those skilled in the art that the EC estimate may be based on CPU, GPU, TPU, and / or memory / storage power over a period of time, but this is merely by way of example, and other types of parameters, sensor readings, etc. may be used to calculate the EC estimate during ML model training and / or inference.

[0048] For example, the energy mix of provided electricity used by the computational resources of the training and inference system 100 can be analyzed to determine an energy mix carbon cost (EMCC) measured in kilograms of CO₂ per kilowatt-hour (kgCO₂ / kWh). The EMCC can be calculated based on a combination of the percentage of electricity generated by each energy source of several energy sources (NES>0) and a corresponding carbon metric determined for each energy source depending on its type. The types of energy sources used to generate electricity can include one or more from the group of coal, oil, natural gas, geothermal, hydroelectric, nuclear, solar, wind, tidal, heat pump, and / or any other type of energy source used to generate electricity by an electricity provider. The carbon metric (CM) of each energy source can be based on an estimate of the amount of carbon dioxide (CO₂) in kilograms emitted per kWh by each ES. As an example, the EMCC can be calculated based on the following:

[0049]

number

[0050] where NES>0 is the number of energy sources used to generate electricity in the energy mix and PES k represents the percentage of electricity generated by the kth energy source in the energy mix, and CM k is the carbon metric determined for the kth energy source and has units of CO2 kg / kWh.

[0051] An EC estimate for training / inferencing an ML model may be calculated based on measuring or estimating the power consumption (e.g., watts or joules / second) of the computational resources (e.g., CPU, GPU, TPU, memory, communication resources, etc.) used by the training / inference system 100 over a time period taken (e.g., in hours) during the training / inference of the ML model. The EC estimate may be generated in terms of kWh. The EC estimate may be converted to a carbon cost or amount of CO2 based on the energy mix by multiplying the EC estimate by the EMCC of the electricity mix used by the computational resources of the training / inference system 100. For example, an EC estimate for training / inferencing an ML model may be based on the following:

[0052]

number

[0053] where NC is the number of components of computational and / or communication resources used to train / infer the ML model, and TCP k is the total component power consumed for the jth component over a time period TTh used for training or inference of an ML model, where TTh is measured in units of time. The components of the computational resources used by training / inference system 100 may include one or more CPUs, one or more GPUs, one or more TPUs, one or more memory units, memory or data storage units, communication and / or network resources, and / or one or more power sources, and / or one or more other components of computational / communication resources that may be capable of being measured as consuming energy, etc. The EC estimate may be calculated as a carbon cost based on multiplying EC by the EMCC.

[0054] The EC estimates may be based on a function of the energy mix and estimated EC per component / communication resource, and / or may be converted to carbon costs based on multiplying the EC by the EMCC, but this is for simplicity and by way of example only, and the invention is not so limited. The estimated EC for each ML model training / inference may further include various modifications to the above functions and / or combinations thereof, for example, additional factors, parameters, and / or components that can more accurately model the EC estimate for training / inferring the ML model, such as, but not limited to, emission factors based on hourly, daily, monthly, and / or yearly averages for a region or further granularity as needed, and the ML models that can be used to perform the training / inference. Those skilled in the art will understand that the ML model may be further estimated based on various factors, including, but not limited to, server, mobile device, and / or laptop emission data, EC consumption for inference, conversion or scaling to different types of target devices (e.g., different server technologies, mobile devices, computing devices, and / or laptops) on which the ML model will run, consideration of expected scaling usage of the ML model, such as the number of target devices or functional units, and / or any other combination of factors, parameters, and emissions for use in estimating EC for training and / or inferring each ML model, including combinations thereof, including modifications thereof, as described herein and / or as required by the application.

[0055] 2a shows a schematic diagram of a method 200 for training an ML model in an energy-efficient manner. The method may be performed by the training and inference system 100 using the EC TOCS unit 106 and the EC ML model training unit 108 of FIG. 1a and / or one or more computers operating at one or more locations. The method 200 may include the following steps.

[0056] In step 202, an EC target threshold for training an ML model is received.

[0057] In step 204, the ML model is trained on the training dataset of the epoch.

[0058] In step 206, an EC for training the ML model over an epoch is estimated. For example, this may include receiving real-time sensor and performance data from hardware of the ML training system used during training of the ML model over an epoch. The real-time sensor and performance data may include, but is not limited to, one or more of temperature sensor data, clock cycles associated with one or more CPUs, GPUs, and / or TPUs of the ML training system, and the number of FLOPS executed by one or more CPUs, GPUs, TPUs associated with the ML training system that train the ML model. An estimated EC is calculated based on the received real-time sensor data and / or performance data to train the ML model over an epoch.

[0059] In step 208, it is determined whether the cumulative EC estimate over the epoch and / or the previous epoch is less than an EC target threshold for training the ML model. This may include generating a cumulative energy expenditure estimate based on the estimated energy expenditure. If the accumulated EC estimate over the epoch is less than the EC target threshold (e.g., "Y"), proceed to step 210; if not (e.g., "N"), proceed to step 214.

[0060] In step 210, it is determined whether the ML model has been trained effectively. The ML model may be trained effectively if the average accuracy / loss of the ML model's output when compared to the training samples exceeds an accuracy / loss threshold or if it is determined whether the accuracy / loss target is achieved. If it is determined that the ML model has been trained effectively (e.g., "Y"), proceed to step 216; if not (e.g., "N"), proceed to step 212.

[0061] In step 212, update the epoch number and / or training data set for the next epoch and proceed to perform step 204 using the updated epoch number and / or updated training data set to train the ML model.

[0062] In step 214, determine whether the ML model is effectively trained by checking whether the average accuracy / loss of the ML model's output when compared to the training samples exceeds an accuracy / loss threshold or achieves an accuracy / loss target. If the accuracy / loss of the output across the ML model meets the performance criteria required for a effectively trained ML model (e.g., "Y"), proceed to step 216; otherwise (e.g., "N"), proceed to step 218.

[0063] In step 216, the trained ML model and associated efficiency metrics, including but not limited to, EC parameters such as cumulative EC, EC per training sample, and / or any other efficiency and / or EC metric / parameter, are output.

[0064] 2b shows a schematic diagram of a method 220 for benchmarking inference and / or optimization of ML models in an energy-efficient manner. The method may be performed by the training and inference system 100 using the EC TOCS unit 106 and the EC ML model optimization and inference unit 110 of FIG. 1a and / or one or more computers operating at one or more locations. The method 220 may include the following steps.

[0065] In step 222, an EC target threshold for inference using the ML model is received.

[0066] In step 224, a batch of input data for ML model inference is received.

[0067] In step 226, ML model inference is performed using the trained ML model on the batch of input data.

[0068] In step 228, the trained ML model with the batch of input data is used to estimate EC for ML model inference. An average EC per inference may be estimated from the per-batch EC estimate by dividing the per-batch EC estimate by the number of input data samples in the batch. The average EC per inference across all batches of input data may be accumulated and averaged to determine the average EC per inference. For example, estimating EC for ML model inference may include receiving real-time sensor and performance data from hardware that implements or executes the trained ML model that performs inference across the batches of input data. The real-time sensor and performance data may include, but is not limited to, one or more of temperature sensor data, clock cycles associated with one or more CPUs, GPUs, and / or TPUs of the ML training system, and the number of FLOPS executed by one or more CPUs, GPUs, TPUs associated with the ML training system that performs inference using the ML model. Calculating the estimated EC may be performed based on the received real-time sensor data and / or performance data during inference using the trained ML model.

[0069] In step 230, determine whether the EC estimate for each inference is greater than the EC target threshold for inference using the trained ML model. If the EC estimate for each inference is greater than the EC target threshold for inference (e.g., "Y"), proceed to step 232; if not (e.g., "N"), proceed to step 236.

[0070] In step 232, ML model optimization is performed on the trained ML model. For example, this may include modifying the trained ML model using, by way of example only and not limitation, quantization and / or weight pruning techniques, or any other suitable optimization techniques for reducing the EC of the trained ML model during inference. The modified ML model may be retrained using method 200 of FIG. 2a using an EC target threshold for training and to achieve accuracy / loss targets associated with the trained ML model, etc. Once retrained, the modified trained ML model forms an optimized ML model.

[0071] In step 234, the optimized ML model is passed to step 224, where the optimized ML model may be used in place of the trained ML model for inference, and method 220 proceeds to step 224 for performing ML model inference using the optimized ML model. Optionally, an optimization loop counter may be used to track the number of attempts to optimize the trained ML model, and once the optimization loop counter reaches a maximum optimization count threshold, method 220 ends as any further optimization may not result in the required per-inference EC threshold.

[0072] In step 236, the trained and / or optimized ML model and associated efficiency metrics including, but not limited to, EC parameters such as, for example, average EC per inference, EC per training sample, and / or any other efficiency and / or EC metric / parameter are output.

[0073] Optionally, in step 232, the ML model may be optimized using quantization and / or weight pruning. Quantization is an optimization technique used to constrain inputs from a continuous or large set of values ​​to a smaller, discrete set of values. This concept may be used in optimizing the trained ML model by constraining a set of N-bit floating-point values, typically to M-bit integers (e.g., for a CPU) or P-bit floating-point values ​​(e.g., for a GPU), where N>M and N>P (e.g., N=32, M=8, or P=16 and / or any other suitable value). For example, the weights of the trained ML model may be quantized, and the “high-accuracy” floating-point weight values ​​of the trained ML model may be quantized to a set of 8-bit integer weight values. Fair adjustment of the quantization of the ML model weight values ​​can reduce the ML model size and improve inference performance with minimal impact on accuracy. A relatively small ML model, when deployed, may run faster and require less memory, and therefore its EC per inference should be reduced. Several quantization techniques may be applied, such as, but not limited to, post-training quantization and quantization-aware training. Post-training quantization is applied to an ML model after the training phase is complete. If the impact of post-training quantization is detrimental to accuracy, quantization-aware may be applied, and method 200 may be applied to perform training, but modified if the training emulates quantization during inference.

[0074] Weight pruning is an optimization technique that enables ML models to be compressed to a smaller size with zero or minimal loss of accuracy. Trained ML models can contain millions of parameters, which can make inference very energy-intensive and expensive, especially on resource-constrained computing devices (e.g., mobile and / or edge devices). Weight pruning is the process of removing (or pruning) weights from a trained ML model that have the smallest magnitude. These weights contribute the least to the output and performance of the final ML model. While the original and pruned models have essentially the same architecture, the pruned model is sparser. Large, sparse ML models may typically outperform small, dense ML models across a variety of different architectures. Therefore, ML model optimization can be performed iteratively and incrementally to ensure minimal accuracy loss for the ML model while attempting to meet an EC target threshold per inference.

[0075] FIG. 3 shows a schematic diagram of another exemplary method 300 for measuring the efficiency during training of a target ML model. This method 300 can be used to illustrate and / or define a training / inference framework for training any type of ML model until their measured EC meets an EC target threshold for training. The method 300 and / or framework can comprise customizable training parameters for controlling the training of the target ML model, an output efficiency metric that can be measured to evaluate the energy efficiency of the trained ML model, whether it is trained effectively, and termination / termination criteria for early termination of the training of the target ML model. The method 300 / framework can include an initialization operation 302, a training operation 303, and an ML model output operation 340. The method 300 includes the following operations:

[0076] The initialization operation 302 defines how efficiency metrics are captured, the configuration of a target ML model, the configuration of training parameters for the target ML model, and the setting of termination criteria (e.g., termination criteria). The initialization phase 302 may include one or more of the following steps:

[0077] In step 304, global constraints such as global variables and efficiency metrics for tracking are defined. For example, one or more global variables and / or efficiency metrics that may be used in method 300 / framework may be selected from, but are not limited to, one or more of: epochs, K-folds, training / validation accuracy and loss, EC sum (e.g., carbon sum), FLOP sum, and time sum.

[0078] In step 306, training parameters for the target ML model are configured. The method 300 / framework can be configured to be reusable by allowing any ML model and any dataset to be plugged in. For example, customizable training parameters that can be used include, but are not limited to, batch size, number of epochs, loss function, optimizer, number of K folds, EC target threshold (e.g., carbon target threshold), and accuracy / loss threshold, etc.

[0079] In step 308, the training and / or test datasets for the target ML model are loaded and normalized.

[0080] At step 310, global tracking and efficiency metrics are initialized. Efficiency metrics are captured at three levels: global level (store global metrics), K-fold level (store K-fold metrics), and epoch level (store epoch metrics). For example, output efficiency metrics may be selected for each trained ML model and may include, without limitation, for example, number of epochs, K-folds, training / validation accuracy and loss, EC sum (e.g., carbon sum), FLOP sum, and time sum, and / or any other suitable efficiency metrics.

[0081] In step 312, the target ML model is configured, which may include, but is not limited to, defining the structure of the target ML model based on, for example, configuring hyperparameters of the target ML model, defining the type of the target ML model, and / or defining structural elements of the ML model.

[0082] In step 314, termination / termination criteria are initialized and a termination / termination criteria flag is set, which in this example is initially set to “No” (e.g., set termination criteria=“No”). The termination or termination criteria flag can be any type of value that indicates whether to terminate the training operation 320. Termination criteria for early termination of training may include, but are not limited to, for example, the required accuracy / loss being achieved, or the required accuracy / loss diverging, or the accuracy / loss not converging to the required accuracy / loss quickly enough, the accumulated EC being too high (e.g., the carbon cost is too high), or exceeding an EC target threshold (e.g., the carbon target threshold). The ML model may be considered effectively trained when the required accuracy / loss is achieved.

[0083] Once initialized, the method 300 / framework proceeds from the initialization operation 302 via connector "A" to step 316 of the ML training operation 303. The ML training operation 303 includes an outer K-fold training loop and an inner epoch training loop for training a target ML model. The outer K-fold training loop is used to control K-fold validation, in which the training dataset is divided into K subsets, where K>1. The outer K-fold loop may be repeated K times, in which each of the K subsets is selected and used as a test set, and the other K-1 subsets are used for training in the inner epoch training loop. The ML training operation 303 includes the following steps:

[0084] In step 316, tracking of K-fold metrics begins for the selected K-1 different subsets of the training dataset. The K-fold metrics for the selected K-1 subsets may include, for example, but are not limited to, the number of epochs, the number of K-folds, the training / validation accuracy and loss, the EC sum (e.g., carbon sum), the FLOP sum, and the time sum for the selected K-1 subsets of the training dataset.

[0085] In step 320, an inner epoch training loop is performed in which the target ML model is trained over the selected K-1 subsets of the training dataset. The inner training loop may terminate based on the EC termination criteria being met 320a, and the method 300 / framework proceeds from the inner training loop 303 via connector "B" to the ML model output operation 340. Alternatively, the inner training loop may also terminate based on completing a number of epochs over the selected K-1 subsets of the training dataset, where it is determined that further K-fold training 320b is required, and therefore the ML training operation 303 proceeds to step 317.

[0086] In step 317, tracking of the K-fold metrics for the selected K-1 subsets of the training dataset is stopped.

[0087] In step 318, the K-fold metrics for the selected K-1 subsets of the training data set are stored.

[0088] In step 319, it is determined whether there are K-1 more subsets of the K subsets that have not yet been selected. If there are more distinct combinations of the K-1 subsets of the K-fold subsets (e.g., "Y"), the ML training operation 303 proceeds to step 316 to initiate a K-fold metric for new, selected, distinct K-1 subsets of the training dataset, and the outer K-fold training loop is repeated. If there are no further combinations of the K-1 subsets of the K-fold subsets that have not been used to train the ML model (e.g., "N"), the ML training operation 303 proceeds to the ML model output operation 340 via connector "C" 323.

[0089] The inner epoch training loop 320 is configured to receive the selected K-1 subsets of the training dataset from step 316, which may be divided into several epochs, each epoch including a batch of training samples from the selected K-1 subsets of the training dataset. The inner epoch training loop 320 may be configured to perform several epoch iterations, each corresponding to one of the batches of training samples for that epoch iteration. The inner epoch training loop 320 for training the ML model includes the following steps:

[0090] Tracking of epoch metrics begins for the current epoch iteration in step 322. The epoch metrics of the batch of training samples for the current epoch iteration may include, for example, but are not limited to, the epoch iteration number, the K-fold number, the training / validation accuracy and loss for the current epoch iteration, the EC sum (e.g., carbon sum) for the current epoch iteration, the FLOP sum for the current epoch iteration, and the time sum for the current epoch iteration.

[0091] In step 324, the target ML model is trained over the batch of training samples for the current epoch iteration (e.g., Fit model). Once all training samples in the batch of training samples for the current epoch iteration have been processed and the target ML model has been updated accordingly, the inner epoch training loop 320 proceeds to step 326. During training of the ML model, epoch metrics related to training / validation accuracy / loss are calculated, and the EC sum for the current epoch iteration is measured and estimated based on the performance of real-time sensor data and / or the hardware performing the training of the ML model over the batch of training samples.

[0092] In step 326, tracking of epoch metrics for the current epoch iteration is stopped.

[0093] In step 328, the epoch metrics for the current epoch iteration are stored.

[0094] In step 330, it is determined (or checked) whether the EC sum (e.g., carbon cost or other appropriate measure) for the current epoch iteration is greater than an EC target threshold (e.g., carbon cost threshold or other appropriate threshold) for the epoch iteration (e.g., "yes"), then proceed to step 332; otherwise (e.g., "no") proceed to step 334. In one example, the EC sum for the current epoch iteration may be added to the accumulated EC sum of past epoch iterations, where the accumulated EC sum over all previous epoch iterations is compared to the EC target threshold. If the accumulated EC sum is greater than the EC target threshold (e.g., "yes"), the inner epoch training loop 320 proceeds to step 332.

[0095] In step 332, the termination criteria flag is set to "yes" or any other suitable value indicating that the inner epoch training loop 320 is terminated.

[0096] In step 334, it is determined (or checked) whether the accuracy / loss of the ML model meets the required accuracy / loss threshold for the ML model. If the accuracy / loss of the ML model meets the required accuracy / loss threshold (e.g., the ML model is effectively trained) (e.g., "yes"), proceed to step 336; if not (e.g., "no"), proceed to step 338. In another example for terminating the inner epoch training loop 320 based on accuracy / loss, it may be determined that the calculated accuracy / loss of the ML model is diverging, where the calculated accuracy / loss for multiple consecutive epoch iterations, including the current epoch iteration, is deteriorating or diverging away from the required accuracy / loss. If so, proceed to step 336, and terminate the inner epoch training loop 320. In another example for terminating the inner epoch training loop 320 based on accuracy / loss, it may be determined that the accuracy / loss of the ML model is not converging to the required accuracy / loss quickly enough. For example, the EC cost per improvement in the accuracy / loss of the ML model compared to the previous epoch iteration has reached a plateau. If so, proceed to step 336 and terminate inner epoch training loop 320. In another example, terminate training of the ML model in response to determining that the convergence rate of the ML model's accuracy / loss towards the required accuracy / loss for the current epoch is below a convergence rate threshold for multiple consecutive epochs. If so, proceed to step 336 and terminate inner epoch training loop 320.

[0097] In step 336, the termination criteria flag is set to "yes" or any other appropriate value indicating that the inner epoch training loop 320 is terminated.

[0098] In step 338, it is determined whether the termination criteria flag is met, i.e., whether the termination criteria flag is set to "yes" or other appropriate value indicating that the inner epoch training loop 320 should be terminated. If the termination criteria flag is met (e.g., "yes"), proceed via EC termination criteria met 320a to connector "B" 321 of the ML training operation 303, which connects to the ML model output operation 340. If the termination criteria flag is not met (e.g., "no"), i.e., whether the termination criteria flag is set to "no" or other appropriate value indicating that the inner epoch training loop 320 should not be terminated, proceed to step 339.

[0099] In step 339, it is determined whether there are more epoch iterations to be performed. If there are more epoch iterations to be performed (e.g., “Y”), the inner epoch training loop 320 proceeds to step 322 to select the next batch of training samples for the next epoch iteration and start the epoch metrics for the next epoch iteration as the current epoch iteration, thus repeating the inner epoch training loop 320. If there are no more epoch iterations to be performed to train the ML model (e.g., “N”), the inner epoch training loop 320 determines whether more K-fold training 320b should be performed and proceeds to step 317 of the ML training operation 303.

[0100] The ML model output operation 340 is input from the ML training operation 303 via connector "B" 321 or connector "C" 323. The ML model output operation 340 includes the following steps:

[0101] In step 342, global metric tracking is stopped.

[0102] In step 344, the global metrics are stored.

[0103] In step 346, the final trained ML model on the test data is evaluated, which may include performing ML model inference on the test data, evaluating the EC for each inference of the trained ML model, and also measuring the accuracy / loss of the ML model when inferring on test data that is not part of the training dataset.

[0104] In step 348, the accuracy / loss and EC for each inference and / or the EC for training the ML model may be output for review by the user.

[0105] The efficiency metrics collected at the global level, the K-fold level, and the epoch level may be exported and / or stored with the trained ML model in step 350. It may also be verified that the trained ML model is effectively trained, i.e., that the required accuracy / loss is achieved, which may be indicated to the user and / or stored with the trained model.

[0106] In step 352, output charts displaying the efficiency and / or performance (e.g., EC and / or carbon cost, and accuracy / loss) of the ML model at a global level, a K-fold level, and / or an epoch level may be displayed to the user for further evaluation and analysis, which may be used by the user to design improved ML models for training, etc.

[0107] Additionally and / or alternatively, successfully trained ML models that meet the EC target threshold for training may be selected for inference benchmarking and evaluated for whether their measured EC meets the EC target threshold for inference or the EC target threshold per inference. The selected trained ML models may be further optimized by quantization and / or weight pruning along with retraining to further reduce their measured EC so as to meet the EC target threshold for inference or the EC target threshold per inference.

[0108] FIG. 4 is a schematic diagram showing a table of test results 400 for training an active learning baseline ML model using the method / framework 200 or 300 of FIGS. 2a and 3. In this example, EC is based on carbon cost. The baseline ML model is based on the Visual Geometry Group (VGG)-16 model architecture. At this stage, no attempt was made to optimize the model or any of the ML model's execution parameters or hyperparameters. The method / framework 200 or 300 of FIGS. 2a and 3 can be used to provide a baseline set of accuracy and EC efficiency measures whose performance may be improved and / or optimized. Some performance characteristics may be affected by factors outside the ML model's direct control, such as hardware (CPU, GPU, or TPU) characteristics. To account for variability, five runs using the same ML model parameters were performed and the results were averaged. The table of test results 400 in FIG. 4 shows the results of the five experiments and the average values ​​of the variable performance measures. In summary, an average test accuracy of 50.8% was achieved in 1730.5 seconds at a carbon cost of 0.074 kg.

[0109] FIG. 5 is a schematic diagram showing test results 500 for training a transfer learning model using the method / framework 200 or 300 of FIGS. 2a and 3. The baseline model used in FIG. 4 is based on the VGG-16 architecture. FIG. 5 shows the results of a transfer learning ML model incorporating the VGG16 model. In this example, all other parameters in the framework remained unchanged compared to the baseline parameters used in FIG. 4, and the same Tesla P100-PCIE GPU was used as in the baseline run of FIG. 4. Test results table 502 in FIG. 5 shows the results of five identical experiments conducted using this set of experimental parameters. Again, the EC used in this case was carbon cost or total carbon emissions. The accuracy performance across five runs of this transfer learning ML model closely matches the very similar carbon metrics. Test results table 504 shows a comparison between the baseline results (average) recorded in the table or test results 400 in FIG. 4 (last row) and the average results in table 502 in FIG. 5 when transfer learning is applied. Comparable training, validation, and test accuracies were recorded as for the transfer learning model in Figure 5, while resulting in a reduction in total training time and a corresponding reduction in total carbon cost (see dashed circle) compared to the average baseline ML model results in Figure 4. This demonstrates the benefits of using a transfer learning architecture in ML compared to the active learning in Figure 4.

[0110] 6 shows a schematic example of a computing system / apparatus 600 for performing any of the methods, operations, or processes described herein and / or for implementing any of the systems, units, and / or apparatuses described herein. The illustrated computing system / apparatus 600 is an example of a computing device or platform. Those skilled in the art will appreciate that other types of computing devices / systems / platforms, such as distributed computing systems, may alternatively be used to perform the methods described herein.

[0111] The apparatus (or system) 600 includes one or more processors 602 (e.g., a CPU, a GPU, a TPU, etc.). The one or more processors 602 control the operation of the other components of the system / apparatus 600. The system / apparatus 600 may be part of a computing device, computing system, distributed computing system, cloud computing platform, etc. for performing the system / apparatus functions and / or one or more methods / operations / processes described herein. The one or more processors 602 may include, for example, a general-purpose processor. The one or more processors 602 may be single-core or multi-core devices. The one or more processors 602 may include a central processing unit (CPU) or a graphical processing unit (GPU), or a tensor processing unit (TPU). Alternatively, the one or more processors 602 may include dedicated processing hardware, e.g., a RISC processor or programmable hardware with embedded firmware. Multiple processors may be included. In some embodiments, the one or more processors 602 may be part of a distributed computing system, such as a cloud computing system and / or a cloud computing platform.

[0112] The system / apparatus comprises a memory system or memory 604 including a working or volatile memory 614. The one or more processors 602 can access the volatile memory 614 to process data and can control the storage of data in the memory. The volatile memory 614 may comprise any type of RAM, e.g., static RAM (SRAM), dynamic RAM (DRAM), or flash memory such as an SD card. In some embodiments, the memory 604 and / or the one or more volatile memories 614 may comprise more than one of a plurality of memories 604 forming part of a distributed computing system, such as a cloud computing system and / or a cloud computing platform.

[0113] The system / apparatus comprises non-volatile memory 612. The non-volatile memory 612 may store a set of operating or operating system instructions 615 for controlling operation of the processor(s) 602 in the form of computer-readable instructions and / or software instructions 616 in the form of computer-readable instructions that, when executed on one or more processors 602, cause the processor(s) to perform the methods, processes, operations, and / or functions of the methods, systems, apparatuses, and / or training / inference frameworks described herein. The non-volatile memory 612 may be any type of memory, such as read-only memory (ROM), flash memory, SD drive, magnetic drive memory, or magnetic disk drive memory, depending on the application. In some embodiments, the non-volatile memory 612 may comprise more than one of multiple non-volatile memories 612 forming part of a distributed computing system, such as a cloud computing system and / or a cloud computing platform.

[0114] The one or more processors 602 are configured to execute operational instructions 615 and / or software instructions 616 to cause the system / device to perform any of the methods or processes described herein. The operational instructions 615 may comprise code (i.e., drivers) associated with hardware components of the system / device 600 as well as code associated with basic operations of the system / device 600. Generally, the one or more processors 602 execute one or more instructions of the operational instructions 615 and / or software instructions 616, which are stored permanently or semi-permanently in non-volatile memory 612, and use volatile memory 614 to temporarily store data generated during execution of said operational instructions 615 and / or software instructions 616.

[0115] The one or more processors 602 may be connected to a network interface 606 including a transmitter (TX) and a receiver (RX) for communicating via a network with other apparatuses and systems, such as wastewater assets, a wastewater network, a wastewater network management system, an environmental data measurement service and / or an operator, and / or any other apparatuses, services, systems, and / or devices as required by the application. The one or more processors 602 may optionally be connected to a user interface (UI) 610 for user or operator input for instructing or using the computing system 600 and / or outputting data therefrom. The one or more processors 602 may optionally be connected to a display 608 for displaying output to a user or operator. The at least one processor 602, together with at least one memory 604 and computer program code 615, 616, is configured to cause the computing system 600 to perform at least the operations, methods, and / or processes as disclosed in connection with, for example, the schematics, flow diagrams, or operations as described using any of FIGS. 1a-5 and their associated features.

[0116] FIG. 7 illustrates a non-transitory medium 700 according to some embodiments. The non-transitory medium 700 may include a computer-readable storage medium 702 and / or an input / output mechanism 704 for enabling the computing system 600 to access the computer-readable medium 702. In this example, the non-transitory medium is a USB stick, but this is merely an example; the invention is not so limited; one skilled in the art will appreciate that the non-transitory medium 700 may be any other type of computer-readable medium, such as, for example, a CD, DVD, USB stick, Blu-ray disc, flash drive, and / or any other computer-readable medium as required by the application. The non-transitory medium 700 stores computer program code that causes an apparatus to perform one or more of the methods, operations, and / or processes of any preceding processes, for example, as disclosed in connection with the flow diagrams and schematic diagrams of FIGS. 1a-5 and their associated features.

[0117] Implementations of the methods or processes described herein may be realized in digital electronic circuitry, integrated circuits, specially designed ASICs (application-specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. They may also include a computer program product (e.g., software stored on a magnetic or optical disk, a memory, a programmable logic device, etc.) comprising computer-readable instructions that, when executed by a computer such as that described in connection with FIG. 6, cause the computer to perform one or more of the methods described herein.

[0118] Any system features described herein may also be provided as method or process features, and vice versa. As used herein, means-plus-function features may alternatively be expressed in terms of their corresponding structure. In particular, method aspects may also apply to system aspects, and vice versa.

[0119] Furthermore, any, some, and / or all of the features in one embodiment may be applied to any, some, and / or all of the features in any other embodiment, in any appropriate combination. It is also understood that specific combinations of the various features described and defined in any embodiment of the invention may be implemented and / or provided and / or used independently.

[0120] While several embodiments have been shown and described, it will be understood by those skilled in the art that changes may be made in these embodiments without departing from the principles of the present disclosure, the scope of which is defined in the claims and their equivalents.

Claims

1. 1. A computer-implemented method of operating a machine learning (ML) training system (100) for training an ML model (108a), comprising: receiving an energy expenditure (EC) target threshold for training (202); training (204) an ML model (108a) based on the training dataset of the epoch; estimating (206) the EC of the ML training system (100) while training the ML model (108a) over the epochs; determining (208) whether the cumulative EC estimate is less than the EC target threshold; In response to determining that the cumulative EC estimate is less than the EC target threshold, updating (210, 212) the training data set for a new epoch and repeating the training, estimating, and generating steps using the updated training data set with the new epoch until the ML model is effectively trained; terminating training of the ML model in response to determining that the cumulative EC meets or exceeds the EC target threshold (218); outputting (216) the successfully trained ML model and associated EC metrics; A computer-implemented method comprising:

2. estimating the EC of the ML training system receiving real-time sensor and performance data from the ML training system's hardware during training of the ML model over the epoch; estimating the EC of the ML training system (100) based on the received real-time sensor and performance data over the epoch; The computer-implemented method of claim 1 further comprising:

3. The hardware of the ML training system (100) comprises: one or more central processing units (CPUs) (132a-132m); one or more graphical processing units (GPUs) (134a-134n); one or more tensor processing units (TPUs) (136a-136p); one or more memory or storage units (138a-138q); one or more communication or network resources, and / or Any other hardware of the ML training system (100) associated with training the ML model (108a) and influencing the EC of the ML training system (100). The computer-implemented method of claim 2 , comprising one or more from the group:

4. the real-time sensor and performance data; the electrical usage of the hardware of the ML training system (100) during training of the ML model (108a); and temperature sensor data associated with the hardware used during training of the ML model (108a); clock cycles associated with one or more CPUs (132a-132m), GPUs (134a-134n), and / or TPUs (136a-136p) of the ML training system; power usage data associated with one or more CPUs (132a-132m), GPUs (134a-134n) and / or TPUs (136a-136p) and / or memory units (138a-138q) of the ML training system (100); the number of floating-point operations (FLOPS) performed by one or more CPUs (132a-132m), GPUs (134a-134n), and / or TPUs (136a-136p), and / or memory units (138a-138q) associated with the ML training system (100) when training the ML model (108a); The computer-implemented method of claim 2 or 3, comprising one or more of:

5. 5. The computer-implemented method of claim 1, wherein estimating the EC of the ML training system further comprises invoking an EC routine configured to monitor performance of the EC of a corresponding portion of a hardware ML training system.

6. The computer-implemented method of claim 1 , further comprising generating the cumulative EC estimate based on the estimated EC of a current epoch and any previous epochs.

7. The EC estimate is calculated based on carbon or CO 2 The computer-implemented method of claim 1 , wherein the energy consumption is measured in terms of emissions.

8. The computer-implemented method of claim 1 , wherein the EC estimate is an EC estimate for each training sample.

9. determining whether the ML model (108a) has been effectively trained based on whether the average accuracy / loss of the ML model (108a) output achieves a required accuracy / loss threshold; terminating training of the ML model (108a) in response to determining that the ML model (108a) has been effectively trained; and The computer-implemented method of claim 1 , further comprising:

10. 10. The computer-implemented method of claim 1, wherein terminating training of the ML model (108a) in response to determining that the accumulated EC meets or exceeds the EC target threshold further comprises determining whether the ML model (108a) has been effectively trained based on whether the average accuracy / loss of the ML model (108a) output achieves a required accuracy / loss threshold.

11. 11. The computer-implemented method of claim 1, further comprising terminating training of the ML model in response to determining that the accuracy / loss of the ML model diverges from the required accuracy / loss threshold over multiple consecutive epochs.

12. 12. The computer-implemented method of claim 1, further comprising terminating training of the ML model in response to determining that a convergence rate of the accuracy / loss of the ML model toward the required accuracy / loss for a current epoch falls below a convergence rate threshold for multiple consecutive epochs.

13. For each of the trained ML models (108a-108m) that have been effectively trained, for each effectively trained ML model (108a), the following steps are performed: receiving 222 an EC target threshold for inferencing using each of the trained ML models; receiving 224 a batch of input data for input to each of the trained ML models for inference; performing ML model inference using the trained ML model on the batches of input data (226); estimating 228 an EC for each inference of the trained ML model when performing ML model inference on the batches of input data; determining (230) whether the EC estimate for each inference is greater than the EC target threshold for the inference; In response to determining that the EC estimate for each inference is less than or equal to the EC target threshold, outputting (236) an EC ML model including the trained ML model and data representative of the EC estimate for each inference; performing EC benchmarking of each of the effectively trained ML models (108a-108m) for inference based on: performing ML model optimization (232) on the trained ML models to reduce the EC estimates per inference in response to determining that the EC estimates per inference are greater than the EC target threshold for inference; and repeating EC benchmarking (234) of the resulting optimized ML model (110a). The computer-implemented method of claim 1 , further comprising:

14. Estimating EC for each inference is estimating a total EC for processing the batch of input data; and estimating an average EC per inference based on dividing the total EC by the number of input data samples in the batch of input samples; The computer-implemented method of claim 10, comprising:

15. Performing optimization of the trained ML model includes: modifying the trained ML model (108a) based on at least one of a quantization technique and a weight pruning technique to form an optimized ML model (110a); retraining the optimized ML model (110a) for training or when the optimized ML model has been effectively trained to meet an EC target threshold; 15. The computer-implemented method of claim 13 or 14, further comprising:

16. 1. A computer-implemented method of operating an ML training / inference system to perform energy consumption (EC) benchmarking of ML models trained for inference, comprising: receiving 222 an EC target threshold for inferencing using each of the trained ML models; receiving 224 a batch of input data for input to each of the trained ML models for inference; performing ML model inference using the trained ML model on the batches of input data (226); estimating 228 an EC for each inference of the trained ML model when performing ML model inference on the batches of input data; determining (230) whether the EC estimate for each inference is greater than the EC target threshold for the inference; In response to determining that the EC estimate for each inference is less than or equal to the EC target threshold, outputting (236) an EC ML model including the trained ML model and data representative of the EC estimate for each inference; In response to determining that the per-inference EC estimate is greater than the EC target threshold for inference, performing ML model optimization (232) on the trained ML model to reduce the per-inference EC estimate, and repeating EC benchmarking (234) of the resulting optimized ML model (110a); A computer-implemented method comprising:

17. Estimating EC for each inference is estimating a total EC for processing the batch of input data; and estimating an average EC per inference based on dividing the total EC by the number of input data samples in the batch of input samples; 17. The computer-implemented method of claim 16, comprising:

18. Performing an optimization of the trained ML model (108a) comprises: modifying the trained ML model (108a) based on at least one of a quantization technique and a weight pruning technique to form an optimized ML model (110a); retraining the optimized ML model (110a) for training or when the optimized ML model has been effectively trained to meet an EC target threshold; 18. The computer-implemented method of claim 16 or 17, further comprising:

19. The trained ML model (108a) receiving an energy expenditure (EC) target threshold for training (202); training (204) the ML model (108a) based on a training dataset of epochs; estimating (206) the EC of the ML training system (100) while training the ML model (108a) over the epochs; determining (208) whether the cumulative EC estimate is less than the EC target threshold; In response to determining that the cumulative EC estimate is less than the EC target threshold, updating (210, 212) the training data set for a new epoch and repeating the training, estimating, and generating steps using the updated training data set with the new epoch until the ML model is effectively trained; terminating training of the ML model in response to determining that the cumulative EC meets or exceeds the EC target threshold (218); outputting (216) the successfully trained ML model and associated EC metrics; 18. The computer-implemented method of claim 15, wherein the training or retraining is based on:

20. 20. A computer program product (700) comprising computer readable instructions that, when executed by a computer (600), cause the computer (600) to perform the method of any one of claims 1 to 19.

21. 20. An apparatus (600) comprising one or more processors (602) and a memory (604), the memory (604) storing computer readable instructions that, when executed by the one or more processors (602), cause the apparatus (600) to perform the method of any one of claims 1 to 19.

22. one or more processors (132a-132m); and a memory (138a), the memory (138a) storing computer-readable instructions that, when executed by the one or more processors (132a-132m), cause the device (100) to perform the method of any one of claims 1 to 19.

23. an ML training unit (108) configured to perform the computer-implemented method of claim 1 for training one or more ML models (108a-108m); an ML inference and optimization unit (110) configured to perform the computer-implemented method of claim 16 for benchmarking one or more effectively trained ML models (108a-108m); estimating an energy expenditure (EC) of the ML training unit (106) that trains each ML model (108a) of the one or more ML models (108a-108m); Estimating the energy consumption (EC) of the ML inference and optimization unit (110) when benchmarking each successfully trained ML model; configured to control the ML training unit (108) when training the ML model or an ML inference and optimization unit (110) when performing ML model inference based on the estimated EC. an energy expenditure training and optimization control unit (106); 1. A machine learning training and inference system comprising:

24. the energy expenditure training and optimization control unit (106) selects effectively trained and / or optimized ML models (108a-108m, 110a-110m) that satisfy corresponding EC target thresholds for training and EC target thresholds for inference; storing the selected effectively trained and / or optimized ML models (108a-108m, 110a-110m) together with their corresponding training EC target thresholds and inference EC target thresholds for use by one or more users for inferring in an energy-efficient manner, wherein each of the one or more users may perform carbon offsetting based on the corresponding training EC target thresholds and inference EC target thresholds of the effectively trained or optimized ML models (108a-108m, 110a-110m) used by each of the one or more users; 24. The machine learning training and inference system (100) of claim 23, configured to:

Images