Hyperparametric optimization using partitioned machine learning models

Abstract

Certain aspects of the present disclosure provide techniques and apparatus for improved machine learning. A plurality of sub-networks of the neural network are determined. Training of a first sub-network of the plurality of sub-networks is facilitated using a first set of training examples from the plurality of sets of training examples, and training of a second sub-network of the plurality of sub-networks is facilitated using a second set of training examples from the plurality of sets of training examples. A first loss is generated by processing the second set of training examples using the first sub-network. An approximate marginal likelihood of the neural network is generated based at least in part on the first loss, and one or more hyper-parameters of the neural network are refined based on the approximate marginal likelihood.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to Greek patent application serial number 20220100793 filed on September 28, 2022, which is hereby incorporated by reference into this application. Background Art

[0003] Aspects of the present disclosure relate to machine learning.

[0004] Machine learning architectures have been used to provide solutions to various computing problems. There are a variety of machine learning model architectures, such as artificial neural networks (which may include convolutional neural networks (CNNs), recursive neural networks (RNNs), deep neural networks, generative adversarial networks (GANs), etc.), random forest models, etc. Many machine learning models rely on well-tuned hyperparameters (such as weight decay, learning rate, etc.) during training to perform well. In conventional systems, iterative training is usually used to define hyperparameters, such as by manually, randomly or automatically (e.g., using Bayesian optimization) to select or define a candidate hyperparameter set. The model can then be trained using the selected hyperparameters, and the model performance can be evaluated. A new set of hyperparameters is then defined for a new round of model training. Such conventional hyperparameter adjustment processes are typically slow, require a large amount of computing resources (e.g., to train the model multiple times), and often result in suboptimal results.

[0005] Some methods attempt to achieve hyperparameter optimization during training by using a validation set of data. However, these methods typically rely on a large validation set (which is not always available). Furthermore, model performance can often be improved by using the validation set itself for training rather than for hyperparameter optimization. Therefore, conventional solutions cannot provide optimal hyperparameter refinement and model accuracy. Summary of the invention

[0006] Certain aspects provide a method comprising: determining a plurality of subnetworks of a neural network; facilitating training of a first subnetwork of the plurality of subnetworks using a first set of training examples from the plurality of sets of training examples; facilitating training of a second subnetwork of the plurality of subnetworks using a second set of training examples from the plurality of sets of training examples; generating an approximate marginal likelihood of the neural network based at least in part on a first loss generated by processing the second set of training examples using the first subnetwork; and refining one or more hyperparameters of the neural network based on the approximate marginal likelihood.

[0007] Other aspects provide: a processing system configured to perform the aforementioned methods and those described herein; a non-transitory computer-readable medium comprising instructions that, when executed by one or more processors of the processing system, cause the processing system to perform the aforementioned methods and those described herein; a computer program product embodied on a computer-readable storage medium comprising code for performing the aforementioned methods and those further described herein; and a processing system comprising components for performing the aforementioned methods and those further described herein.

[0008] The following description and associated drawings set forth in detail certain illustrative features of the one or more aspects. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] The drawings depict certain aspects of the one or more aspects and therefore are not to be considered limiting of the scope of the present disclosure.

[0010] Figure 1 Depicts an example workflow for hyperparameter optimization using a partitioned machine learning model.

[0011] Figure 2A , Figure 2B and Figure 2C Depicts an example workflow for performing hyperparameter optimization using a partitioned machine learning model.

[0012] Figure 3 is a flowchart depicting an example method for hyperparameter optimization using a machine learning model.

[0013] Figure 4 is a flowchart depicting an example method for training machine learning model partitions for hyperparameter optimization.

[0014] Figure 5 is a flow chart depicting an example method for approximating marginal likelihood using a partitioned machine learning model to achieve improved hyperparameter optimization.

[0015] Figure 6 is a flowchart depicting an example method for parameter and hyperparameter optimization using federated learning.

[0016] Figure 7 is a flowchart depicting an example method for refining hyperparameters of a partitioned machine learning model.

[0017] Figure 8 An example processing system configured to perform various aspects of the present disclosure is depicted.

[0018] To facilitate understanding, identical reference numerals have been used, where possible, to designate identical elements that are common to the figures. It is contemplated that elements and features of one aspect may be beneficially incorporated in other aspects without further recitation. DETAILED DESCRIPTION

[0019] Aspects of the present disclosure provide apparatus, methods, processing systems, and non-transitory computer-readable media for improved machine learning hyperparameter optimization.

[0020] In some aspects, a machine learning model can be partitioned or labeled into a set of model partitions, where each partition can be trained using a corresponding set of training data (e.g., a corresponding partition of a training data corpus). As discussed in more detail below, such partitioning of the model can then be used to efficiently determine or approximate the marginal likelihood of the model, which can be used to drive hyperparameter optimization.

[0021] In one aspect, the marginal likelihood can be used as a target during training to enable active optimization of hyperparameters using the training set itself (e.g., using data used to train or refine model parameters) without using a validation set or other data. Typically, the marginal likelihood is robust to overfitting. However, in conventional methods, determining the marginal likelihood of a model is intractable for various common architectures such as deep neural networks. For example, some methods involve training many models in parallel in order to compute the marginal likelihood, which is often prohibitively expensive in terms of computational resources.

[0022] In various aspects of the present disclosure, partitioned models and training can be used to efficiently approximate marginal likelihoods, as discussed in more detail below. In one aspect, marginal likelihood approximation is performed using differentiable techniques, thereby enabling efficient gradient-based hyperparameter optimization. In addition, in some aspects described herein, mini-batch estimates of marginal likelihoods can be generated, thereby bringing various benefits of stochastic gradient descent to hyperparameter optimization. In addition, in some aspects, the techniques and systems disclosed herein can be easily applied to a federated learning environment, and the communication overhead of such a federated learning approach can be further reduced, as discussed in more detail below.

[0023] Using aspects of the present disclosure, machine learning model accuracy can be significantly improved (e.g., in terms of accuracy or test log-likelihood) and total training time can be maintained or reduced compared to conventional methods. In addition, learning invariance can use some aspects of the present disclosure to improve performance in low data states.

[0024] In some aspects of the disclosure, partitioning of a neural network model is described as an example application of the techniques disclosed herein. However, aspects of the disclosure can be readily applied to a variety of model architectures. In general, aspects of the disclosure can be applied to improve hyperparameter optimization of any model architecture that can be partitioned (e.g., having parameters such as weights that can be labeled or partitioned into groups or subsets and trained separately). In addition, while some of the examples described below involve partitioning a model into a defined number of subsets (e.g., three subnets, four subnets, etc.), a model can generally be partitioned into any number of partitions or subsets depending on the particular implementation.

[0025] In some aspects, the training data set may be partitioned or divided into discrete (e.g., non-overlapping) blocks or subsets of the training data. To estimate the marginal likelihood within a single machine learning model, the system may partition the parameters of the model (e.g., weights of a neural network) corresponding to the blocks of training data (e.g., one partition for each subset of the training data). Each partition may then be trained using examples from a corresponding subset of the training data (or from a corresponding subset and one or more previous subsets without using training data from subsequent subsets to train the partition, as discussed in more detail below). As discussed in more detail below, each given partition may then be evaluated using one or more unseen subsets of the training data (e.g., data not used to train a given partition) to generate a marginal likelihood.

[0026] Example workflow for hyperparameter optimization

[0027] Figure 1 An example workflow 100 for hyperparameter optimization is depicted.

[0028] In the illustrated example, the training component 120 accesses or evaluates the machine learning model 102 and the training data set 115 to generate the hyperparameters 125. In one aspect, the workflow 100 corresponds to a training process for the machine learning model 102. That is, the training component 120 can generate the hyperparameters 125 (also based on the training data 115) while the machine learning model 102 is being trained (e.g., based on the training data 115). That is, in one aspect, the training component 120 (or another component) can use the training data 115 to refine one or more parameters of the machine learning model 102 while also using the same training data 115 to refine one or more hyperparameters 125.

[0029] For example, the training component 120 (or another component) can pass one or more examples from the training data 115 through the machine learning model 102 to generate an output prediction or inference, which can then be compared to the ground truth label associated with the example to generate a loss. The loss can then be used to refine the parameters of the machine learning model 102 (e.g., using backpropagation). In the illustrated workflow, the training component 120 can also use the generated inferences and / or losses to approximate the marginal likelihood of the machine learning model 102, as discussed in more detail below, and use this to refine or update the hyperparameters 125. The training component 120 can then start the next round or epoch of training.

[0030] In general, workflow 100 may be used to update parameters and / or hyperparameters 125 using each individual training example separately (e.g., using stochastic gradient descent) or based on batches of training examples (e.g., using batch gradient descent).

[0031] The training data 115 generally corresponds to any data used to train, refine, or update the machine learning model 102. In some aspects, the training data 115 includes a plurality of training examples, where each example includes input data and a corresponding label or target value (e.g., for supervised learning). In some aspects, the example may not include a target label (e.g., for unsupervised learning). The specific content of the training data 115 may vary depending on the specific implementation and task. For example, for a computer vision task, each training example may include an image, and the label or target may correspond to the desired goal of the model, such as indicating what object the image depicts, the location of one or more specific objects in the image, etc. Although illustrated as residing separately from the training component 120 for conceptual clarity, in some aspects, some or all of the training data 115 may be stored or maintained locally by the training component. In addition, although depicted as a single repository for conceptual clarity, in various aspects, the training data 115 may be accessed from multiple data sources and / or storage devices.

[0032] The training component 120 can generally be implemented using hardware, software, or a combination of hardware and software, and can be implemented as a discrete system or as part of a larger computing system.

[0033] Hyperparameters 125 generally correspond to any variables or values ​​that control the learning process of the machine learning model 102 (as opposed to parameters that are learned during training and used during inference). For example, hyperparameters 125 may include variables such as learning rate, batch size, mini-batch size, weight decay, etc. In contrast, model parameters may include variables such as weights and / or biases (e.g., for a neural network architecture) of one or more edges or links in the model.

[0034] In the illustrated example, the machine learning model 102 is an artificial neural network. However, as discussed above, aspects of the present disclosure can be readily applied to a variety of model architectures. As illustrated, the machine learning model 102 includes a set of nodes 105 and a set of edges 110, wherein each edge 110 is associated with one or more trainable parameters (e.g., wherein the value of each parameter is learned based on training data during a training phase). For example, each edge 110 may have a corresponding weight, bias, etc.

[0035] In some aspects, the parameters of each edge 110 are learned based on the training data 115. For example, as discussed above, each training example from the training data 115 can be processed using the machine learning model 102 to generate an output inference, which can then be compared to the ground truth labels to generate a loss for refining the parameters of one or more edges 110.

[0036] In the illustrated example, the edges 110 are partitioned into a plurality of subgroups or subnetworks as indicated by solid, dashed, and dotted lines. That is, a first set of edges 110 is associated with a first partition or subnetwork (e.g., edges 110 with solid lines), a second set of edges 110 is associated with a second partition or subnetwork (e.g., edges 110 with dashed lines), and a third set of edges 110 is associated with a third partition or subnetwork (e.g., edges 110 with dashed lines). While the illustrated example includes three discrete subnetworks, in various aspects, the machine learning model 102 may be partitioned into any number of subnetworks.

[0037] In some aspects, a random or pseudo-random process is used to partition the model parameters, such as by randomly assigning each weight or edge 110 to a partition. In one aspect, if the model is large enough (e.g., if the neural network is wide enough and / or if there are a sufficient number of parameters), it is generally unlikely that any generated partition will isolate any of the nodes 105. That is, all partitions are likely to have at least one edge 110 entering each node 105, and at least one edge 110 leaving each node 105, thereby enabling backpropagation during training. However, if such isolation occurs in any partition, the overall training process can remain relatively unaffected (e.g., as long as the number of isolated nodes is low).

[0038] In at least some aspects, to prevent orphans altogether, a variety of steps can be taken. For example, in one aspect, the training component 120 (or another component) can evaluate each partition to ensure that there are no orphans before starting training. If so, the training component 120 (or another component) can repartition the machine learning model 102. As another example, in some aspects, the training component 120 (or another component) can use a partitioning technique that ensures that no orphan nodes are created.

[0039] In general, the partitions of the machine learning model 102 can have a corresponding relationship with the partitions of the training data 115 (and vice versa). For example, if the training data 115 is partitioned into K data blocks or sets, the machine learning model 102 can be partitioned into K subnets (e.g., by the training component 120). Similarly, if the machine learning model 102 is divided into C subnets, the training data 115 can be partitioned into C data sets (e.g., by the training component 120). As discussed above and discussed in more detail below, each partition of the training data 115 has a one-to-one correspondence with a partition of the machine learning model 102.

[0040] Parameters associated with any given partition of the machine learning model 102 can then be refined using the corresponding partition from the training data 115 and / or training examples from one or more previous partitions. As used herein, partitions of the training data 115 can be referred to as "first" or "last" based on a fixed ordering, where the ordering can be randomly assigned or defined at the beginning of training. For example, a corpus of training data 115 can be randomly partitioned (e.g., by the training component 120) into a first set, a second set, a third set, etc., where the first set is "before" the second set and the third set, and the third set is "after" the first set and the second set.

[0041] In some aspects, the subnets or model partitions may be similarly ordered based on this ordering of the training data subsets. For example, each model partition may be assigned to or otherwise associated with a particular set of training data, where the model partition may inherit the ordering of the training data. In this manner, for conceptual clarity, the subnets or model partitions may be similarly referred to as "first" or "last".

[0042] For example, a first model partition may be trained using data from a first subset of training data 115 (e.g., indicated by edge 110 depicted using a solid line). A second model partition may be trained using only data from a second (non-overlapping) subset of training data 115 or based on the second subset of training data 115 and all (or at least some) previous subsets (e.g., the first subset) combined (e.g., indicated by edge 110 depicted using a dashed line). As discussed above and discussed in more detail below, by training model partitions using non-overlapping sets of training data 115, training component 120 is able to efficiently approximate the marginal likelihood of machine learning model 102. That is, because each partition of machine learning model 102 is trained on an appropriate subset of training data 115, some or all of the training data 115 that has not yet been trained for each partition (e.g., any subsequent subset of training data 115) can be used to efficiently approximate the marginal likelihood, as discussed in more detail below.

[0043] In some aspects, the machine learning model 102 is partitioned into disjoint (non-overlapping) subnetworks. In other aspects, the machine learning model 102 can be partitioned into overlapping subnetworks. For example, a first subnetwork can correspond to solid edges 110, a second subnetwork can correspond to a combination of solid edges 110 and dashed edges 110, and a third subnetwork can correspond to all edges 110. That is, if the machine learning model 102 has a set of parameters w, then these parameters can be partitioned into disjoint sets (w1, w2, ..., w c Each subnet may then include a single set of parameters (e.g., subnet of w1, subnet of w2), etc., or may include partially overlapping parameters (e.g., subnet of w1, subnet of w1 and w2 combined, etc.).

[0044] In one aspect, even if the subnets overlap, the training component 120 can still optimize the model parameters with respect to each specific partition. For example, if the second subnet includes parameters w1 and w2, the training component 120 can still update or refine the parameters w2 while training the second subnet, while keeping the parameters w1 fixed or frozen (e.g., because the parameters w1 are learned when training the first subnet, and the second set of training data 115 used to refine the parameters of the second subnet (w2) should not be used to change the parameters of the first subnet (w1)).

[0045] For example, in the illustrated example, the parameter w of the machine learning model 102 is partitioned into a set (w1, w2, w3) (e.g., indicated by solid lines, dashed lines, and dotted lines, respectively). The three subnets can then be defined as subnet1 = (w1, 0, 0), subnet2 = (0, w2, 0), and subnet3 = (0, 0, w3), or as subnet1 = (w1, 0, 0), subnet2 = (w1, w2, 0), and subnet3 = (w1, w2, w3). In addition, the training data 115 Can be similarly partitioned into three subsets

[0046] To train the machine learning model 102, the training component 120 may use To train subnet1, such as by calculating relative to the data and subnet1 (for example, ) and use the loss to update the parameter w1. Similarly, the training component 120 can use and / or To train subnet2, such as by calculating relative to the data and / or and subnet2 (for example, ) and use the loss to update the parameter w2, and use and / or To train subent3, such as by calculating relative to the data and / or and subent3 (e.g. ) and use this loss to update parameter w3.

[0047] In this way, the cth subnetwork (or cth parameter partition) of the machine learning model 102 is trained using data from the corresponding cth block of the training data 115 (and in some aspects, the previous (1, ..., c-1)th block of the training data 115).

[0048] In the illustrated workflow 100, once each subnetwork of the machine learning model 102 is so trained, the training component 120 can generate a marginal likelihood estimate or approximation for the model by evaluating the performance of each given partition / subnetwork using data from subsequent partitions of the training data 115 (that were not seen during training or were not used to refine the weights of the given subnetwork). For example, if the data partition is used To train a subnet, you can use data partitioning The loss of the subnetwork (e.g., test log likelihood) is calculated to estimate the marginal likelihood. Continuing with the above example, the training component 120 can use the data (For example, ) generates the first loss for the first subnet (only in the data training), using data (For example, ) generates a second loss for the second subnet (only in the data and / or training), etc.

[0049] More generally, the approximate marginal likelihood of the machine learning model 102 can be generated using the following Equation 1, where C is the number of partitions / subsets of the training data 115 and the machine learning model 102, is the i-th partition of the training data 115, is the aggregation of data from the first partition of training data 115 to the (i-1)th partition of training data 115, w is a parameter of machine learning model 102, It is based on data The approximate posterior distribution on the parameter w conditional on a subset of is the expectation with respect to a sample of parameter w drawn from the probability distribution q(w), and It is the training data 115 The probability of an example being in the subset, given the model parameters w.

[0050]

[0051] In some ways, the approximate posterior distribution By using data A point estimate of the weight w obtained by training a subnetwork of the machine learning model 102 on a subset of .

[0052] As discussed above, the approximate marginal likelihood can then be used (e.g., by the training component 120) to generate, update, or refine one or more hyperparameters 125 for the machine learning model 102. These updated hyperparameters 125 can then be used to further train the machine learning model 102 (e.g., during subsequent rounds or epochs of training), thereby significantly improving model performance.

[0053] Although not included in the illustrated example, in some aspects, once the machine learning model 102 is trained (e.g., after one or more rounds or one or more epochs of training), the entire model can then be used for runtime inference, and the hyperparameters 125 can be discarded, stored, or otherwise not used. Similarly, partition designations can be deleted, removed, or ignored. For example, a machine learning model 102 including nodes 105 and edges 110 (e.g., all parameters from all partitions of the model) can be deployed to generate output predictions or inferences based on input data during runtime. Typically, the machine learning model 102 can be deployed for inference on the same system that performed the training and / or one or more other systems.

[0054] Example workflow for hyperparameter optimization using partitioned machine learning models

[0055] Figure 2A , Figure 2B and Figure 2C Depicts an example workflow for performing hyperparameter optimization using a partitioned machine learning model. In some aspects, Figure 2A , Figure 2B and Figure 2C The workflows 200A, 200B and 200C depict Figure 1 Additional details of the workflow 100.

[0056] like Figure 2AAs illustrated, a first set of training data 215A is used to train a first subnetwork 202A of a machine learning model, where subnetwork 202A includes a set of nodes 205 and edges 210A. For example, as discussed above, training component 120 can process each example in training data 215A to generate a corresponding inference, compare the inference with the corresponding ground truth, and refine or update the parameters of edge 210A in subnetwork 202A based on the difference or loss. For example, as discussed above, the first data partition A first subnetwork or set of weights w1 can be used to train.

[0057] In some aspects, when training subnetwork 202A, training component 120 can ignore any parameters (e.g., edges or weights) that are not included in subnetwork 202A. That is, training component 120 can process training data 215A during a forward pass as if the non-included edges did not exist, can use a value of zero for the weights of the non-included edges, etc. More generally, when training any given partition, training component 120 can exclude, ignore, refrain from processing, or use a fixed value of zero to represent any edges (or other parameters) that are not included in the partition.

[0058] In at least one aspect, rather than ignoring or deleting such edges or parameters when training subnet 202A, training component 120 can instead retain these edges / parameters and use the values ​​that were used to initialize the non-included edges. For example, if the parameters were initialized to random values, training component 120 can use these random values ​​for any non-included edges during the forward pass. In some aspects, this can improve training stability.

[0059] After being trained, as illustrated, the second set of training data 215B is then used in conjunction with the trained subnet 202A to generate a first loss term 225A (e.g., a test log-likelihood computed by processing the training data 215B using the subnet 202A). Notably, the training data 215B does not overlap with the training data 215A. That is, the data used to generate the loss term 225A for the first subnet 202A (which is used to generate the approximate marginal likelihood of the model) does not include any data used to train or refine the parameters of the subnet 202A. For example, as discussed above, the second data partition The subnet / partition that can be used to evaluate weight w1.

[0060] like Figure 2BAs illustrated, the second set of training data 215B is also used to train a second subnetwork 202B of the machine learning model, where subnetwork 202B includes a set of nodes 205 and a set of edges 210B. For example, as discussed above, training component 120 can process each example in training data 215B to generate a corresponding inference, compare the inference with the corresponding ground truth, and refine or update parameters of some or all edges 210B in subnetwork 202B based on the difference or loss. For example, as discussed above, the second data partition A first subnetwork or set of weights w2 can be used to train.

[0061] In the illustrated example, subnet 202B includes a first set of parameters w1 (corresponding to first subnet 202A) as indicated using a solid line, and a second set of parameters w2 as indicated using a dashed line. In some aspects, when training subnet 202B, training component 120 can use training data 215B to refine second set of parameters w2, leaving first set w1 unchanged.

[0062] In addition, although the illustrated example depicts the use of training data 215B to train subnet 202B, in some aspects, training component 120 may also use the previous example. For example, second subnet 202B may be further trained based on training data 215A used to train first subnet 202A.

[0063] After such training, as illustrated, the third set of training data 215C is then used in conjunction with the trained subnet 202B to generate a second loss term 225B (e.g., a test log-likelihood computed by processing the training data 215C using the subnet 202B). Notably, as discussed above, the training data 215C does not overlap with the training data 215B and / or the training data 215A. That is, the data used to generate the loss term 225B for the second subnet 202B (which is used to generate the approximate marginal likelihood of the model) does not include any data used to train or refine the parameters of the subnet 202B. For example, as discussed above, the third data partition The subnet / partition that can be used to evaluate weight w2.

[0064] In some aspects, loss term 225B may be aggregated (eg, using a summation) with loss term 225A to generate an estimate or approximate marginal likelihood of the aggregate model (including both subnetworks 202A and 202B).

[0065] like Figure 2CAs shown, the third set of training data 215C is also used to train a third subnetwork 202C of the machine learning model (which can correspond to the entire model), where subnetwork 202C includes a set of nodes 205 and a set of edges 210C. For example, as discussed above, training component 120 can process each example in training data 215C to generate a corresponding inference, compare the inference with the corresponding ground truth, and refine or update parameters of some or all edges 210C in subnetwork 202C based on the difference or loss. For example, as discussed above, the third data partition A third subnetwork or set of weights w3 can be used to train.

[0066] In the illustrated example, subnet 202C includes a first set of parameters w1 (corresponding to first subnet 202A), as indicated using a solid line, a second set of parameters w2 (corresponding to second subnet 202B), as indicated using a dashed line, and a third set of parameters w3, as indicated using a dotted line. In some aspects, when training subnet 202C, training component 120 can use training data 215C to refine third set of parameters w3, leaving first set w1 and second set w2 unchanged.

[0067] In addition, although the illustrated example depicts the use of training data 215C to train subnet 202C, in some aspects, training component 120 may also use the previous examples in addition. For example, third subnet 202C may be further trained based on training data 215A (used to train first subnet 202A) and / or training data 215B (used to train second subnet 202B).

[0068] In the illustrated example, after such training, subnet 202C (or an aggregation of subnets in the model) can be used for inference (if training is complete). However, since there is no further partitioning of the remaining training data (e.g., there are three sets of training data in the illustrated workflow), the final subnet 202C cannot be used to generate an approximate marginal likelihood. In a similar manner, although an approximate marginal likelihood is generated based on the second and third sets of training data 215B and 215C, the first set of training data 215A is used for training but not for generating an approximate marginal likelihood (e.g., because there is no 0th subnet and all subnets in the model may have been trained based on training data 215A).

[0069] As discussed above, the individual loss terms 225A and 225B from one or more subnetworks 202 can then be combined (e.g., summed) to generate an approximate marginal likelihood, and the approximate marginal likelihood can then be used to refine one or more hyperparameters of the model (e.g., variables used to control the training process). These updated hyperparameters (and the updated parameters themselves) can then be used in subsequent rounds of training.

[0070] In this way, aspects of the present disclosure enable hyperparameters to be jointly learned along with the parameters themselves and using a single set (partitioned into subsets) of training data 215. This significantly reduces the amount of data relied upon to refine hyperparameters (e.g., eliminating the use of different validation data), improves the computational efficiency of training (e.g., reduces computational cost and latency), and generally improves the accuracy and performance of the final model (e.g., through higher prediction accuracy achieved using optimized hyperparameters).

[0071] Although not included in the illustrated workflow 200, in some aspects, the training process can be distributed across several devices, such as in a federated learning system. In one such aspect, each subnet of the model can be trained by a corresponding client (or client group) in the federated system using the client's corresponding local training data. For example, subnet 202A can be trained by a first set of clients using the client's local data, a second subnet 202B can be trained by a second set of clients using the client's local data, and so on. In order to generate a loss term 225 for generating an approximate marginal likelihood, in some aspects, each client can use the (c-1)th subnet to process its local data. That is, if a given client is training the cth subnet, the previous subnet (the (c-1)th subnet) can be used to generate the loss term.

[0072] In some aspects, to reduce network congestion and consumed bandwidth, the federated learning system may send an appropriate subset of the model to each client. For example, for clients in group c (training the cth subnet), the central server may send the subnet weights from subnet 1 to c (as opposed to sending the entire model), suppressing the sending of subnets from c+1 to C. This reduces overhead because the amount of data sent to each client is less than the entire model on average. Each client can then return updates (e.g., parameter gradients and / or hyperparameter gradients) to the central server, which can aggregate them to generate an updated model. The process can be repeated similarly until training is complete.

[0073] Example method for hyperparameter optimization using partitioned machine learning models

[0074] Figure 3 is a flowchart depicting an example method 300 for performing hyperparameter optimization using a machine learning model. In some aspects, the method 300 is performed by a training component, such as Figure 1 , Figure 2A , Figure 2B and / or Figure 2C The training component 120.

[0075] At block 305, the training component identifies two or more subsets of training data (e.g., Figure 1In some aspects, as discussed above, identifying subsets of the training data can include actively partitioning or dividing the training data into subsets (e.g., where the training data is a single corpus or repository). In some aspects, as discussed above, identifying the training data can additionally or alternatively include identifying or determining predefined or pre-created partitions or subsets, such as where the data is partitioned by another system, by a user, or inherently / structurally partitioned (e.g., in a federated learning system where each client has its own local data, these different sets of data are already partitioned).

[0076] In some aspects, the number of subsets used may vary depending on the particular implementation. For example, a user or other system may specify the number of subsets that should be used, and the training component may partition the training data into the indicated number of subsets. In general, any number of partitions may be used.

[0077] In some aspects, the technique for partitioning the training data may vary depending on the particular implementation. For example, in at least one aspect, the data is partitioned using a random or pseudo-random process such that the data within each subset is random. Similarly, the size of each subset may vary depending on the particular implementation. For example, the training component may partition the data equally (such that each subset is of equal or approximately equal size), or may partition the data unequally (such that some subsets are larger than other subsets).

[0078] At block 310, the training component identifies two or more model partitions (e.g., subnets) from the machine learning model being trained. As discussed above, each model partition typically corresponds to a subset of trainable parameters from the model. For example, in the case of a neural network, each model partition can be a subnet (e.g., a subset of weights), where the partitions collectively comprise the entire network.

[0079] In some aspects, as discussed above, identifying partitions of the model may include actively partitioning or dividing model parameters into subsets. In some aspects, as discussed above, identifying partitions may additionally or alternatively include identifying or determining predefined or pre-created partitions or subsets of model parameters. In one aspect, as discussed above, the number of model partitions may generally equal the number of subsets of training data.

[0080] In some aspects, the techniques used to partition the model parameters may vary depending on the particular implementation. For example, in at least one aspect, the parameters are partitioned using a random or pseudo-random process such that the particular parameters within each partition are random. Similarly, the size of each partition may vary depending on the particular implementation. For example, the training component may partition the model equally (such that each partition has equal or approximately equal size), or may partition the model unequally (such that some subnets are larger than other subnets).

[0081] In some aspects, as discussed above, each subnet includes parameters from a corresponding partition of the model. For example, if the model is partitioned into parameters {w1, w2, ..., w c}, then each subnet may correspond to parameters from one such subset. In at least one aspect, as discussed above, each subnet may optionally include parameters from one or more previous subsets. For example, the i-th subnet may include parameters {w1, w2, ..., w i-1 ,w i}.

[0082] At box 315, the training component selects one of the model partitions. In general, the training component can use any suitable technique or method to select the model partition, because all model partitions will be processed using method 300. In at least one aspect, the training component selects the partitions sequentially. For example, the training component can first select the first partition (e.g., a subnet with a single set of weights w1), then select the second partition (e.g., including weights w2 and weights w1), and so on. In this way, the training component can train the parameters of a given subnet and use these trained parameters to form part of a subsequent subnet for the current round and / or future rounds of training. In other aspects, training in a given round can be used to refine weights / parameters from a previous round (as opposed to using updated weights in an earlier subnet when refining weights in a subsequent subnet).

[0083] Although the illustrated examples depict a sequential process (selecting each partition in turn) for conceptual clarity, in some aspects, some or all partitions may be selected and processed in parallel. For example, the training component may train non-overlapping partitions or subnets in parallel (while sequentially training overlapping subnets based on subnet dependencies), or may use previous parameters (from a previous round of training) to perform a current round of training on all subnets.

[0084] At block 320, the training component trains the selected model partition based on the corresponding subset of training examples (or based on the corresponding subset and any previous subset used to train previous model partitions). For example, as discussed above, the training component may use the training data to train the cth partition, and can (optionally) use the training data To further train the cth partition. Figure 4 An example of training the selected partition is discussed in more detail.

[0085] At block 325, the training component determines whether there is at least one additional model partition that remains to be trained. If so, the method 300 returns to block 315. If not (e.g., if all parameters of the model have been trained or refined for the current epoch or training round), the method 300 continues to block 330.

[0086] At block 330, the training component generates an approximation or estimated marginal likelihood of the machine learning model based on the partitions. For example, as discussed above, the training component can generate a test loss (e.g., a test log likelihood) for each partition using the training data used to train subsequent partitions (e.g., using the data To generate the test loss for the cth partition). By aggregating these test losses for one or more of the partitions, the training component can efficiently approximate the marginal likelihood of the model. Figure 5 One example technique for generating approximate marginal likelihoods is discussed in more detail.

[0087] At block 335, the training component can then refine the hyperparameters of the model based on the approximate marginal likelihood. In general, the specific techniques or operations used to refine the hyperparameters can vary depending on the specific implementation and the specific hyperparameters being refined.

[0088] For example, for hyperparameters that define a mask on the input data (e.g., where a parameterized random mask, such as a mask defined using a Bernoulli distribution or a continuous relaxation of a Bernoulli distribution, is applied to the input data), the system can use marginal likelihood estimates to refine the parameters of that distribution.

[0089] As another example, for hyperparameters related to data augmentation (e.g., rotation on an input image), a differentiable affine augmentation operation (parameterized using hyperparameters) can be used to generate augmented input data, and marginal likelihood estimates can be used to refine the augmentation parameters.

[0090] At block 340, the training component determines whether training is complete (e.g., whether one or more termination criteria are met). In various aspects, termination criteria may vary depending on the particular implementation and may include variables such as a maximum number of training iterations, a minimum model accuracy, etc. If training is not complete, the method 300 returns to block 315 to begin the next iteration. If training is complete, the method 300 continues to block 345.

[0091] At box 345, once the machine learning model is trained, the training component deploys the model for inference. In some aspects, the training hyperparameters may be discarded, stored, or otherwise not used because they are not used during inference. In one aspect, deploying the trained model includes deploying the entire parameter set, regardless of the parameter partitions used during training. That is, the partition labels may not be used or referenced during inference, although these labels are relevant during training. The deployed model can then be used to generate output predictions or inferences based on input data during runtime. Typically, deploying the model may include locally instantiating the model for inference on the same system that performs training and / or one or more other systems.

[0092] Example method for training machine learning model partitions for hyperparameter optimization

[0093] Figure 4 4 is a flowchart depicting an example method 400 for training a machine learning model partition for hyperparameter optimization. In some aspects, the method 400 is performed by a training component, such as Figure 1 , Figure 2A , Figure 2B and / or Figure 2C In one aspect, the method 400 is directed to Figure 3 Block 320 provides additional details.

[0094] At block 405, the training component identifies the corresponding training data set for the model partition currently being trained. As discussed above, in one aspect, there is a one-to-one correspondence between the model partitions and the training data subsets, such that each subset corresponds to a single partition and each partition corresponds to a single training subset. For example, when training the cth partition, the training component may identify the cth subset of training examples.

[0095] At block 410, the training component trains the selected partition based on the identified subset of training data. In general, the specific operations used to train the partition may vary depending on the specific implementation and model architecture. For example, to train a neural network partition (e.g., a subnet), the training component may process training examples (from the subset of examples) using the subnet (e.g., using a subset of edges or parameters included in the subnet) to generate output inferences. The output inferences may be processed along with the ground truth values ​​of the examples to generate a loss, which may then be used to generate parameter gradients to refine the parameters of the subnet (e.g., using batch gradient descent or stochastic gradient descent).

[0096] As discussed above, in some aspects, training an identified model partition may correspond to updating a single subset of model parameters even if the partition includes multiple subsets. For example, assume that a first subnet includes a first set of weights, and a second subnet includes both the first set of weights and a second set of weights. In one aspect, training the second partition may include refining only the second set of weights (where the first set of weights is refined while the first partition is trained based on another set of training data).

[0097] At block 415, the training component may optionally train the selected partition based on one or more previous subsets of the training data, if any. For example, for the c-th model partition, the training component may use data from a previous subset (e.g., ) to refine the partition parameters of the selected partition.

[0098] In this way, the training component can train each partition of the model using a corresponding subset (or multiple subsets) of the training data, thereby allowing the marginal likelihood of the entire model to be determined by using each given partition to process unseen training data (e.g., a subset of parameters that were not used to train the given partition. This enables efficient and accurate hyperparameter optimization to be performed, thereby improving model accuracy without incurring significant overhead.

[0099] Example method for approximating marginal likelihood using partitioned machine learning models to achieve improved hyperparameter optimization

[0100] Figure 5 is a flowchart depicting an example method 500 for approximating marginal likelihood using a partitioned machine learning model to achieve improved hyperparameter optimization. In some aspects, the method 500 is performed by a training component, such as Figure 1 , Figure 2A , Figure 2B and / or Figure 2C In one aspect, the method 500 is directed to Figure 3 Box 330 provides additional details.

[0101] At block 505, the training component selects partitions from the machine learning model being trained. In general, the training component can select model partitions using any suitable technique or method, including randomly or pseudo-randomly. As discussed in more detail below, in some aspects, the training component can select from a subset of these partitions. For example, in at least one aspect, the final model partition (which can include the final set of parameters w c and / or the entire model) are not used to generate marginal likelihoods because there is no "subsequent" data set (e.g., because the final partition can be trained on all subsets of the training data, leaving no unseen examples to evaluate marginal likelihoods).

[0102] In at least one aspect, the training component selects partitions sequentially. For example, the training component can first select a first partition (e.g., a subnet having a single set of weights w1), then select a second partition (e.g., including weights w2 and weights w1), etc. Although the illustrated example depicts a sequential process (selecting each partition in turn) for conceptual clarity, in some aspects, some or all partitions can be selected and processed in parallel to generate approximate marginal likelihoods.

[0103] At block 510, the training component identifies one or more subsequent subsets of the training data relative to the selected partition. As discussed above, if the training data is not used to train or refine the parameters of the selected subset, the data can generally be referred to as "subsequent". For example, for the c-th subnet, the training component can identify the (c+1)-th subset of the training data (e.g., training data used to train the (c+1)-th subnet).

[0104] Although some examples described herein involve evaluating each model partition using a subsequent partition of the training data, in some aspects, any partition of the training data may be used to train and / or evaluate the model partition, as long as the test set (used to generate gradients to refine hyperparameters) is not included in the training set (used to refine the parameters of the partition). For example, the training component may use all subsequent partitions of the training data to evaluate each model partition, or may use a single partition of the training data to train the model partition (as opposed to all previous sets) and use one or more previous partitions of the training data to evaluate the model partition. More generally, at box 510, the training component may identify any partitions or subsets of the training data that were not used to train or refine the parameters of the selected model partition. In some aspects, such selection may be more generally described or discussed as a cross-validation objective, rather than specifically approximating marginal likelihood.

[0105] At block 515, the training component uses the identified subsequent data subset to generate a loss term using the selected model partition. For example, the training component may generate a loss term using the partition (e.g., w 1:c ) processes the identified subset (e.g., ) to generate the loss (e.g., test log likelihood). As discussed above, the process of generating the loss may vary depending on the specific implementation and model architecture. For example, in the case of a neural network, the training component may use the selected subnet w 1:c To process the identified subset For each given example in , we generate an output inference and compare the output inference with the corresponding label of the example to generate a loss (e.g., test log-likelihood).

[0106] At block 520, the training component determines whether there is at least one additional model partition remaining. That is, the training component determines whether there is at least one partition from the partitions used to generate the approximate marginal likelihood that has not been used. For example, as discussed above, in some aspects, the training component does not use the final subnet (e.g., the total weight set) because the partition has been trained on the entire training data set and there is no unseen data to be used to approximate the marginal likelihood.

[0107] If at least one partition remains, the method 500 returns to block 505. If all partitions (that can be used to generate marginal likelihoods) have been used to generate corresponding loss terms, the method 500 continues to block 525. At block 525, the training component aggregates the loss terms generated for each partition to generate an overall approximation or estimated marginal likelihood of the model. For example, as discussed above, the training component can sum the loss terms, determine an average of the loss terms, etc.

[0108] As discussed above, this efficiently generated approximate marginal likelihood can then be used to refine one or more hyperparameters of the model. For example, the training component can use the approximate marginal likelihood to generate the gradient of each hyperparameter and update each hyperparameter accordingly.

[0109] Example methods for parameter and hyperparameter optimization using federated learning

[0110] Figure 6 is a flowchart depicting an example method 600 for parameter and hyperparameter optimization using federated learning. In some aspects, the method 600 is performed by a training component, such as Figure 1 , Figure 2A , Figure 2B and / or Figure 2C Training component 120. For example, the training component can operate on a central server or host that manages federated learning.

[0111] At block 605, the training component partitions the set of participating clients into a set of client groups. In at least one aspect, the training component partitions the clients based on a desired or defined number of model partitions and / or training data subsets. For example, if there are (or will be) C subnets in the model, the training component may partition the clients into C client groups. In some aspects, this client partitioning may be performed using any suitable technique, including random or pseudo-random. In some aspects, the training component generates the client groups uniformly and randomly (e.g., such that there are an equal number of clients in each group).

[0112] In at least one aspect, the training component can generate groups to evenly distribute training examples between the groups. That is, the training component can generate groups so that each client group can have any number of clients as long as the total number of examples associated with the clients in each group is approximately even or equal between the groups. In some aspects, if new clients are added to the federation during training, the training component can assign them to one of the pre-existing client groups using similar techniques (e.g., randomly, or in an attempt to balance the groups).

[0113] At block 610, the training component selects one of the client groups from the set created at block 605. In general, the training component can select the client group using any criteria, including randomly or pseudo-randomly, as the training component will select and process each client group during each iteration or round of federated learning. Additionally, while the illustrated example depicts sequential selection of each client group in turn, in some aspects, the training component can select and process some or all of the client groups in parallel.

[0114] At block 615, the training component sends the current model partition corresponding to the selected client group to some or all clients in the selected group. In some aspects, the training component may randomly select one or more clients in each client group and send the model partition to these selected clients for the current round.

[0115] In some aspects, the training component may send the current version of the entire machine learning model to the clients in the selected client group. In at least one aspect, the training component sends an appropriate subset of the model. In one such aspect, the training component may send a partition of the model corresponding to the selected group. For example, for the cth client group, the training component may identify and send the cth partition or subnet, which may include the parameter w c , and / or may include the previous parameter w 1:c In this way, the total bandwidth consumed can be reduced compared to conventional federated learning systems that send the entire model to all participating clients.

[0116] Upon receiving the partition, each participating client in the selected client group may then use the client's local training data to train parameters specific to the client group / partition (e.g., to w c ) performs local optimization / training. These parameter updates (e.g., updated parameters and / or parameter gradients) can then be returned to the central server.

[0117] In addition, each local client may use its local training data to generate an approximate marginal likelihood or loss term, such as by processing the client's local data using the previous partition, which may be included in the transmission at block 615. For example, for clients in group c, the client may use the client's local data to process the previous partition w 1:c-1 Generate an approximate marginal likelihood and generate hyperparameter gradients or updates based on the marginal likelihood.

[0118] At block 620, the training component receives a hyperparameter update (e.g., a gradient) from each participating client in the selected client group, the hyperparameter update being generated based on a previous partition of the model relative to the current group. For example, the training component may average, sum, or otherwise combine the hyperparameter updates provided by the clients. Additionally, at block 625, the training component receives a parameter update (e.g., a gradient) generated for the current partition corresponding to the client group from each participating client. For example, the training component may average, sum, or otherwise combine the parameter updates provided by the clients.

[0119] In one aspect, the training component can aggregate and use these updates from each client to generate an updated version of the model partition corresponding to the selected client group. If subsequent iterations of federated learning are performed, the updated version can be distributed during subsequent iterations of federated learning.

[0120] At block 630 , the training component determines whether there is at least one additional client group that has not been processed during the current iteration of federated learning. If so, the method 600 returns to block 610 . If not, the method 600 continues to block 635 .

[0121] At block 635, the training component can aggregate the hyperparameters and parameter updates received from each client group to generate a refined or updated version of the model and a refined or updated set of hyperparameters. For example, the training component can sum or average the updates from each client group to produce an overall update, which is used to update the model, as discussed above. In some aspects, the training component can then begin a new iteration of federated learning (e.g., returning to block 610).

[0122] In some aspects, subsequent rounds of training are performed by selecting a subset of clients from each group and sending the updated model to each. In at least one aspect, the training component can optionally track the history of client selections so that only those parameters that have been updated since the last communication with each client can be sent at the current iteration.

[0123] In this way, the training component can perform federated learning that enables efficient hyperparameter optimization using approximate marginal likelihood of partitioned machine learning models.

[0124] Example method for refining hyperparameters of a partitioned machine learning model

[0125] Figure 7 is a flowchart depicting an example method 700 for refining hyperparameters of a partitioned machine learning model. In some aspects, the method 700 is performed by a training component, such as Figure 1 , Figure 2A , Figure 2B and / or Figure 2C The training component 120.

[0126] At block 705, a plurality of sub-networks of a neural network are determined.

[0127] At block 710 , training of a first sub-network of a plurality of sub-networks is facilitated using a first set of training examples from a plurality of sets of training examples.

[0128] At block 715 , training of a second sub-network of the plurality of sub-networks is facilitated using a second set of training examples from the plurality of sets of training examples.

[0129] At block 720, an approximate marginal likelihood of the neural network is generated based at least in part on a first loss generated by processing the second set of training examples using the first sub-network.

[0130] At block 725 , one or more hyperparameters of the neural network are refined based on the approximate marginal likelihood.

[0131] In some aspects, determining the plurality of sub-networks includes partitioning parameters of the neural network based on a defined grouping criterion.

[0132] In some aspects, method 700 also includes partitioning the corpus of training examples into a plurality of sets of training examples.

[0133] In some aspects, the method also includes facilitating training of a third sub-network of the plurality of sub-networks using a third set of training examples from the plurality of sets of training examples; and generating an approximate marginal likelihood of the neural network further based on summing the first loss and a second loss generated by processing the third set of training examples using the second sub-network.

[0134] In some aspects, the first sub-network includes a first set of weights and the second sub-network includes the first set of weights and a second set of weights.

[0135] In some aspects, training the second sub-network includes refining only the second set of weights.

[0136] In some aspects, method 700 also includes partitioning clients in the federated learning system into multiple sets of clients based on multiple sets of training examples; sending a first subnetwork to a first client in a first set of the multiple sets of clients; and sending a second subnetwork to a second client in a second set of the multiple sets of clients.

[0137] In some aspects, method 700 also includes receiving, from each respective client in the federated learning system, a respective set of weight updates and a respective set of hyperparameter gradients for a respective subnetwork; aggregating the set of weight updates; and the aggregated set of hyperparameter gradients.

[0138] In some aspects, during training, the second sub-network is not sent to the first client.

[0139] In some respects, the approximate marginal likelihood is defined as Where: C is the number of sets of training examples, is the i-th set of training examples from the multiple sets of training examples, is from arrive The aggregation of training examples, w is the parameter of the neural network, The training example set The approximate posterior distribution on the parameter w conditional on is the expectation with respect to a sample of parameter w drawn from the probability distribution q(w), and is the set of training examples The probability of an example in , given by parameter w.

[0140] In some aspects, the plurality of subnetworks comprises C subnetworks, and approximates the posterior distribution Including in the training example set The point estimate of the parameter w obtained by training the subnetwork on .

[0141] In some aspects, method 700 also includes accessing input data for runtime inference; and generating output inferences by processing the input data using the neural network.

[0142] Example processing system for hyperparameter optimization using partitioned machine learning models

[0143] In some respects, reference Figures 1 to 7 The described workflows, techniques, and methods may be implemented on one or more devices or systems. Figure 8 Depicted are various aspects configured to implement the present disclosure including, for example, reference Figures 1 to 7 An example processing system 800 for describing techniques and methods. In one aspect, the processing system 800 may correspond to a training component, such as Figure 1 , Figure 2A, Figure 2B and / or Figure 2C Training component 120. Although depicted as a single system for conceptual clarity, in at least some aspects, as discussed above, the operations described below with respect to processing system 800 may be distributed across any number of devices.

[0144] Processing system 800 includes a central processing unit (CPU) 802, which may be a multi-core CPU in some examples. Instructions executed at CPU 802 may be loaded, for example, from a program memory associated with CPU 802, or may be loaded from a partition of memory 824.

[0145] The processing system 800 also includes additional processing components tailored for specific functions, such as a graphics processing unit (GPU) 804 , a digital signal processor (DSP) 806 , a neural processing unit (NPU) 808 , a multimedia processing unit 810 , and wireless connectivity components 812 .

[0146] An NPU, such as 808, is generally a dedicated circuit configured to implement control and arithmetic logic for executing machine learning algorithms, such as algorithms for processing artificial neural networks (ANNs), deep neural networks (DNNs), random forests (RFs), etc. An NPU is sometimes alternatively referred to as a neural signal processor (NSP), a tensor processing unit (TPU), a neural network processor (NNP), an intelligence processing unit (IPU), a vision processing unit (VPU), or a graphics processing unit.

[0147] The NPU, such as 808, is configured to accelerate the execution of common machine learning tasks, such as image classification, machine translation, object detection, and various other predictive models. In some examples, multiple NPUs may be instantiated on a single chip, such as a system on a chip (SoC), while in other examples, the NPU may be part of a dedicated neural network accelerator.

[0148] The NPU can be optimized for training or inference, or in some cases configured to balance performance between the two. For NPUs that can perform both training and inference, the two tasks can generally still be performed independently.

[0149] NPUs designed to accelerate training are generally configured to accelerate the optimization of new models, which is a highly computationally intensive operation that involves inputting an existing data set (usually labeled or tagged), iterating on the data set, and then adjusting model parameters (such as weights and biases) to improve model performance. Typically, optimization based on error predictions involves passing back through the layers of the model and determining the gradient to reduce the prediction error.

[0150] NPUs designed to accelerate inference are generally configured to operate on complete models. Such NPUs can thus be configured to input new data slices and quickly process the new data through the trained model to generate model output (e.g., inference).

[0151] In one specific implementation, the NPU 808 is part of one or more of the CPU 802 , the GPU 804 , and / or the DSP 806 .

[0152] In some examples, the wireless connectivity component 812 may include, for example, subcomponents for third generation (3G) connectivity, fourth generation (4G) connectivity (e.g., 4G LTE), fifth generation connectivity (e.g., 5G or NR), Wi-Fi connectivity, Bluetooth connectivity, and other wireless data transmission standards. The wireless connectivity component 812 is further connected to one or more antennas 814.

[0153] The processing system 800 may also include one or more sensor processing units 816 associated with any manner of sensors, one or more image signal processors (ISPs) 818 associated with any manner of image sensors, and / or a navigation component 820, which may include a satellite-based positioning system component (e.g., GPS or GLONASS) and an inertial positioning system component.

[0154] The processing system 800 may also include one or more input and / or output devices 822, such as a screen, a touch-sensitive surface (including a touch-sensitive display), physical buttons, speakers, microphones, and the like.

[0155] In some examples, one or more of the processors of processing system 800 may be based on the ARM or RISC-V instruction set.

[0156] The processing system 800 also includes a memory 824, which represents one or more static and / or dynamic memories, such as dynamic random access memory, flash-based static memory, etc. In this example, the memory 824 includes computer-executable components that can be executed by one or more of the aforementioned processors of the processing system 800.

[0157] Specifically, in this example, memory 824 includes a parameter component 824A, a hyperparameter component 824B, and an inference component 824C. Figure 8 Although depicted as discrete components in the drawings, the illustrated components (and other components not depicted) can be implemented together or separately in various aspects.

[0158] In the illustrated example, the memory 824 also includes training data 824D, model parameters 824E, and model hyperparameters 824F. The training data 824D may generally correspond to a set of training examples (e.g., Figure 1 115), as discussed above. Model parameters 824E may generally correspond to learnable or trainable parameters of one or more machine learning models, as discussed above. For example, in the case of a neural network, model parameters 824E may include edge weights. In some aspects, model parameters 824E may be partitioned into partition sets or subsets, as discussed above. Model hyperparameters 824F may generally correspond to one or more variables used to control or guide the training process of a machine learning model, such as a learning rate.

[0159] Although depicted as residing in memory 824 for conceptual clarity, in some aspects, some or all of training data 824D, model parameters 824E, and hyperparameters 824F may reside in any other suitable location. For example, in the case of a federated learning approach, training data 824D may be maintained locally by participating clients.

[0160] Processing system 800 also includes parameter circuitry 826, hyperparameter circuitry 827, and inference circuitry 828. The depicted circuitry, as well as other circuitry not depicted, may be configured to perform various aspects of the techniques described herein.

[0161] For example, parameter component 824A and parameter circuit 826 may be used to update model parameters 824E within each model partition based on the corresponding training data set 824D, as discussed above. Hyperparameter component 824B and hyperparameter circuit 827 may be used to generate approximate marginal likelihoods using training data 824D and model partitions, and to update model hyperparameters 824F based on the approximate marginal likelihoods, as discussed above. Inference component 824C and inference circuit 828 may be used to perform runtime inference using the trained model parameters 824E of the entire model, as discussed above.

[0162] Although for the sake of clarity Figure 8 Although depicted as separate components and circuits, parameter circuit 826, hyperparameter circuit 827, and inference circuit 828 may be implemented together or separately in other processing devices of processing system 800, such as within CPU 802, GPU 804, DSP 806, NPU 808, etc.

[0163] Generally, the processing system 800 and / or its components may be configured to perform the methods described herein.

[0164] It is worth noting that in other aspects, aspects of the processing system 800 may be omitted, such as where the processing system 800 is a server computer, etc. For example, in other aspects, the multimedia processing unit 810, the wireless connectivity component 812, the sensor processing unit 816, the ISP 818, and / or the navigation component 820 may be omitted. Furthermore, aspects of the processing system 800 may be distributed among multiple devices.

[0165] Sample Clauses

[0166] Specific implementation examples are described in the following numbered clauses:

[0167] Clause 1: A method comprising: determining a plurality of subnetworks of a neural network; facilitating training of a first subnetwork of the plurality of subnetworks using a first set of training examples from a plurality of sets of training examples; facilitating training of a second subnetwork of the plurality of subnetworks using a second set of training examples from the plurality of sets of training examples; generating an approximate marginal likelihood of the neural network based at least in part on a first loss generated by processing the second set of training examples using the first subnetwork; and refining one or more hyperparameters of the neural network based on the approximate marginal likelihood.

[0168] Clause 2: The method of clause 1, wherein determining the plurality of sub-networks comprises partitioning parameters of the neural network based on a defined grouping criterion.

[0169] Clause 3: The method of clause 1 or 2, further comprising partitioning the corpus of training examples into said plurality of sets of training examples.

[0170] Clause 4: The method of any one of clauses 1 to 3, further comprising: using a third set of training examples from the multiple sets of training examples to facilitate training of a third subnetwork of the multiple subnetworks; and further generating the approximate marginal likelihood of the neural network based on summing the first loss and a second loss generated by processing the third set of training examples using the second subnetwork.

[0171] Clause 5: A method according to any one of clauses 1 to 4, wherein: the first sub-network comprises a first set of weights, and the second sub-network comprises the first set of weights and a second set of weights.

[0172] Clause 6: A method according to any of clauses 5, wherein training the second sub-network comprises refining only the second set of weights.

[0173] Clause 7: The method according to any one of clauses 1 to 6 further includes: partitioning the clients in the federated learning system into multiple sets of clients based on the multiple sets of training examples; sending the first subnetwork to a first client in a first set of the multiple sets of clients; and sending the second subnetwork to a second client in a second set of the multiple sets of clients.

[0174] Clause 8: The method according to any one of clauses 1 to 7 further includes: receiving a corresponding set of weight updates for a corresponding sub-network and a corresponding set of hyperparameter gradients from each corresponding client in the federated learning system; aggregating the weight update set; and aggregating the hyperparameter gradient set.

[0175] Clause 9: The method of any of clauses 1 to 8, wherein during training, the second sub-network is not sent to the first client.

[0176] Clause 10: A method according to any one of clauses 1 to 9, wherein the approximate marginal likelihood is defined as Where: C is the number of said multiple sets of training examples, is the i-th set of training examples from the plurality of sets of training examples, is from arrive is the aggregation of the training example set, w is the parameter of the neural network, The training example set is the approximate posterior distribution on the parameter w conditional on, is the expectation with respect to a sample of parameter w drawn from the probability distribution q(w), and is the set of training examples The probability of an example in , given the parameter w.

[0177] Clause 11: A method according to any one of clauses 1 to 10, wherein: the plurality of sub-networks comprises C sub-networks, and the approximate posterior distribution Including in the training example set A point estimate of the parameter w obtained by training the subnetwork.

[0178] Clause 12: A method according to any one of clauses 1 to 11, further comprising: accessing input data for runtime inference; and generating output inferences by processing the input data using the neural network.

[0179] Clause 13: A processing system, the processing system comprising: a memory, the memory comprising computer-executable instructions; and one or more processors, the one or more processors configured to execute the computer-executable instructions and cause the processing system to perform a method according to any one of clauses 1 to 12.

[0180] Clause 14: A processing system comprising means for performing the method of any one of clauses 1 to 12.

[0181] Clause 15: A non-transitory computer readable medium comprising computer executable instructions which, when executed by one or more processors of a processing system, cause the processing system to perform the method of any one of clauses 1 to 12.

[0182] Clause 16: A computer program product embodied on a computer readable storage medium, the computer program product comprising: code for performing the method according to any one of clauses 1 to 12.

[0183] Additional considerations

[0184] The foregoing description is provided to enable any person skilled in the art to practice the various aspects described herein. The examples discussed herein are not limited to the scope, applicability or aspects set forth in the claims. Various modifications to these aspects will be apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects. For example, without departing from the scope of the present disclosure, the functions and arrangements of the elements discussed may be changed. Various examples may omit, replace or add various procedures or components as appropriate. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted or combined. In addition, the features described for some examples may be combined in some other examples. For example, any number of aspects set forth herein may be used to implement a device or practice method. In addition, the scope of the present disclosure is intended to cover such devices or methods practiced using other structures, functionality or structures and functionality that are supplementary or alternative to the various aspects of the present disclosure set forth herein. It should be understood that any aspect of the disclosure disclosed herein may be embodied by one or more elements of the present invention.

[0185] As used herein, the word “exemplary” means “serving as an example, instance, or illustration.” Any aspect described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other aspects.

[0186] As used herein, a phrase referring to "at least one of" a list of items refers to any combination of those items (including single members). For example, "at least one of a, b, or c" is intended to cover a, b, c, ab, ac, bc, and abc, as well as any combination with multiple identical elements (e.g., aa, aaa, aab, aac, abb, acc, bb, bbb, bbc, cc, and ccc, or any other ordering of a, b, and c).

[0187] As used herein, the term "determining" encompasses a wide variety of actions. For example, "determining" may include calculating, computing, processing, deriving, investigating, searching (e.g., searching in a table, a database, or another data structure), ascertaining, and the like. Additionally, "determining" may include receiving (e.g., receiving information), accessing (e.g., accessing data in a memory), and the like. Furthermore, "determining" may include resolving, selecting, choosing, establishing, and the like.

[0188] The method disclosed herein includes one or more steps or actions for implementing the method. The steps and / or actions of the method can be interchangeable with each other without departing from the scope of the claims. In other words, unless a specific order of steps or actions is specified, the order and / or use of specific steps and / or actions can be modified without departing from the scope of the claims. In addition, the various operations of the method described above can be performed by any appropriate component capable of performing the corresponding function. The component may include various hardware and / or software components and / or modules, including but not limited to circuits, application specific integrated circuits (ASICs) or processors. Typically, in the case of operations illustrated in the accompanying drawings, those operations may have corresponding corresponding components with similar numbers plus functional components.

[0189] The following claims are not intended to be limited to the various aspects shown herein, but should be given the full scope consistent with the language of the claims. Within the claims, unless otherwise specified, the reference to the singular element is not intended to mean "one and only one", but "one or more". Unless otherwise specified, the term "some" refers to one or more. No claim element should be interpreted according to the provisions of 35 U.S.C. § 112 (f), unless the phrase "part for..." is used to expressly record the element, or in the case of a method claim, the phrase "step for..." is used to record the element. All structural and functional equivalents of the elements of the various aspects described throughout the present disclosure that are known or will be known later to those of ordinary skill in the art are expressly incorporated herein by reference and are intended to be covered by the claims. In addition, nothing disclosed herein is intended to be dedicated to the public, regardless of whether such disclosure is explicitly recorded in the claims.

Claims

1. A computer-implemented method comprising: Identify multiple sub-networks of a neural network; facilitating training of a first subnetwork of the plurality of subnetworks using a first set of training examples from the plurality of sets of training examples; facilitating training of a second subnetwork of the plurality of subnetworks using a second set of training examples from the plurality of sets of training examples; generating an approximate marginal likelihood of the neural network based at least in part on a first loss generated by processing the second set of training examples using the first subnetwork; as well as One or more hyperparameters of the neural network are refined based on the approximate marginal likelihood.

2. The computer-implemented method of claim 1 , wherein determining the plurality of sub-networks comprises partitioning parameters of the neural network based on a defined grouping criterion. 3 . The computer-implemented method of claim 1 , further comprising partitioning a corpus of training examples into the plurality of sets of training examples.

4. The computer-implemented method of claim 1 , further comprising: facilitating training of a third subnetwork of the plurality of subnetworks using a third set of training examples from the plurality of sets of training examples; as well as The approximate marginal likelihood of the neural network is further generated based on summing the first loss and a second loss generated by processing the third set of training examples using the second sub-network.

5. The computer-implemented method of claim 1 , wherein: The first subnetwork includes a first set of weights, and The second sub-network includes the first set of weights and a second set of weights.

6. The computer-implemented method of claim 5, wherein training the second subnetwork comprises refining only the second set of weights.

7. The computer-implemented method of claim 1 , further comprising: partitioning clients in the federated learning system into a plurality of sets of clients based on the plurality of sets of training examples; sending the first sub-network to a first client in a first set of the plurality of sets of clients; as well as The second subnetwork is sent to a second client in a second set of the plurality of sets of clients.

8. The computer-implemented method of claim 7, further comprising: receiving, from each respective client in the federated learning system, a respective set of weight updates for a respective subnetwork and a respective set of hyperparameter gradients; Aggregating the weight update set; as well as Aggregate the set of hyperparameter gradients.

9. The computer-implemented method of claim 7, wherein during training, the second sub-network is not sent to the first client.

10. The computer-implemented method of claim 1, wherein the approximate marginal likelihood is defined as in: C is the number of the plurality of sets of training examples, is the i-th set of training examples from the plurality of sets of training examples, is from arrive Aggregation of training examples, w is the parameter of the neural network, The training example set is the approximate posterior distribution on the parameter w conditional on, is the expectation with respect to a sample of parameter w drawn from the probability distribution q(w), and is the set of training examples The probability of an example in , given the parameter w.

11. The computer-implemented method of claim 10, wherein: The plurality of sub-networks include C sub-networks, and The approximate posterior distribution Including in the training example set A point estimate of the parameter w obtained by training the subnetwork.

12. The computer-implemented method of claim 1 , further comprising: Access input data for runtime inference; as well as Output inferences are generated by processing the input data using the neural network.

13. A processing system comprising: a memory including computer executable instructions; and one or more processors configured to execute the computer-executable instructions and cause the processing system to perform operations including: Identify multiple sub-networks of a neural network; facilitating training of a first subnetwork of the plurality of subnetworks using a first set of training examples from the plurality of sets of training examples; facilitating training of a second subnetwork of the plurality of subnetworks using a second set of training examples from the plurality of sets of training examples; generating an approximate marginal likelihood of the neural network based at least in part on a first loss generated by processing the second set of training examples using the first subnetwork; as well as One or more hyperparameters of the neural network are refined based on the approximate marginal likelihood.

14. The processing system of claim 13, wherein determining the plurality of sub-networks comprises partitioning parameters of the neural network based on a defined grouping criterion.

15. The processing system of claim 13, the operations further comprising partitioning a corpus of training examples into the plurality of sets of training examples.

16. The processing system of claim 13, the operations further comprising: facilitating training of a third subnetwork of the plurality of subnetworks using a third set of training examples from the plurality of sets of training examples; as well as The approximate marginal likelihood of the neural network is further generated based on summing the first loss and a second loss generated by processing the third set of training examples using the second sub-network.

17. The processing system of claim 13, wherein: The first subnetwork includes a first set of weights, and The second sub-network includes the first set of weights and a second set of weights.

18. The processing system of claim 17, wherein training the second sub-network comprises refining only the second set of weights.

19. The processing system of claim 13, the operations further comprising: partitioning clients in the federated learning system into a plurality of sets of clients based on the plurality of sets of training examples; sending the first sub-network to a first client in a first set of the plurality of sets of clients; as well as The second subnetwork is sent to a second client in a second set of the plurality of sets of clients.

20. The processing system of claim 19, the operations further comprising: receiving, from each respective client in the federated learning system, a respective set of weight updates for a respective subnetwork and a respective set of hyperparameter gradients; Aggregating the weight update set; as well as Aggregate the set of hyperparameter gradients.

21. The processing system of claim 19, wherein during training, the second sub-network is not sent to the first client.

22. The processing system of claim 13, wherein the approximate marginal likelihood is defined as in: C is the number of the plurality of sets of training examples, is the i-th set of training examples from the plurality of sets of training examples, is from arrive Aggregation of training examples, w is the parameter of the neural network, The training example set is the approximate posterior distribution on the parameter w conditional on, is the expectation with respect to a sample of parameter w drawn from the probability distribution q(w), and is the set of training examples The probability of the example in , given the parameter w.

23. The processing system of claim 22, wherein: The plurality of sub-networks include C sub-networks, and The approximate posterior distribution Including in the training example set A point estimate of the parameter w obtained by training the subnetwork.

24. The processing system of claim 13, further comprising: Access input data for runtime inference; as well as Output inferences are generated by processing the input data using the neural network.

25. A non-transitory computer readable medium comprising computer executable instructions that, when executed by one or more processors of a processing system, cause the processing system to perform operations comprising: Identify multiple sub-networks of a neural network; facilitating training of a first subnetwork of the plurality of subnetworks using a first set of training examples from the plurality of sets of training examples; facilitating training of a second subnetwork of the plurality of subnetworks using a second set of training examples from the plurality of sets of training examples; generating an approximate marginal likelihood of the neural network based at least in part on a first loss generated by processing the second set of training examples using the first subnetwork; as well as One or more hyperparameters of the neural network are refined based on the approximate marginal likelihood.

26. The non-transitory computer readable medium of claim 25, the operations further comprising: facilitating training of a third subnetwork of the plurality of subnetworks using a third set of training examples from the plurality of sets of training examples; as well as The approximate marginal likelihood of the neural network is further generated based on summing the first loss and a second loss generated by processing the third set of training examples using the second sub-network.

27. The non-transitory computer readable medium of claim 25, wherein: The first subnetwork includes a first set of weights, The second subnetwork includes the first set of weights and a second set of weights, and Training the second sub-network includes refining only the second set of weights.

28. The non-transitory computer readable medium of claim 25, the operations further comprising: partitioning clients in the federated learning system into a plurality of sets of clients based on the plurality of sets of training examples; sending the first sub-network to a first client in a first set of the plurality of sets of clients; as well as The second sub-network is sent to a second client in a second group in the plurality of sets of clients, wherein the second sub-network is not sent to the first client during training.

29. The non-transitory computer readable medium of claim 25, further comprising: Access input data for runtime inference; as well as Output inferences are generated by processing the input data using the neural network.

30. A processing system comprising: A component for determining a plurality of sub-networks of a neural network; means for facilitating training of a first subnetwork of the plurality of subnetworks using a first set of training examples from the plurality of sets of training examples; means for facilitating training of a second subnetwork of the plurality of subnetworks using a second set of training examples from the plurality of sets of training examples; means for generating an approximate marginal likelihood of the neural network based at least in part on a first loss generated by processing the second set of training examples using the first subnetwork; as well as Means for refining one or more hyperparameters of the neural network based on the approximate marginal likelihood.

Images