Synthetic classification data set through optimal transmission interpolation

Abstract

Apparatuses, systems, and methods for generating a synthetic data set are generally discussed herein. A method includes: obtaining a first training tag data set; obtaining a second training mark data set; determining an optimal transmission (OT) map from the target tagged data set to the first training tagged data set; determining an OT mapping from the target tagged data set to a second training tagged data set; on the basis of OT mapping, in a generalized geodesic coil formed by the first training mark data set and the second training mark data set in the distribution space, points close to the target mark data set in the distribution space are identified; and generating a synthetic markup ML dataset by combining the first training markup dataset and the second training markup dataset based on the distance between the probability distribution representation of the first training markup dataset and the second training markup dataset and the point in the distribution space.

Description

Background Art

[0001] Recent advances in machine learning (ML) are characterized by the rapid adoption of large pre-trained models as basic building blocks. These models are usually pre-trained on a large amount of general pre-training data and then adapted (e.g., fine-tuned) to a specific task of interest. Such pre-training datasets usually come from multiple heterogeneous data sources (e.g., generated from different domains or sources). Traditionally, all available datasets are fully used during pre-training, such as by pooling the datasets together into a single dataset (when they all share the same set of labels) or by using all datasets sequentially one by one for training. However, these strategies have important disadvantages.

[0002] Training on the union of multiple datasets may be costly or too time-consuming, and may even be disadvantageous. Indeed, a growing body of research shows evidence that removing pre-training data sometimes helps transfer performance. On the other hand, sequential learning (e.g., consuming datasets one by one) is prone to catastrophic forgetting, as information from earlier datasets gradually disappears as the model is trained on new datasets. The foregoing suggests that it is advantageous to train on only some subset of the pre-training dataset, but it is not clear how to choose that subset. However, when it is known in advance which target dataset the model will be used on, the answer is much easier: intuitively, only train on those datasets that are relevant to the target dataset (e.g., those that are most similar to the target dataset). Indeed, recent work has shown that selecting pre-training datasets based on the distance to the target is a successful strategy. Summary of the invention

[0003] The present invention summary is provided to introduce various aspects of the embodiment in a simplified form, and the embodiment will be further explained in the detailed description below. The present invention summary is not intended to identify necessary or required features of the claimed subject matter, and the combination and order of the elements listed in the present invention summary are not intended to provide limitations on the elements of the claimed subject matter.

[0004] Embodiments may generate synthetic datasets useful for training, fine-tuning, or testing machine learning (ML) classifiers. Embodiments may determine corresponding optimal transport (OT) mappings from a labeled dataset to a plurality of training labeled datasets. The OT mapping may be used to form a geodesic hull. Points in the geodesic hull are selected to represent a target dataset. A first training labeled dataset and a second training labeled dataset are then combined using the distance from the point to the training labeled dataset. The combined first training labeled dataset and the second training labeled dataset form a synthetic dataset.

[0005] Generating a synthetic data set may include obtaining a first training labeled data set. Generating a synthetic data set may include obtaining a second training labeled data set. Obtaining a data set may include retrieving a data set, accessing a data set, downloading a data set from a cloud, receiving a data set from a client, and the like. Generating a synthetic data set may include determining an optimal transmission (OT) mapping from a target labeled data set to a first training labeled data set. Generating a synthetic data set may include determining an OT mapping from a target labeled data set to a second training labeled data set. Generating a synthetic data set may include identifying, based on the OT mapping, points in a distribution space that are close to a target labeled data set in a generalized geodesic bag formed by the first training labeled data set and the second training labeled data set in the distribution space that are close to the target labeled data set. Generating a synthetic data set may include generating a synthetic labeled ML data set by combining the first training labeled data set and the second training labeled data set based on a probability distribution representation of the first training labeled data set and the second training labeled data set in the distribution space and a distance between points.

[0006] The target labeled dataset may include more, fewer, or different tags than tags of one or more of the first training labeled dataset and the second training labeled dataset. Combining the first training labeled dataset and the second training labeled dataset may include representing the tags of the first training labeled dataset and the second training labeled dataset as corresponding one-hot vector tags of all tags in the first training labeled dataset and the second training labeled dataset. Generating the synthetic dataset may also include using the synthetic labeled ML dataset to further train a pre-trained ML model that has been trained based on the target labeled dataset.

[0007] Determining the OT map may include performing a barycentric projection of the target labeled data set onto a geodesic bag. The barycentric projection may include projected labels of the sample data and the separately labeled data. Determining the OT map may include operating an OT neural map, the OT neural map including three classifiers: a label classifier, a discriminator, and a feature classifier label. The discriminator loss of the discriminator may be independent of the label.

[0008] Identifying points in the dataset space that are close to the target labeled dataset may include determining a point in a generalized geodesic bag that is closest to the target labeled dataset. Identifying points that are close to the target labeled dataset may include operating a quadratic problem solver based on a (2, v) transfer metric. Generating the synthetic dataset may include receiving a request for a synthetic labeled ML dataset from an application before obtaining the first training labeled dataset and the second training labeled dataset. Generating the synthetic dataset may include providing the synthetic labeled ML dataset to the application in response to generating the synthetic labeled ML dataset. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1 A diagram of an embodiment of a method for generating a synthetic dataset using generalized geodesics is shown by way of example.

[0010] Figure 2 A graphical diagram illustrating an embodiment of generating a data set based on a training data set is shown by way of example.

[0011] Figure 3 Plots of dataset generalization capabilities for some public labeled datasets are shown by example.

[0012] Figure 4 A diagram of an embodiment of a system for synthetic dataset generation is shown by way of example.

[0013] Figure 5 A diagram of an embodiment of another system for synthetic dataset generation is shown by way of example.

[0014] Figure 6 A diagram illustrating another embodiment of a system for synthetic dataset generation is shown by way of example.

[0015] Figure 7 A diagram showing an embodiment in which images are organized into columns to corresponding tags is shown by way of example.

[0016] Figure 8 A diagram showing an embodiment of a method for synthetic dataset generation is shown by way of example.

[0017] Fig. 9 is an example block diagram of an environment including a system for neural network training.

[0018] Fig.10 A block diagram of an embodiment of a machine (eg, a computer system) for implementing one or more embodiments is shown by way of example. DETAILED DESCRIPTION

[0019] In the following description, reference is made to the accompanying drawings which are a part of this article, and specific embodiments that can be put into practice are shown in the accompanying drawings by way of illustration. These embodiments are described in sufficient detail to enable those skilled in the art to practice the embodiments. It should be understood that other embodiments may be utilized, and structural, logical and / or electrical changes may be made without departing from the scope of the embodiments. Therefore, the description of the following embodiments is not to be construed as being restrictive, and the scope of the embodiments is defined by the appended claims.

[0020] While recent work has shown that selecting a pre-training dataset based on the distance to the target is a successful strategy, such approaches are limited to selecting (only) among individual datasets already present in the collection.

[0021] Embodiments utilize the concept of distances between datasets based on optimal transport (OT), referred to as optimal transport distances (OTD) or optimal transport dataset distances (OTDD), which provide a space of joint distributions with meaningful metrics from which distances can be derived. Typically, embodiments may use OT interpolation along geodesics between datasets to generate new synthetic datasets between datasets in the dataset space. While there are other methods for generating unlabeled datasets (e.g., simple image interpolation), there are almost no methods that handle the properties of labeled data and are accurate. For some specific settings (such as robotics), there are methods for synthesizing data (such as generating simulated physics engines). For more typical machine learning (ML) settings (such as image classification), options for synthetic labeled data generation are very limited.

[0022] The main challenge of synthesizing labeled data is to define what it means to interpolate between discrete category labels. The embodiments extend this previous work to provide a theoretically sound, computationally feasible, and functional method for generating synthetic data. Compared with previous work, the embodiments have the following advantages: 1. Embodiments may combine samples from more than two datasets. 2. The embodiments are computationally more stable than existing work. 3. Embodiments allow for the generation of additional data points on demand with little computational cost. Existing methods require solving the entire optimization problem again and are not suitable for generating new additional data points after running the initial method. 4. The embodiment has a functional solution to the problem of how to combine discrete labels (the prior art uses heuristic algorithms).

[0023] When given access to a target dataset of interest, the synthetic dataset that is closest to the target (in terms of the metric between datasets) can be identified among all combinations of pre-trained datasets. By characterizing the datasets as datasets sampled from an underlying probability distribution, the metric between datasets can be understood as a problem generalized (from Euclidean space to probability space) to finding a point in the convex hull of a set of reference points closest to a query point. While this problem has a simple closed form solution in Euclidean space (via orthogonal projection), solving it in probability space is more challenging. An embodiment addresses this problem from the perspective of interpolation.

[0024] Formally, the combination of data sets is modeled as an interpolation between their data distributions, which is formalized by the concept of geodesics in a probability space endowed with a Wasserstein metric. Embodiments may rely on generalized geodesics, i.e., constant velocity curves connecting a pair (or more pairs) of distributions parameterized relative to a "reference" distribution (i.e., the target data set). Computing such geodesics requires access to the OT coupling or OT mapping between the reference distribution and each other reference distribution. The former can be computed very efficiently using off-the-shelf OT solvers, but is limited to generating as many samples as were originally used to solve the problem. In contrast, OT mappings allow on-demand out-of-sample mappings and can be estimated using neural OT methods. However, most existing OT methods assume an unlabeled (feature-only) distribution. However, the goal of an embodiment is to interpolate between classified (i.e., labeled) data sets. Therefore, a generalization of OT can be used for labeled data sets to compute couplings. Neural OT methods can be adapted and generalized to labeled settings to estimate OT mappings.

[0025] In summary, the embodiments provide: (i) a novel approach to generate new synthetic classification datasets from existing datasets by using geodesic interpolation, which is applicable even if they have disjoint labeled sets, (ii) two efficient methods for computing generalized geodesics, both of which may have independent research interest, and (iii) empirical validation of the method in a transfer learning setting.

[0026] Details of geodesics and OT between data sets are discussed further below. Reference will now be made to the drawings to provide further details and applications of the embodiments.

[0027] Figure 1 A diagram of an embodiment of a method 100 for generating a synthetic data set using generalized geodesics is shown by way of example. The method 100 shown includes three data sets, a training data set 110 (optional if the data distribution and label distribution are known) and a plurality of training data sets 112, 114. Although two training data sets are shown, embodiments may use any number of training data sets greater than one. The target data set 110 is a reference data set toward which the training data sets 112, 114 will be altered. The target data set 110 does not exist or has an insufficient number of samples. In this context, "insufficient" means that the model trained on the data set has a classification accuracy that is lower than the expected classification accuracy for one or more categories.

[0028] Each of the training dataset 110 and the target datasets 112, 114 includes a set of (feature, tag) pairs 111, 113, 115, respectively. This means that the target dataset 110 and the training datasets 112, 114 are classification datasets. Classification datasets are datasets used to train or test ML classifier models. Supervised ML model training is usually performed using supervised learning techniques. Each of the target dataset 110 and the training datasets 112, 114 may include the same or different number of categories (sometimes referred to as "tags"). That is, the target dataset 110 and the training datasets 112, 114 may include different categories, different numbers of categories, and even categories of different categories. The categories of categories include objects (e.g., types of animal groups, types of alphabetic characters, types of symbols, types of flora, types of flora, types of food, etc.), phonemes, types (e.g., spam or non-spam), specific people (e.g., using facial recognition), etc. Well-known training datasets include: the National Institute of Standards and Technology (MNIST) handwritten Arabic numeral database, where the categories are "zero", "one" ... "9"; the clothing MNIST database of clothing items, where the categories are "T-shirt / top", "pants", "pullover", "dress", "jacket", "sandals", "shirt", "sneakers", "bags" and "boots"; the Chinese MNIST database of handwritten Chinese numerals, where the categories are 15 Chinese numerals; the sign language MNIST database, where the categories are 24 English alphabet characters "A", "B" ... "I", "K", "L" ... "Y" (all standard English alphabet characters, except "J" and "Z"), etc. There are many other training datasets 112, 114, including private datasets. The training datasets 112, 114 include sufficient samples for training and testing a given model.

[0029] At operation 116, a first statistical distribution of each feature in the (feature, label) pairs 111, 113, and 115 (sometimes referred to as samples) may be determined. The statistical distribution may be represented by a distribution type (e.g., Gaussian distribution, uniform distribution, etc.) and corresponding parameters of the distribution type (e.g., mean, standard deviation, variance, etc.). At operation 116, a statistical distribution of each label in the (feature, label) pairs 111, 113, 115 may be determined. For overlapping labels (labels in more than one dataset), the label is determined for each dataset so that each label in each dataset has a different distribution. The distribution for each label is determined by the features of the sample mapped to the given label. Therefore, each label is represented as [the distribution of feature 1 mapped to the label, the distribution of feature 2 mapped to the label ... the distribution of feature f mapped to the label], where f is a positive integer. A certain ML model may only look at two features, so f is a positive integer greater than 1.

[0030] At operation 118, a corresponding mapping from the target data set 110 to each of the training data sets 112, 114 is estimated. Operation 118 is performed based on the distribution determined at operation 116. There are multiple ways to perform operation 118, which will be discussed in more detail below. Some ways to perform operation 118 include using a modified OTDD neural map and an OTDD centroid projection.

[0031] At operation 120, generalized geodesics of the training data sets 112, 114 are constructed. Generalized geodesics are constant velocity curves connecting a pair (or more pairs) of distributions parameterized relative to a "reference" distribution (target data set 110). Parameterization is achieved by estimating a mapping from the target data set 110 to the training data sets 112, 114, rather than estimating a mapping from the training data sets 112, 114 to the target data set 110 at operation 118. More details about generalized geodesics are provided below. The generalized geodesics connecting the training data sets 112, 114 form a finite data set space. The training data sets 112, 114 can be combined to form a data set at any point in the data set space.

[0032] At operation 122, a target point in the generalized geodesic of the training data set is identified. Operation 122 may include projecting the distribution of the target data set 110 onto a data set space formed by the generalized geodesic constructed at operation 120. The projection onto the data set space may be estimated as the point in the data set space that is closest to the target data set 110. This point may be estimated using a quadratic programming solver. More details on how to identify this are provided below.

[0033] In some cases, the target dataset 110 is not required. In this case, operation 122 may include selecting a point in the dataset space formed by the generalized geodesic constructed at operation 120. Datasets located at points equidistant from each training dataset 112, 114 can help optimally fill the dataset space, making it more likely that future target datasets will be closer to the datasets in the dataset space. However, datasets can be generated at any point in the constructed dataset space. Intuitively, training datasets that are closer to the target dataset in the dataset space will help improve the performance of the model trained based on the target dataset and the training dataset.

[0034] At operation 124, samples of the training data set 114 are combined to generate a data set at the point identified at operation 122. The training data set 114 can be combined by determining the relative distance between the point and each data set in the training data set. The samples of the training data sets 112, 114 can be combined using a distance based on the point, so the weighted contribution of samples from data sets closer to the point is greater than that of samples from data sets farther from the point. More details on operation 124 are provided below.

[0035] Figure 2 A graphical diagram of an embodiment of generating a data set based on a training data set is shown by way of example. Figure 2 In this example, Q is the target dataset 110, P is x (where x is a positive integer) represents the corresponding training data sets 112, 114, 220. The corresponding mappings 222, 224, 226 (respectively) between the target data set 110 and the training data sets 112, 114, 220 are estimated. )(For example, Figure 1 Operation 118 in the method). represents the mapping from the target dataset 110 to the training dataset 112, represents the mapping from the target dataset 110 to the training dataset 114, Represents a mapping from the target dataset 110 to the training dataset 220. The generalized geodesic 228 between the datasets is determined, and a dataset space (the space within the generalized geodesic 228) is defined. Any dataset within the dataset space can be determined as a convex combination of samples from the training datasets 112, 114, 220. The point in the dataset space that is closest to the target dataset 110 can be identified. The convex combination of the training datasets 112, 114, 220 at that point can be determined to generate a synthetic dataset The synthetic dataset 230 is the dataset that is most like the target dataset 110 in the dataset space, and is therefore the dataset that is most likely to improve the training and testing of models run on the target dataset 110 .

[0036] In some embodiments, the target dataset 110 is optional. For example, if the point corresponding to the dataset 230 is the desired dataset (e.g., it is the center point (e.g., the centroid) of the datasets 220, 112, 114), the target dataset 110 can be synthesized based on the distance between the distributions representing the datasets 220, 112, 114 in the distribution space. The weighted contribution of each of the datasets 220, 112, 114 can be inversely proportional to their distance to the point in the distribution space. Generating such a synthetic dataset helps to reduce the distance to the potential target dataset that is outside the geodesic bag formed by the geodesic 228. The dataset at the center of the datasets 220, 112, 114 can have the best generalization ability for all datasets 220, 112, 114.

[0037] In order to provide more details about the operation of the embodiment, it is helpful to first provide some baseline explanations and notations. First, distribution interpolation and a baseline distance metric using optimal transport (OT) are explained to provide some baselines for a more detailed explanation of the operation of the embodiment. A more detailed explanation of some operations is then provided, followed by some experimental results.

[0038] Regarding interpolation between distribution representations of samples using OT, consider For some Euclidean space The space of probability distributions with finite second-order moments on . For a given Finding Mappings for the Monge Form of the Optimal Transport Problem This mapping transforms μ into v with minimal cost. Formally, the goal of the problem is where the minimization is performed over all maps that push the distribution μ onto the distribution v. Although a solution to this problem may not exist, the relaxation attributed to Kantorovich is shown to have a solution. This modified version yields the 2-Wasserstein distance: The constraint set contains all couplings with edges μ and v. Such an optimal coupling is called an OT scheme. A famous result states that whenever With a density relative to the Lebesgue measure, the optimal T * exists and is unique. In this case, the Kantorovich and Monge forms agree, and their solutions are determined by π * =(Id,T * ) #μ , where Id is the identity map. Compared to other distances for distributions, the Wasserstein-2 distance enjoys many desirable geometric properties. One such property is the characterization of geodesics in probability space. Equipped with metrics When , the unique minimum geodesic between any two distributions μ0 and μ1 is completely determined by the OT plan π between them through the following relationship: This formula is called displacement interpolation. If the Monge map exists, the geodesic can also be written as This interpolation is called McCann interpolation. Such interpolation is defined only between two distributions. m}, the Wasserstein center of gravity Generalized McCann interpolation. Intuitively, the interpolation parameter a=[a1,…,a m ] determines the “mixing ratio” of each dataset in the combination, similar to a convex combination of points in Euclidean space. In particular, when a is a one-hot vector with a i =1, then That is, the centroid is the i-th distribution. Centroids have recently attracted great attention in machine learning, but they remain challenging to compute in high dimensions.

[0039] Another limitation of these interpolation concepts is that Non-convexity. In Euclidean space, given three points function where x t is the interpolated value x t =(1-t)x1+tx2, which is convex. On the contrary, in Wasserstein space, neither function is convex, nor does it guarantee that the function is convex. This lack of convexity guarantees complicates theoretical analysis, for example in gradient flow. To circumvent this problem, others have introduced methods that exploit the basis v of {μ1,…,μ m} generalized geodesic, and is defined as Among them is is from v to μ i The optimal mapping of .

[0040] Lemma 1. Function is convex along a generalized geodesic, and

[0041] Thus, unlike centroids, generalized geodesics do yield the notion of convexity properties satisfied under the Wasserstein distance, and are also easier to compute. For these reasons, in embodiments, generalized geodesics can be used for interpolation. Generalized geodesics in the form discussed so far are not suitable for application to labeled datasets.

[0042] Now we will introduce the dataset distance, of which operation 116 is a part. Consider the dataset The optimal transmission dataset distance (OTDD) measures the distance from one dataset to another dataset D. Q The distance is: It defines a reasonable measure between datasets. Here, α y ,α y′ is the class-conditional measure corresponding to P(x|y) and Q(x|y′). This distance is consistent with the transfer learning performance (i.e., Train the model and then It is strongly correlated with the accuracy achieved during fine-tuning and evaluation. Therefore, it can be used to select a pre-training dataset for a given target domain.

[0043] For simplicity, the symbol P is now used to refer to both the dataset and its underlying distribution. To avoid confusion, ν and μ refer to the distribution in feature space, i.e., the distribution in Euclidean space, and P and Q refer to the distribution in the product space of features and labels. Label

[0044] Embodiments include at least two operations: (i) estimating an optimal transmission mapping between a target data set and all training data sets (e.g., operation 118), and (ii) using the mapping to generate a convex combination of these data sets by interpolating along generalized geodesics (e.g., operation 120). For some applications, a projection of the target data set into a "convex hull" of the training data sets may be performed (e.g., at operation 122).

[0045] There are multiple ways to perform operation 118. One is to use neural OT, another is to use entropy regularized OT ("barycentric projection"). OTDD is a special case of the Wasserstein distance, so it is natural to consider an alternative Monge (mapping-based) formulation to equation (2).

[0046] The barycentric projection can be computed efficiently for entropy regularized OT using the Sinkhorn algorithm. Assume that the samples are independent and identically distributed (iid) from two distributions ν and μ: In solving the optimal coupling After that, the barycentric projection can be expressed as T B (Xv )=N v π*X μ The example extends the method to two datasets Z Q ={X Q ,Y Q},Z p ={X P ,Y P}, where additional tag data exists First, determine the optimal coupling π for OTDD(2) * , so that the token can be represented as a one-hot vector The barycentric projection can be divided into two parts, such as T B (Z Q )=[N Q π * X P ,N Q π * Y P ](3).

[0047] However, this approach has at least two limitations: it does not naturally map out-of-sample data, and it does not scale well to large datasets (due to a quadratic dependence on sample size).

[0048] OTDD Neural Mapping. Embodiments may include a framework for estimating OTDD using an OTDD neural network. Existing methods for solving Monge OT problems with general cost functions include solving the maximum-minimum dual problem By introducing additional classifiers in the mapping, the embodiment extends it to include the distribution of labels. Given two datasets P and Q, the mapping can be parameterized as in is the push-forward feature map, and is a frozen classifier pre-trained on dataset P. Note that at cost Under this condition, the Monge form of OTDD(2) is The embodiment therefore proposes to solve the maximum-minimum dual problem as follows:

[0049] The implementation details are provided below. Compared to the previous conditional Monge mapping solver, the two methods proposed here: (i) do not assume category overlap across datasets, thereby allowing mapping between datasets with different label sets; (ii) are invariant to category permutations and relabeling; (iii) do not enforce one-to-one alignment between categories (e.g., using this embodiment, samples can be mapped to dissimilar categories).

[0050] Computing datasets based on generalized geodesics requires constructing convex combinations of data points from different datasets. Given a weight vector Features can be naturally combined into But combining labels is not that simple because: (i) embodiments allow datasets with different numbers of labels, so directly adding them is not possible; (ii) embodiments do not assume that different datasets have the same set of labels, such as MNIST (digits) vs. CIFAR10 (objects). All labels can be represented in the same dimensional space by padding all entries that are not labels in a given dataset with zeros. As an example, consider three datasets with 2, 3, and 4 categories, respectively. Given a first label vector for the first dataset Second label vector for the second dataset and the third label vector for the third dataset The label vector can be embedded into The combined tag vector in Then, for a given i-th token, the entries of the other datasets are set to the zero token, so, for example and The labeling for the synthetic dataset can then be calculated as The representation is lossless and preserves the distinction of labels across datasets.

[0051] Now we turn to the projection of the dataset onto the generalized geodesic. First, operation 118 is performed to compute the Q and all other datasets using a discrete or neural OT method. OTDD mapping between * . Then, for any interpolation vector a∈Δ m , the dataset along the generalized geodesic can be identified as By using the convex combination method discussed previously, embodiments can effectively sampling.

[0052] We will now discuss how to position The dataset with the smallest distance to Q That is, the projection of Q onto the generalized geodesic. From the Euclidean perspective, assuming that in Euclidean space There are several distributions on and an additional distribution ν, Lemma 1 guarantees that there exists a minimum The only parameter a * However, positioning a * Not directly, because there is no mapping The closed-form formula for , and for all possible a To solve this problem, another transmission distance can be used, namely the (2,ν)-transmission metric.

[0053] Definition 1. The (2,ν)-transmission metric is given by: in is from ν to μ i The optimal mapping of .

[0054] When ν has density with respect to the Lebesgue measure, is a valid measure. It can be concluded that the mapping The closed form formula of .

[0055] Proposition 1.

[0056] This equation means that given the distribution {μ i},v, can be minimized by a quadratic programming solver The optimal a * The transmission metric for a dataset can be defined as in Definition 2.

[0057] Definition 2. The squared (2,Q) dataset distance is given by in in It is from Q to P i OTDD mapping.

[0058] Will Expressed as satisfying d OT The set of all probability measures P for which (P,Q)<∞ and an OTDD mapping from Q to P exists. The following results show that the (2,Q) dataset distance is a reasonable distance.

[0059] Proposition 2. yes An effective measure of .

[0060] Unfortunately, in this case, It does not have the same analytical form as before, because Brenier's theorem may not hold for the general transmission cost problem. However, we still consider And define the approximate projection Minimizer as a function It is an analogy of Proposition 1. Unlike the Wasserstein distance, Easier to compute, since it does not involve optimization, thus locating The minimizer of is computationally relatively cheap. Experimentally, it is observed that the predictable Model transferability across tasks.

[0061] Figure 2 The optimal mapping is shown to be effective in estimating the projection of a dataset onto the generalized geodesic bag of three training datasets. i Compared to the usual blending, using the mapping estimated via barycentric projection Leading to better preservation of category structure, since random coupling destroys the category structure.

[0062] Embodiments have been used to generate new pre-training datasets for few-shot learning. Given m labeled pre-training datasets {P i}, consider a small sample test dataset, where only part of the data is labeled (e.g., 5 samples per class). Assume that training resources and time are limited, so that the user can only choose one dataset to train the model, and at the same time, the user expects the model to have the best possible ability to generalize. To this end, assume that the training dataset is selected from the generalized geodesic {P a By selecting a one-hot weight vector a, P a Restore the original dataset P for some i i Otherwise, P a will be the dataset {P i}. Note that the generalization ability of the trained model is related to the distance Has a strong correlation.

[0063] Association with generalization ability. The closed form expression of (Proposition 1) provides the relationship between the reference distribution ν in Euclidean space and the distribution along the generalized geodesic The simulation (5) for the labeled datasets Q and {P} is Figure 3 Available in.

[0064] In order to study the generalization ability of the model trained on different datasets, the simplex Δ3 is discretized to obtain 36 interpolation parameters a. The interpolation parameters are used to a A 5-layer LeNet classifier is trained on . Then, all these classifiers are fine-tuned on a small-sample test dataset Q with only 20 samples per class. The same number of training iterations and fine-tuning iterations is used in all experiments. Figure 3 The second row shows the fine-tuning accuracy. Comparing the first and second rows, the accuracy and W2(P a ,Q) are highly correlated. This means that in W2(P a ,Q) minimizer datasets tend to have better generalization capabilities. Figure 3 For all heatmaps on the datasets in , the same color bar range is fixed to highlight the different effects of choosing the training dataset. For some test datasets, the choice of the training dataset can greatly affect the fine-tuning accuracy. For example, when Q is EMNIST and the training dataset is FMNIST, the fine-tuning accuracy is only about 60%, but this can be improved to about 70% by choosing an interpolated dataset that is closer to MNIST. This is reasonable because MNIST shares more similarities with EMNSIT than with FMNIST or USPS. For some test datasets such as FMNIST and KMNIST, the difference is not so obvious because all the training datasets are far away from the test dataset.

[0065] Next, a comparison between the embodiment and multiple benchmark methods on the NIST dataset is provided. In each set of experiments, one dataset is selected as the target dataset, and the remaining "NIST" datasets are training datasets. Assume that the test dataset is 5 times. To this end, 5 samples are randomly selected from each category as labeled data, and the remaining samples are regarded as unlabeled. The model was trained on 5 test datasets and fine-tuned on 5 test datasets. The OTDD mapping from the test dataset to the training dataset is approximated using either barycentric projection or neural mapping. The results are shown in the first two rows of Table 1 below. The first baseline approach is to create a synthetic dataset as the training dataset by mixing among the datasets. To this end, for each training dataset, a random sample of data with weights The convex combination is determined as described above, so this benchmark is equivalent to the embodiment with suboptimal OTDD mapping. The other two benchmarks (bottom boxes in Table 1) skip the transfer learning part and directly train the model or solve 1-NN on a small sample test dataset. In general, transfer learning can bring additional knowledge from other domains and improve the test accuracy by up to 21%. Among the methods in the first block, training on the dataset generated by the OTDD centroid projection outperforms the other datasets except the USPS dataset, where the difference is only about 2.6%.

[0066] Table 1: Pre-training on synthetic data. 5 transfer accuracies are shown (mean ± SD of 5 runs).

[0067] Figure 4A diagram of an embodiment of a system for synthetic dataset generation is shown by way of example. The system 400 shown includes a user 401 having a device 402. The device 402 may issue a request 404 to a synthetic dataset generation ML system 406. The request 404 may indicate the location of a first dataset, include a first dataset, etc. The request 404 may indicate a desired result, such as an ML model, data that meets certain criteria, etc.

[0068] Device 402 is a computing device, such as a computer (e.g., a laptop, a desktop computer, a handheld computer, a smart phone, a tablet, a phablet, etc.). Device 402 can access a synthetic dataset generation ML system 406. Synthetic dataset generation ML system 406 can operate on a first dataset to satisfy a dataset goal. Figure 4 In the example of , user 401 has requested more data, such as for training or classification (eg, using a traditional ML paradigm). Synthetic dataset generation ML system 406 may include processing circuitry configured to implement operations 410 , 412 , 414 .

[0069] The processing circuit system may include electronics or electronic components, software or firmware executed on the electronics or electronic components, or a combination thereof. The electronics or electronic components may include one or more resistors, transistors, capacitors, diodes, inductors, logic gates (e.g., AND, OR, XOR, NOT, buffers, etc.), switches, power supplies, oscillators, analog-to-digital converters, digital-to-analog converters, amplifiers, memory devices, processing devices (e.g., central processing units (CPUs), field programmable gate arrays (FPGAs), graphics processing units (GPUs), application specific integrated circuits (ASICs), etc.), combinations thereof, etc.

[0070] The request 404 shown in the figure includes a first data set or a distribution of the first data set and a desired output (classification, data set, ML model, etc.). The distribution of the first data set may include mean, covariance, shape (e.g., Gaussian mixture, etc.).

[0071] Operation 410 includes determining a corresponding mapping from a first labeled data set to at least two labeled training data sets. Operation 412 may include identifying one or more points closest to the first data set in the data set space formed by the second labeled data set. Closer data sets may provide more valuable data for training, fine-tuning, testing, etc. of an ML model configured to operate on the first data. At operation 414, the second data sets may be combined. The proportion of the second data set in the combination may be based on the distance from operation 412 to the identified point.

[0072] The third data set 408 generated as a result of operation 414 may then be provided to the user 401, for example, via the device 402. The third data set 408 may then be used as additional data to the first data set for further training, fine-tuning, or testing of the ML model. The user 401 may then have more data to train and / or test the ML model using the traditional ML paradigm.

[0073] Additionally or alternatively, the privacy of people associated with data in the first or second datasets may be protected by operating on the third dataset 408. The third dataset 408 may be considered a sample from a distribution representing the first dataset. The additional data provided by the third dataset 408 may help improve the accuracy of, reduce bias, etc., an ML model or the like of interest to the user 401.

[0074] Figure 5 A diagram of an embodiment of another system 500 for dataset optimization is shown by way of example. System 500 is similar to system 400, wherein a request 520 of system 500 is different from request 404 of system 400, causing synthetic data generation ML system 406 to provide a different output ( Figure 5 The synthetic data generation ML system 406 of the system 500 receives the request 520 communicated by the device 402. Figure 5 In the example of , request 520 is for ML model 522. Synthetic data generation ML system 406 can perform operation 410 as in system 400.

[0075] The synthetic data generation ML system 406 may perform operation 412 as in the ML system 400. At operation 514, the synthetic data generation ML system 406 synthesizes a third dataset of synthesized samples based on the points identified at operation 412, the mapping determined at operation 410, and the samples from the second labeled dataset. Operation 514 may include generating a generalized geodesic bag of the second dataset. Operation 514 may include combining samples from the second labeled dataset based on distances between the respective datasets and the identified points. Operation 516 includes fine-tuning or otherwise further tuning the ML model based on the synthesized samples.

[0076] Figure 6 A diagram of another embodiment of a system 600 for data set optimization is shown by way of example. System 600 is similar to systems 400 and 500, wherein Figure 6The synthetic data generation ML system 406 performs some operations different from those of the systems 400 and 500. In the system 600, the user 401 issues a request 638 for classification 636. In any of the systems 400, 500, and 600, the user 401 may provide (i) a distribution (e.g., mean and covariance) of features mapped to labels of a first data set, wherein the data set contains the data labels to be classified, (ii) features associated with the labels, (iii) the first data set, or a combination thereof.

[0077] The synthetic data generation ML system 406 may perform operations 410, 412 similar to the systems 400, 500. The synthetic data generation ML system 406 may perform operations 514 similar to the system 500.

[0078] At operation 632 , the synthetic data generation ML system 106 may also train the ML model (pre-trained on the target dataset) using the third dataset. The synthetic data generation ML system 406 may operate the trained ML model on the one or more samples provided in the request 638 to generate the classification 636 .

[0079] The problem with determining the distance between feature-marker pairs is that features are continuous (vectors) and marks are discrete. Determining the distance between features can be performed in many ways. However, the discrete nature of marks makes it more difficult to determine the distance between feature-marker pairs. The solution provided by the embodiment is to represent the mark as a distribution of features mapped to the mark. Then, a differentiable distance metric can be used to determine the distance between distributions (markers).

[0080] The complexity of solving the OTDD centroid projection by Sinkhorn algorithm is O(N 2 ), where N is the number of samples in both datasets. This can be expensive for large datasets. In practice, it is beneficial to solve for batch centroid projection, i.e., extract batches from the source and target datasets and solve for the projection from the source batches to the target batches, and the batch size B is usually fixed to 10. 4 This increases the complexity from O(N 2 ) is reduced to O(BN). The complexity of solving the OTDD neural map is O(BKH), where K is the number of iterations and H is the size of the network. In the experiment, K = O(N). Since the dataset distance will be solved between each pair of training datasets, the complexity of solving all (2, Q) dataset distances in (5) is O(m 2 N). The parts are combined to approximate the interpolation parameters minimized for equation (5) The complexity is O(N(B+m 2 )).

[0081] The generation of synthetic datasets relies on solving the OTDD mapping from the target dataset to each training dataset. These OTDD mappings are customized to the target dataset under consideration and cannot be reused for new target datasets. Another limitation is that the framework is based on model training and fine-tuning pipelines. This may be resource-demanding for large-scale models such as Generative Pre-Trained Transformer (GPT) models.

[0082] The OTDD centroid projection can be performed using the existing OTDD solver available at https: / / github.com / microsoft / otdd (last accessed October 2022). Regarding the OTDD neural mapping, in order to solve problem (4), f, G, Parameterized as three neural networks. In the NIST dataset experiment, f can be parameterized as ResNet and the feature map G can be regarded as UNet. Pre-trained classifiers can be used Generate Tags And use VGG-5 or LeNet with spine layer to parameterize In the 2D Gaussian mixture experiment, a residual MLP can be used to represent them all. The discriminator condition on the label can be removed to simplify the loss function to

[0083] In (6), the first term in the first integral is the feature loss, the second term in the first integral is the label loss, and the second and third integrals combined are the discriminator loss. In (6), assume that y and Both marks are hard marks, but in fact, The output of is a soft label. Simply taking argmax to get hard labels can destroy the computation graph, so the label loss can be Replace with in is a label-to-label matrix, where Where y is the one-hot label from the dataset Q. The matrix M is precomputed before training and is frozen during training. The feature map G can be pretrained as the identity map before the main adversarial training. The exponential moving average of the trained feature map can be used as the final feature map.

[0084] For all NIST dataset experiments, the images are rescaled to size 32×32 and their channels are repeated 3 times to obtain 3-channel images. The default train-test split from torchvision can be used. For the experimental results presented, the OTDD neural map can be used with a learning rate of 10 -3and batch size 64. LeNet is trained for 2000 iterations and fine-tuned for 100 epochs. Regarding the comparison with other baselines, for the transfer learning method, SpinalNet is trained for 10 4 2000 iterations are performed on the test dataset and fine-tuned on the test dataset for 2000 iterations. Training from scratch on the test dataset can be performed for 2000 iterations.

[0085] The embodiments are significantly different from blending and intra-domain interpolation, data set synthesis in ML, and discrete OT, neural OT, and gradient flow. Blending and the related intra-domain interpolation of generating training data through convex combinations were once popularized by blending, which is a simple data augmentation technique that interpolates features and labels between pairs of points. Blending improves intra-domain model robustness and generalization by increasing the intra-distribution diversity of the training data. Although there is some sharing of intuitive principles with blending, the embodiments interpolate the entire data set rather than individual data points, with the goal of improving cross-distribution diversity compared to intra-distribution diversity and thus improving out-of-domain generalization.

[0086] Dataset synthesis in machine learning generates data beyond what is provided as a training dataset and is a key component of ML in practice. Basic transformations such as rotation, cropping, and pixel transformations can be found in most state-of-the-art computer vision models. Generative adversarial networks (GANs) have been used to generate synthetic data in a variety of contexts, a technique that has proven particularly successful in the field of medical imaging. Since GANs are trained to replicate the dataset they are trained on, these methods are often limited to generating in-distribution diversity and typically only manipulate features.

[0087] Discrete OT, neural OT, and gradient flow barycentric projection are typical and effective methods for approximating OT mapping using discrete regularized OT. In addition, the optimal mapping in Euclidean space based on neural networks has made great progress recently, and its capabilities have been revealed in image generation and style transfer. However, the study of the optimal mapping between two datasets is relatively scarce. Some conditional Monge solvers utilize label information in a semi-supervised manner, where they assume that the label-to-label correspondence between the two distributions is known. The embodiment mapping is different from this because the embodiment does not force the label-to-label mapping. Based on the optimal coupling or mapping, geodesics and interpolation in general metric spaces have been widely studied in the OT and metric geometry literature, although most of them are in theoretical settings. As an alternative method for interpolation between distributions, gradient flow has become increasingly popular in ML to model existing processes or solve optimization problems on datasets, but they are computationally more expensive than embodiments.

[0088] Figure 7The diagram marks an embodiment of an example in which images are organized into columns and corresponding marks. Figure 7 For example, Extended MNIST (EMNIST) is chosen as the target dataset. Each training dataset is mapped to have 26 tags (the number of letters in the English alphabet), regardless of the number of tags in the training dataset. At this point, three characteristics of the OTDD mapping are confirmed: 1) No correspondence between the training tags and the target tags is assumed. This allows mapping between two datasets with different tags, such as EMNIST and FastinMNIST; 2) The mapping is invariant to permutations of tag assignments. For example, given two different original tags, the final OTDD mapping will be the same; 3) The mapping does not enforce tag-to-tag mapping, but follows feature similarity. From Figure 7 Many cross-category mapping behaviors can be noticed. For example, when the training domain is the USPS dataset, lowercase “l” is always mapped to the number 1, and uppercase “L” is mapped to other numbers such as 6 or 0, because the mapping follows feature similarity.

[0089] Our OTDD graph can be extended to generate McCann interpolation between datasets. An embodiment can use the analogy of McCann interpolation (1) in dataset space. The McCann interpolation between datasets P0 and P1 can be changed and defined as in is the optimal OTDD mapping from P0 to P1, and t is the interpolation parameter. The superscript M in denotes McCann. The same convex combination method discussed previously can be used to obtain Get a sample. Assume And P0 and P1 contain 7 and 3 categories respectively, that is Then, the feature combination is x t =(1-t)x0+tx1, and the marking combination is

[0090] Therefore (x t ,y t ) is from samples. An embodiment may use a modified version of McCann's method to map labeled data from a target dataset to a training dataset and interpolate between them. Therefore, it can be used to map rich data from an external dataset to a scarce dataset for data augmentation. For example, a target dataset with only 30 samples and a source dataset with 60,000 samples may be used. An OTDD neural mapping between them may be determined, and an interpolation between them may be solved to create new data outside the domain of the target distribution, which cannot be achieved with mixing.

[0091] Figure 8 A diagram of an embodiment of a method 800 for synthetic data set generation is shown by way of example. The method 800 as shown includes: obtaining a first training labeled data set at operation 880; obtaining a second training labeled data set at operation 882; determining an optimal transmission (OT) mapping from a target labeled data set to the first training labeled data set at operation 884; determining an OT mapping from the target labeled data set to the second training labeled data set at operation 886; identifying a point in the distribution space close to the target labeled data set in a generalized geodesic bag formed by the first training labeled data set and the second training labeled data set in the distribution space based on the OT mapping; and generating a synthetic labeled ML data set by combining the first training labeled data set and the second training labeled data set based on the distance between the probability distribution representation of the first training labeled data set and the second training labeled data set in the distribution space at operation 890.

[0092] The method 800 may also include, wherein the target labeled dataset includes more, fewer, or different tags than tags of one or more of the first training labeled dataset and the second training labeled dataset. The method 800 may also include, wherein combining the first training labeled dataset and the second training labeled dataset includes: representing the tags of the first training labeled dataset and the second training labeled dataset as corresponding one-hot vector tags of all tags in the first training labeled dataset and the second training labeled dataset. The method 800 may also include further training a pre-trained ML model that has been trained based on the target labeled dataset using the synthetic labeled ML dataset.

[0093] The method 800 may also include, wherein determining the OT map includes performing a barycentric projection of the target labeled data set onto a geodesic bag. The method 800 may also include, wherein the barycentric projection includes projected labels of the sample data and the separately labeled data. The method 800 may also include, wherein determining the OT map includes operating an OT neural map, the OT neural map including three classifiers: a label classifier, a discriminator, and a feature classifier label. The method 800 may also include, wherein the discriminator loss of the discriminator is independent of the label.

[0094] The method 800 may also include, wherein identifying a point in the dataset space that is close to the target labeled dataset comprises determining a point in the generalized geodesic bag that is closest to the target labeled dataset. The method 800 may also include, wherein identifying a point that is close to the target labeled dataset comprises operating a quadratic problem solver based on a (2, ν) transmission metric. The method 800 may also include, prior to obtaining the first training labeled dataset and the second training labeled dataset, receiving a request for a synthetic labeled ML dataset from an application. The method 800 may also include, in response to generating the synthetic labeled ML dataset, providing the synthetic labeled ML dataset to the application.

[0095] Artificial intelligence (AI) is a field concerned with developing decision-making systems to perform cognitive tasks that traditionally require living things (e.g., humans). A neural network (NN) is a computing structure whose structure is roughly modeled after biological neurons. Typically, a NN encodes information (e.g., data or decisions) via weighted connections (e.g., synapses) between nodes (e.g., neurons). Modern NNs are the basis of many AI applications, such as object recognition, device behavior modeling (as in this application), etc. Operation 118, synthetic data generation ML system 406, operation 410, operation 516, operation 632, operation 634, or other components or operations may include or be implemented using one or more NNs.

[0096] Many NNs are represented as matrices of weights (sometimes called parameters) corresponding to the modeled connections. A MN operates by receiving data into a set of input neurons, which typically have many output connections to other neurons. At each traversal between neurons, the corresponding weights modify the input and are tested against a threshold at the destination neuron. If the weighted value exceeds the threshold, the value is weighted again, or transformed by a nonlinear function, and transmitted further along the NN graph to another neuron; if the threshold is not exceeded, the value is typically not transmitted to downstream neurons, and the synaptic connection remains inactive. The process of weighting and testing continues until an output neuron is reached; the pattern and value of the output neuron constitutes the result of the NN processing.

[0097] Most NNs rely on accurate weights for optimal operation. However, NN designers typically do not know which weights will work for a given application. NN designers typically choose multiple layers of neurons or specific connections between layers, including recurrent connections. A training process can be used to determine appropriate weights by selecting initial weights.

[0098] In some examples, the initial weights may be randomly selected. The training data is fed into the NN and the results are compared to an objective function that provides an error indication. The error indication is a measure of how wrong the NN results are compared to the expected results. The error is then used to correct the weights. Over multiple iterations, the weights will converge together to encode the operational data into the NN. This process may be referred to as optimization of an objective function (e.g., a cost or loss function), thereby minimizing the cost or loss.

[0099] Gradient descent techniques are commonly used to perform objective function optimization. Gradients (e.g., partial derivatives) are calculated relative to layer parameters (e.g., aspects of weights) to provide the direction and possible degree of correction, but do not result in a single correction to set the weights to the "correct" value. That is, over multiple iterations, the weights will move toward the "correct" or operationally useful value. In some embodiments, the amount or step size of the move is fixed (e.g., the same from iteration to iteration). Small step sizes tend to take a long time to converge, while large step sizes may oscillate near the correct value or exhibit other undesirable behaviors. Variable step sizes can be attempted to provide faster convergence without the disadvantages of large step sizes.

[0100] Backpropagation is a technique in which training data is fed forward through a NN - here "forward" means that the data starts at the input neuron and follows the directed graph of neuron connections until the output neuron is reached - and the objective function is applied backward through the NN to correct the synaptic weights. At each step in the backpropagation process, the weights are corrected using the results of the previous step. Thus, the results of the output neuron correction are applied to the neurons connected to the output neuron, and so on, until the input neuron is reached. Backpropagation has become a popular technique for training various NNs. Any well-known optimization algorithm for backpropagation can be used, such as stochastic gradient descent (SGD), Adam, etc.

[0101] Fig. 9 is an example block diagram of an environment including a system for neural network training. The system includes an artificial NN (ANN) 905 trained using a processing node 910. The processing node 910 can be a central processing unit (CPU), a graphics processing unit (GPU), a field programmable gate array (FPGA), a digital signal processor (DSP), an application specific integrated circuit (ASIC), or other processing circuit system. In an example, multiple processing nodes can be used to train different layers of the ANN 905, or even different nodes 907 within a layer. Therefore, a group of processing nodes 910 are arranged to perform the training of the ANN 905.

[0102] The group of processing nodes 910 is arranged to receive a training set 915 for the ANN 905. The ANN 905 includes a group of nodes 907 (shown as multiple rows of nodes 907) arranged in layers and a group of inter-node weights 908 (e.g., parameters) between the nodes in the group of nodes. In the example, the training set 915 is a subset of the complete training set. Here, the subset can enable processing nodes with limited storage resources to participate in training the ANN 1005.

[0103] The training data may include a plurality of values ​​representing a domain, such as image features, etc. After training the ANN 905, each value of the training or input 917 to be classified is provided to a corresponding node 907 in the first layer or input layer of the ANN 905. These values ​​propagate through the layers and are changed by the objective function.

[0104] As described above, the group of processing nodes is arranged to train neural networks to create trained neural networks. For example, after training ANN, the data input into ANN will produce effective classification 920 (for example, input data 917 will be assigned to categories). The training performed by the group of processing nodes 907 is iterative. In an example, each iteration of training ANN 905 is performed independently between the layers of ANN 905. Therefore, two different layers can be processed in parallel by different members of the group of processing nodes. In an example, the different layers of ANN 905 are trained on different hardware. The members of different members of the group of processing nodes can be located in different packages, shells, computers, cloud-based resources, etc. In an example, each iteration of training is performed independently between the nodes in the group of nodes. This example is additional parallelization, whereby individual nodes 907 (for example, neurons) are trained independently. In an example, nodes are trained on different hardware.

[0105] Fig.10A block diagram of an embodiment of a machine 1000 (e.g., a computer system) for implementing one or more embodiments is shown by way of example. One or more of a target data set 110, training data sets 112, 114, or other data may be stored by the machine 900. One or more of operations 116, 118, 120, 122, 124, 410, 412, 414, 514, 516, 632, 634, 802, 804, 806, device 402, synthetic data generation ML system 406 may be performed or implemented using the machine 1000. An example machine 1000 (in the form of a computer) may include a processing unit 1002, a memory 1003, a removable storage device 1010, and a non-removable storage device 1012. Although an example computing device is shown and described as a machine 1000, in different embodiments, the computing device may be in different forms. For example, the computing device may alternatively be a smartphone, a tablet, a smartwatch, or include a computer system such as the one described above. Fig.10 Other computing devices that have the same or similar elements as shown and described. Devices such as smart phones, tablets, and smart watches are generally referred to as mobile devices. In addition, although various data storage elements are shown as part of the machine 1000, the memory may also or alternatively include cloud-based storage accessible via a network (such as the Internet).

[0106] The memory 1003 may include volatile memory 1014 and nonvolatile memory 1008. The machine 1000 may include a computing environment or may have access to a computing environment that includes various computer-readable media, such as the volatile memory 1014 and nonvolatile memory 1008, removable storage 1010, and non-removable storage 1012. Computer memory includes random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM) and electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD ROM), digital versatile disk (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage, or other magnetic storage devices capable of storing computer-readable instructions for execution to perform the functions described herein.

[0107] The machine 1000 may include or may access a computing environment including input 1006, output 1004, and communication connection 1016. The output 1004 may include a display device, such as a touch screen, which may also be used as an input device. The input 1006 may include one or more of a touch screen, a touch pad, a mouse, a keyboard, a camera, one or more device-specific buttons, one or more sensors integrated into the machine 1000 or coupled to the machine 1000 via a wired or wireless data connection, and other input devices. The computer may operate in a networked environment using a communication connection to connect to one or more remote computers, such as a database server, including cloud-based servers and storage. The remote computer may include a personal computer (PC), a server, a router, a network PC, a peer device or other public network node, etc. The communication connection may include a local area network (LAN), a wide area network (WAN), a cellular, an Institute of Electrical and Electronics Engineers (IEEE) 802.11 (Wi-Fi), Bluetooth, or other networks.

[0108] Computer readable instructions stored on a computer readable memory device are executable by a processing unit 1002 of the machine 1000. Hard drives, CD-ROMs, and RAM are some examples of items that include non-transitory computer readable media (such as storage devices). For example, the computer program 1018 can be used to cause the processing unit 1002 to perform one or more methods or algorithms described herein.

[0109] Additional notes and examples:

[0110] Example 1 includes a computer-implemented method for generating a synthetic labeled machine learning (ML) dataset, the method comprising: obtaining a first training labeled dataset; obtaining a second training labeled dataset; determining an optimal transport (OT) mapping from a target labeled dataset to the first training labeled dataset; determining an OT mapping from the target labeled dataset to the second training labeled dataset; based on the OT mapping, identifying points in a distribution space that are close to the target labeled dataset in a generalized geodesic bag formed by the first training labeled dataset and the second training labeled dataset in the distribution space; and generating a synthetic labeled ML dataset by combining the first training labeled dataset and the second training labeled dataset based on the distance between the probability distribution representation of the first training labeled dataset and the second training labeled dataset in the distribution space and the point.

[0111] In Example 2, Example 1 also includes, wherein the target labeled dataset includes more, fewer, or different labels than labels of one or more of the first training labeled dataset and the second training labeled dataset.

[0112] In Example 3, Example 2 also includes, wherein combining the first training labeled dataset and the second training labeled dataset includes: representing the labels of the first training labeled dataset and the second training labeled dataset as corresponding one-hot vector labels of all labels in the first training labeled dataset and the second training labeled dataset.

[0113] In Example 4, at least one of Examples 1 to 3 further includes using the synthetic labeled ML dataset to further train a pre-trained ML model that has been trained based on the target labeled dataset.

[0114] In Example 5, at least one of Examples 1 to 4 further includes, wherein determining the OT map comprises performing a barycentric projection of the target marker dataset onto a geodesic hull.

[0115] In Example 6, Example 5 also includes, wherein the centroid projection includes the sample data and the projected indicia of the individual labeled data.

[0116] In Example 7, at least one of Examples 1 to 6 also includes, wherein determining the OT map includes operating the OT neural map, the OT neural map including three classifiers: a label classifier, a discriminator, and a feature classifier label.

[0117] In Example 8, Example 7 also includes, wherein the discriminator loss of the discriminator is independent of the labeling.

[0118] In Example 9, at least one of Examples 1 to 7 also includes, wherein identifying a point in the dataset space that is close to the target labeled dataset comprises determining a point in the generalized geodesic bag that is closest to the target labeled dataset.

[0119] In Example 10, Example 9 also includes, wherein identifying points close to the target labeled data set includes operating a quadratic problem solver based on a (2,ν) transport metric.

[0120] In Example 11, at least one of Examples 1 to 10 further includes: before obtaining the first training labeled dataset and the second training labeled dataset, receiving a request for a synthetic labeled ML dataset from an application; and in response to generating the synthetic labeled ML dataset, providing the synthetic labeled ML dataset to the application.

[0121] Example 12 may include a system including a memory and a processing circuit system configured to implement the method of at least one of Examples 1-11.

[0122] Example 13 may include a machine-readable medium including instructions that, when executed by a machine, cause the machine to perform the method of at least one of Examples 1-11.

[0123] In some embodiments, the operations, functions or algorithms described herein can be implemented in software. Software can include computer executable instructions stored on a computer or other machine-readable medium or storage device, such as one or more non-transient memories (e.g., non-transient machine-readable media) or other types of local or networked hardware-based storage devices. In addition, such functions can correspond to subsystems, which can be software, hardware, firmware or a combination thereof. Multiple functions can be performed in one or more subsystems as needed, and the described embodiments are only examples. Software can be executed on a digital signal processor, ASIC, microprocessor, central processing unit (CPU), graphics processing unit (GPU), field programmable gate array (FPGA) or other types of processors operating on a computer system (such as a personal computer, server or other computer system), thereby converting such a computer system into a specifically programmed machine. Functions or algorithms can be implemented using processing circuit systems, such as can include electrical and / or electronic components (e.g., one or more transistors, resistors, capacitors, inductors, amplifiers, modulators, demodulators, antennas, radios, regulators, diodes, oscillators, multiplexers, logic gates, buffers, caches, memories, GPUs, CPUs, field programmable gate arrays (FPGAs), etc.).

[0124] As discussed in the background, data used to pre-train machine learning (ML) models often consists of a collection of heterogeneous datasets. Although training on a union of such datasets is reasonable in an agnostic setting, this may be suboptimal when the target domain (where the ML model will eventually be used) is known in advance. In this case, it is ideal to pre-train only on datasets that are most similar to the target dataset. Instead of limiting the selection to those datasets that are already in the pre-training set, an embodiment extends the available datasets to all datasets, which can be synthesized as a "combination" of heterogeneous datasets in a data space spanned by the heterogeneous datasets. This combination is sometimes referred to as "multi-dataset interpolation" or "synthetic dataset". Multi-dataset interpolation can be achieved through generalized geodesics from optimal transport (OT) theory. Generalized geodesics are curves that connect datasets and along which interpolation can be performed. Generalized geodesics can be combined to form an envelope, which is calculated using the distance between labeled datasets. Alternative interpolation schemes can then be used to combine the datasets: using barycentric projection or optimal transport mapping, etc. For example, the optimal transfer mapping can be computed using a neural OT approach (a neural network that solves the OT problem). Embodiments are scalable, efficient, and can even be used to interpolate between datasets with different and unrelated label sets. Through various experiments in transfer learning, embodiments are shown to be useful for targeted on-demand dataset synthesis.

[0125] Although some embodiments have been described in detail above, other modifications are possible. For example, the logic flows depicted in the accompanying drawings do not require the order or sequence shown to achieve the desired results. Other steps can be provided, or steps can be eliminated from the described flows, and other components can be added to or removed from the described systems. Other embodiments may be within the scope of the appended claims.

Claims

1. A computer-implemented method for generating a synthetic labeled machine learning (ML) dataset, the method comprising: Obtain a first training labeled data set; Obtain a second training labeled data set; determining an optimal transfer (OT) mapping from a target labeled dataset to the first training labeled dataset; Determining an OT mapping from the target labeled dataset to the second training labeled dataset; Based on the OT mapping, in a generalized geodesic bag formed by the first training labeled dataset and the second training labeled dataset in the distribution space, identifying a point in the distribution space that is close to the target labeled dataset; as well as The synthetic labeled ML dataset is generated by combining the first training labeled dataset and the second training labeled dataset based on the distance between the probability distribution representations of the first training labeled dataset and the second training labeled dataset and the point in the distribution space.

2. The computer-implemented method of claim 1 , wherein the target labeled dataset comprises more, fewer, or different labels than labels of one or more of the first training labeled dataset and the second training labeled dataset.

3. The computer-implemented method of claim 2, wherein combining the first training labeled data set and the second training labeled data set comprises: The labels of the first training labeled dataset and the second training labeled dataset are represented as corresponding one-hot vectors of all labels in the first training labeled dataset and the second training labeled dataset.

4. The computer-implemented method of claim 1 , further comprising using the synthetic labeled ML dataset to further train a pre-trained ML model that has been trained based on the target labeled dataset. 5 . The computer-implemented method of claim 1 , wherein determining the OT map comprises performing a barycentric projection of the target marker dataset onto the geodesic envelope.

6. The computer-implemented method of claim 5, wherein the centroid projection comprises a projection of the sample data and the individually labeled data. 7 . The computer-implemented method of claim 1 , wherein determining the OT map comprises operating an OT neural map, the OT neural map comprising three classifiers: a label classifier, a discriminator, and a feature classifier.

8. The computer-implemented method of claim 7, wherein a discriminator loss of the discriminator is independent of the labels.

9. The computer-implemented method of claim 1, wherein identifying the point in the dataset space that is close to the target labeled dataset comprises determining a point in the generalized geodesic bag that is closest to the target labeled dataset.

10. The computer-implemented method of claim 9, wherein identifying the points close to the target labeled data set comprises operating a quadratic problem solver based on a (2, v) transport metric.

11. The computer-implemented method of claim 1 , further comprising: Before obtaining the first training labeled dataset and the second training labeled dataset, receiving a request for the synthetic labeled ML dataset from an application; as well as In response to generating the synthetic labeled ML dataset, providing the synthetic labeled ML dataset to the application.

12. A machine-readable medium comprising instructions that, when executed by a machine, cause the machine to perform operations for generating a synthetic labeled machine learning (ML) dataset, the operations comprising: Obtain a first training labeled data set; Obtain a second training labeled data set; determining an optimal transfer (OT) mapping from a target labeled dataset to the first training labeled dataset; Determining an OT mapping from the target labeled dataset to the second training labeled dataset; Based on the OT mapping, in a generalized geodesic bag formed by the first training labeled dataset and the second training labeled dataset in the distribution space, identifying a point in the distribution space that is close to the target labeled dataset; as well as The synthetic labeled ML dataset is generated by combining the first training labeled dataset and the second training labeled dataset based on the distance between the probability distribution representations of the first training labeled dataset and the second training labeled dataset and the point in the distribution space.

13. The machine-readable medium of claim 12, wherein the target labeled dataset includes more, fewer, or different labels than labels of one or more of the first and second training labeled datasets.

14. The machine-readable medium of claim 13, wherein combining the first training labeled data set and the second training labeled data set comprises: The labels of the first training labeled dataset and the second training labeled dataset are represented as corresponding one-hot vector labels of all labels in the first training labeled dataset and the second training labeled dataset.

15. A system for generating a synthetic labeled machine learning (ML) dataset, the system comprising: Processing circuit system; as well as A memory coupled to the processing circuitry, the memory comprising instructions which, when executed by the processing circuitry, cause the processing circuitry to perform a method according to one of claims 1 to 11.

Images