Gradient flow in dataset space

Abstract

Generally, discussed herein are devices, systems, and methods for machine learning (ML) by flowing data sets toward target data sets. A method can include receiving a request for a first data set operation including a first feature-label pair, identifying a second data set from a plurality of data sets, the second data set including a second feature-label pair, determining a distance between the first feature-label pair and the second feature-label pair, and flowing the first data set using a data set target operated based on the determined distance to generate an optimized data set.

Description

BACKGROUND

[0001] Current practice of machine learning (ML) is model-centric. Current practice reduces the problem to adjustment of model parameters while assuming that data is either fixed or subject to extrinsic and unavoidable variation. This practice fails to capture important existing aspects of ML, such as data operations (e.g., augmentation), which involve usual ML design, training, and operation. Moreover, this practice is ill-suited to formalize new data-centric problems, such as model-agnostic transfer learning or data set synthesis. SUMMARY

[0002] This Summary is provided to introduce a selection of concepts in a simplified form, further descriptions of which follow in the detailed description. This Summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter to the elements listed in this Summary. Embodiments

[0003] Embodiments can perform machine learning (ML) operations in a manner that is different from traditional ML paradigms. Embodiments can flow a first data set toward a second data set based on a specified goal. Embodiments can allow modification (“flow”) of a data set to meet a goal, rather than modification of model parameters to meet a goal.

[0004] Methods, systems, computer-readable media, devices, and the like can be configured to implement one or more embodiments. An ML method of flowing a data set toward a goal data set can include receiving a request for a first data set operation that includes a first feature-label pair. The method can include identifying a second data set from a plurality of data sets, the second data set including a second feature-label pair. The method can include determining a distance between the first feature-label pair and the second feature-label pair. The method can include flowing the first data set using a data set goal that is based on the determined distance operation to generate an optimized data set.

[0005] The method can further include, wherein each label is represented by a probability distribution of features that map to the label in the distance determination. The method can further include operating a first ML model trained on the second data set with the altered first data set as input to provide a result. The method can further include training a second ML model using at least a portion of the second data set and the altered first data set. The method can further include providing the altered first data set.

[0006] The method may further include keeping the first ML model static during training. The method may further include determining a distance, wherein determining the distance includes determining a differentiable distance. The method may further include that the distance includes the optimal transport distance (OTD). The method may further include that determining the OTD includes determining the distance between corresponding labels based on features of data points associated with corresponding labels of the first and second datasets. The method may further include that making the first dataset flow includes determining the gradient of the dataset objective relative to the first dataset, and making the first dataset flow also includes updating the first dataset using the determined gradient. The method may further include that the dataset objective includes at least one of the following: minimizing internal energy, minimizing potential energy, minimizing distance, or minimizing interaction energy.

[0007] The method may further include, where the dataset objective includes minimizing the distance between feature label pairs. The method may further include, where updating the first dataset using the determined gradients includes feature-driven updates, jointly driven fixed-label updates, or jointly driven variable-label updates. The method may further include, where the first and second datasets comprise image datasets, or where the first and second datasets comprise natural language processing (NLP) datasets. Attached Figure Description

[0008] Figure 1 A logical block diagram of an embodiment of a system for dataset optimization is shown as an example.

[0009] Figure 2 A logic block diagram of another embodiment of a system for dataset optimization is shown as an example.

[0010] Figure 3 A logic block diagram of yet another embodiment of a system for dataset optimization is shown as an example.

[0011] Figure 4 A flowchart illustrating an example of a technique for dataset optimization is provided.

[0012] Figure 5 A flowchart illustrating another embodiment of a technique for dataset optimization is shown as an example.

[0013] Figure 6 A series of plots illustrating gradient flow at various time steps are shown as examples.

[0014] Figure 7 Another series of plots illustrating gradient flow at various time steps are shown as examples.

[0015] Figure 8The error bar charts for different techniques used in transfer learning using four image classification datasets are shown as examples.

[0016] Figure 9 Several image sequences in the embodiment are shown as examples, illustrating images at various times during the gradient flow process.

[0017] Figure 10 A block diagram of an embodiment of a machine (e.g., a computer system) for implementing one or more embodiments is shown by way of example. Detailed Implementation

[0018] In the following description, reference is made to the accompanying drawings, which form a part of this document, and specific embodiments that can be practiced are illustrated by way of illustration. These embodiments are described in sufficient detail to enable those skilled in the art to practice them. It should be understood that other embodiments may be used, and structural, logical, and / or electrical changes may be made without departing from the scope of the embodiments. Therefore, the following description of the embodiments should not be construed as limiting, and the scope of the embodiments is defined by the appended claims.

[0019] In some embodiments, the operations, functions, or algorithms described herein may be implemented in software. This software may include computer-executable instructions stored on a computer or other machine-readable medium or storage device, such as one or more non-transitory memories (e.g., non-transitory machine-readable media) or other types of hardware-based storage devices (local or network). Furthermore, such functionality may correspond to a subsystem, which may be software, hardware, firmware, or a combination thereof. Multiple functions may be performed in one or more subsystems as needed; the described embodiments are merely examples. The software may execute on a digital signal processor, ASIC, microprocessor, central processing unit (CPU), graphics processing unit (GPU), field-programmable gate array (FPGA), or other types of processor operating on a computer system (such as a personal computer, server, or other computer system), thereby turning such a computer system into a specially programmed machine. Functions or algorithms may be implemented using processing circuitry, which may include electrical and / or electronic components (e.g., one or more transistors, resistors, capacitors, inductors, amplifiers, modulators, demodulators, antennas, radios, regulators, diodes, oscillators, multiplexers, logic gates, buffers, caches, memories, GPUs, CPUs, field-programmable gate arrays (FPGAs), etc.).

[0020] In traditional machine learning (ML), the learning task focuses on inference, regularization, and data preprocessing from a model-centric perspective. In traditional ML paradigms, the optimization problem is transformed into finding model parameters that minimize certain dataset objectives (e.g., empirical risk) on a given dataset. Throughout this process, it is typically assumed that the data distribution is fixed and immutable. If not, as in transfer learning and domain adaptation paradigms, it is assumed that the source of variation is external and uncontrollable. Therefore, these paradigms are also transformed into learning optimal model parameters, with the added constraint that these paradigms should account for variations in the data distribution. Thus, from an optimization perspective, most ML can be understood as model parameter optimization.

[0021] In contrast to normal ML practice, this embodiment treats the dataset as a variable and optimizes it. This view views the ML task as an optimization of the data generation distribution. The embodiment addresses this type of problem through gradient flows in the dataset's probability space. The embodiment provides a practical and efficient particle-based approach that is flexible, performs well, and is effective for a wide variety of dataset objectives. The embodiment provides a framework that is effective for dataset interpolation, synthesis, aggregation, data privacy, and other applications in ML.

[0022] The examples provide devices, systems, and methods for generating optimization problems using fundamentally different classes of optimization problems. Optimization problems focus on dynamic datasets rather than dynamic models. Broadly speaking, optimization problems involve finding or modifying datasets to optimize the objective function of a dataset of interest. The dataset objective can be model-specific (e.g., on which dataset does the model perform best?) or model-agnostic (e.g., what are the most similar linearly separable datasets?). This dataset optimization paradigm is a very different ML approach from model parameter optimization. The examples provide solutions to common ML problems (such as dataset generation) that cannot be solved using traditional ML paradigms. Maintaining enough data to train a sufficiently accurate ML model across the domain of interest can be very challenging. This is at least partly because training an ML model to sufficient accuracy may require thousands, tens of thousands, hundreds of thousands, or even millions of data sample (feature, label) pairs. Annotation (determining or validating labels for a set of particle features) can be very time-consuming and costly. Furthermore, an ML model trained on a dataset containing features with a first distribution may not be accurate enough when operating on data with a second, different distribution (even if the distributions are similar). Therefore, generating datasets is a technical problem. The embodiments provide technical solutions to such problems and others.

[0023] The examples provide tools for formalizing various well-known data processing heuristics that form part of most state-of-the-art ML pipelines, such as data augmentation, dataset pooling, or instance hybridization. The examples also allow for posing new problems or solving existing problems from fresh perspectives, such as dataset synthesis with specific (e.g., size or privacy) constraints, missing data imputation, or data compression. Meta-learning is another problem related to dataset-centric approaches. Instead of the usual approach of "retro-fitting" new tasks to the domain in which the model is trained, meta-learning adapts model parameters to the task by optimizing the dataset based on dataset objectives.

[0024] At least two features distinguish the model parameter optimization and dataset optimization paradigms. First, the dataset optimization paradigm is inherently model-independent. This allows for dataset operations that are not specific to a particular model. The generated dataset can benefit all models trained or operated on in the generated dataset. Second, while model parameter optimization is typically transformed into a finite-dimensional problem (e.g., in R² for a model with k parameters), the dataset optimization paradigm is fundamentally different from the model-specific one. k While dataset optimization is most naturally formulated as optimization of a dataset over an infinite-dimensional space, the exact number of samples is rarely relevant and is often unspecified (e.g., in a streaming setting). Instead, the real object of interest is the generative process that produces the dataset. For example, in the context of supervised learning, a dataset can be characterized by a joint distribution p(X, Y), where X and Y are the features and labels, respectively, and can be represented by the distribution of the feature and label pairs. In this case, the optimization problem would be defined over the space of the joint distribution P(X×Y) (e.g., finding P*∈P(X×Y) that minimizes the objective F(p) for a given dataset). There are several ways to handle and formalize this problem. One approach is to use gradient flow, which is key to applied mathematics for modeling dynamics in very general infinite-dimensional spaces. Gradient flow using gradient descent is an iterative technique of finding the first derivative of a differentiable function's local minimum. To find a local minimum of a function using gradient descent, a step is taken at the current point that is proportional to the negative of the function's gradient (or approximate gradient). Gradient flows have a variety of attractive properties: they are inherently flexible in terms of the underlying space and the types of dynamics that can be modeled, they can be subjected to rigorous convergence analysis, and they can generate complete iterative trajectories in addition to the final minimization solution, which is often useful.

[0025] However, utilizing gradient flows for labeled dataset optimization presents various challenges. It is difficult to define appropriate representations and meaningful metrics for feature-label pairings between datasets. A class of flows must then be found that are expressive enough to model the objective function of the dataset of interest on the dataset, but perform well enough to allow for tractable optimization. In response to the first of these challenges, the embodiment leverages the recently proposed concept of distance between datasets based on optimal transport (OT), termed Optimal Transport Distance (OTD) or Optimal Transport Dataset Distance (OTDD), which provides a meaningful metric for the space of the joint distribution. This metric can be optimized by making it differentiable. The embodiment can use dataset objectives that can be repurposed to encode various dataset-related objectives. As a result, the embodiment provides a flexible and efficient framework for dataset optimization.

[0026] Typically, implementations can use differential distance metrics and gradient flows based on determined distances to optimize a dataset targeting a dataset. Details of example differentiable distance metrics (OTDD) and gradient flows are discussed further below. Further details and applications of the implementation will now be provided with reference to the accompanying drawings.

[0027] Figure 1 A diagram illustrating an embodiment of a system 100 for dataset optimization is shown by way of example. The system 100 shown includes a user 101 with a device 102. Device 102 can issue a request 104 to a data optimization ML system 106. Request 104 can indicate the location of a first dataset, include the first dataset, etc. Request 104 can indicate desired results, such as an ML model, data that meets specific criteria, etc.

[0028] Device 102 is a computing device, such as a computer (e.g., laptop, desktop, handheld, smartphone, tablet, phablet, etc.). Device 102 can access data optimization ML system 106. Data optimization ML system 106 can operate on a first dataset to satisfy dataset objectives. Figure 1 In the example, user 101 requests more data, such as for training or classification (e.g., using a traditional ML paradigm). Data optimization ML system 106 may include processing circuitry configured to implement operations 110 and 112.

[0029] The processing circuitry may include electrical or electronic components, software or firmware executed on the electrical or electronic components, or a combination thereof. The electrical or electronic components may include one or more resistors, transistors, capacitors, diodes, inductors, logic gates (e.g., AND, OR, XOR, inverting, buffers, etc.), switches, power supplies, oscillators, analog-to-digital converters, digital-to-analog converters, amplifiers, memory devices, processing devices (e.g., central processing unit (CPU), field-programmable gate array (FPGA), graphics processing unit (GPU), application-specific integrated circuit (ASIC), etc.), combinations thereof, etc.

[0030] The request 104 shown includes a first dataset or the distribution of the first dataset and the desired output (classification, dataset, ML model, etc.). The distribution of the first dataset may include the mean, covariance, shape (e.g., Gaussian mixture, etc.).

[0031] Operation 110 includes determining a differentiable distance between feature label pairs of a first dataset provided by user 101 and feature label pairs of one or more second datasets. The differentiable distance may include OTDDs. Operation 110 may include identifying one or more second datasets that are closest to the first dataset. Datasets that are closer may require less processing (fewer iterations) to achieve the dataset objective compared to datasets that are farther away. In operation 112, the identified second datasets may be modified to maximize the objective. Operation 112 may include minimizing the dataset objective using gradient flow and the determined differentiable distance. The dataset objective may include potential energy, Wasserstein distance, internal energy, interaction energy, or a combination thereof. The dataset objective may impose constraints on data modification, such as separability between labels (e.g., by potential energy or interaction energy), constraints on feature norms, dataset similarity (e.g., by distance to the dataset objective), entropy considerations (e.g., using internal energy), another constraint, or a combination thereof.

[0032] Table 1 shows the differentiable distances between various datasets:

[0033] MNIST EMNIST FashionMNIST KMNIST USPS MNIST 1.04 1.74 1.41 1.26 EMNIST 1.04 1.57 1.28 1.32 FashionMNIST 1.74 1.57 1.67 1.10 KMNIST 1.41 1.28 1.67 1.30 USPS 1.26 1.32 1.10 1.30

[0034] Table 1: OTDD between some National Institute of Standards and Technology (NIST) datasets and the U.S. Postal Service dataset.

[0035] The results in Table 1 show that MNIST and EMNIST are the closest (most similar) datasets in Table 1, while fashionMNIST is the furthest (least similar) to MNIST in the same dataset. If the first dataset is USPS, fashionMNIST could be the second dataset 108, as it is the closest to USPS among the available datasets.

[0036] Then, for example, a modified second dataset 108 can be provided to user 101 via device 102. The modified second dataset 108 can then be used as supplementary data to the first dataset, especially if the distance is the dataset target at operation 112. User 101 can then have more data to train and / or test ML models using traditional ML paradigms.

[0037] Alternatively, the privacy of individuals associated with data in the first or second dataset can be protected by modifying the dataset towards the other dataset and then manipulating the modified dataset. The modified second dataset 108 can be considered as a sample from a distribution representing the first dataset. The additional data provided by the modified second dataset 108 can help improve the accuracy of the ML model of interest to user 101, reduce bias, etc.

[0038] Figure 2 A diagram illustrating an embodiment of another system 200 for dataset optimization is shown as an example. System 200 is similar to system 100, but request 220 of system 200 differs from request 104 of system 100, causing data optimization ML system 106 to provide different outputs ( Figure 2 (Example: fine-tuned model 222). The data optimization ML system 106 of system 200 receives request 220 transmitted by device 102. Figure 2 In the example, request 220 is for ML model 222. Data optimization ML system 106 can perform operation 110 just like in system 100.

[0039] At operation 112, the data optimization ML system 106 can modify the second dataset based on the determined differentiable distance and dataset objective. Then, at operation 226, the first dataset can be used to fine-tune the ML model trained based on the modified second dataset. The fine-tuned ML model 222 can be provided to device 102.

[0040] Figure 3 A diagram of another embodiment of system 300 for dataset optimization is shown as an example. System 300 is similar to systems 100 and 200. Figure 3 The data optimization ML system 106 performs some operations that differ from those of systems 100 and 200. In system 300, user 101 issues a classification request 338. In any of systems 100, 200, and 300, user 101 may provide (i) the distribution of features mapped to the labels of a first dataset (e.g., mean and covariance), where the data to be classified is a member of the labels of the first dataset, (ii) the features associated with the labels, and (iii) the first dataset, or a combination thereof.

[0041] At operation 110, the data optimization ML system 106 can determine a differentiable distance based on the data provided by user 101. At operation 330, the data optimization ML system 106 can modify the first dataset or other data provided by user 101 based on the determined differentiable distance and the dataset objective. If the dataset objective is to minimize the OTDD or other distance between the first and second datasets, operation 330 can shift the distribution of the first dataset toward the distribution of the second dataset. This is beneficial because the second dataset can be used to train the ML model. If the first dataset or other data to be classified provided by user 101 is within the distribution of the second dataset, the ML model can operate to determine the classification of the data, but in the domain of the second dataset, the domain may not have labels overlapping with the first dataset. Since the gradient flow and distance determination are differentiable, the process is reversible. This means that the classification flow in the domain of the second dataset can be returned to the domain of the first dataset, preserving the label variables, to determine the appropriate label in the domain of the first dataset based on the classification in the domain of the second dataset.

[0042] To execute the data flow, at operation 332, the ML model trained in the second dataset domain is used to operate on the modified first dataset or other modified data provided by user 101. Then, at operation 334, the operations opposite to those in operation 330 are performed on the features and labels generated in operation 332. Operation 334 includes modifying the results based on the determined differentiable distance between the results and the first dataset to flow the results into the first dataset domain.

[0043] Figure 3 This operation can be used in, for example, Natural Language Processing (NLP). In NLP, storing the bidirectional encoder representation (BERT) from the transducer is prohibited because BERT is too large. In user email models, the model (BERT) is typically pre-trained based on user-specific data. Figure 3 The operation involves learning BERT on a standard corpus (such as ENRON emails) and moving user-domain data to the BERT domain, rather than changing BERT to the user domain. BERT can then be used to operate on the changed user-domain data. The results of operating on BERT can then be moved to the user domain to generate user-specific NLP data.

[0044] Operations involving modifying the dataset and determining differentiable distances can include backpropagation of the dataset using gradient flow and OTDD (or other differentiable distance metrics), respectively. These operations will be discussed in more detail below, including mathematical descriptions.

[0045] The problem with determining the distance between feature-label pairs is that features are continuous (vectors), while labels are discrete. The distance between features can be determined in various ways. However, the discreteness of labels makes determining the distance between feature-label pairs much more difficult. The solution provided by the examples is to represent labels as a distribution of features mapped to the labels. Then, a differentiable distance metric can be used to determine the distance between the distributions (labels).

[0046] Figure 4 A logic diagram illustrating feature label pairing by label grouping and determining the distribution (mean and covariance) of features for each label is shown as an example. Figure 4 In the dataset, the first dataset 440 includes samples, which include feature label pairs 442, 444, 446, 448, 450, and 452. Each feature is mapped to an associated label, such as through machine learning techniques or manual analysis. The features mapped to the labels are correlated with each other and form feature label pairs 442, 444, 446, 448, 450, and 452.

[0047] Feature label pairs 442, 444, 446, 448, 450, 452 are grouped and sorted by label, etc. At operation 454, the features mapped to label 0 are used to determine the distribution of features associated with label 0 (mean and variation (e.g., standard deviation, covariance, variance, etc.) and shape). At operation 456, the features mapped to label 1 are used to determine the distribution of features associated with label 1. At operation 456, the features mapped to label 2 are used to determine the distribution of features associated with label 2. Note that more or fewer labels can be used.

[0048] At operation 460, the distribution of features determined at operations 454, 456, and 458, along with the features themselves, is used to determine the differentiable distance between feature label pairs (where the labels of the label pairs are represented by the determined distribution). At operation 462, either the first or second dataset is then modified based on the distance and a dataset objective (which can be minimized, such as having or not having constraints or another dataset objective). The modified first or second dataset can then be used for applications in traditional ML paradigms, such as transfer learning, classification, feature label pair generation, dataset interpolation, etc.

[0049] Mathematical descriptions and discussions of OTD, OTDD, gradient flow, and their interoperability are provided. Let χ be a Polish space with metric d, and P(χ) be a set of nonnegative Borel measurements with finite second moments on the Polish space. Consider both continuous and discrete measurements, the latter represented as empirical distributions: Where δ xIt is the Dirac at position x∈χ. For a measurement ρ and a measurable mapping T: χ→χ, T #ρ This indicates the advancement of measurement. The example operates in a supervised ML setting; therefore, the dataset is defined as a set of feature label pairs {(x...} i y i We can assume that the dataset is sampled from some distribution, such that... The dataset is represented as D α To make sampling explicit. Shorthand notation. and Used. Δ· represents the convergence operator.

[0050] Typically, OTD stands for Differentiable Distance metric. OTD is model-independent and does not involve training. OTD relies on optimal transport. OTD uses hybrid Euclidean-Wasserstein for feature-label pairings across domains (different datasets). Each label is modeled as a distribution of feature vectors associated with that label. Modeling feature vectors in this way allows for comparison of datasets even if the label sets are completely disjoint (non-overlapping).

[0051] In mathematical terms, optimal transport can be described as a function of α, β ∈ P(χ) and cost c: As shown below:

[0052]

[0053] Where Π(α, β) is the set of couplings (transportation plans) between α and β, such that:

[0054]

[0055] For p≥1, c(x,y)=d(x,y) p , This is known as the p-Wasserstein distance. W p The distance on P(χ) is defined. It is a metric space, known as the (p-)Wasserstein space.

[0056]

[0057] in It is called the Kantorovich power, and It is its c-conjugation: For c(x, x′) = ||xx′|| 2 φ c It is Fenchel conjugate.

[0058] The dynamic formula for optimal transport (OT) is:

[0059]

[0060] The minimum value is taken from the pair that satisfies μ0 = α, μ1 = β and the continuity equation:

[0061]

[0062] In this formula, for the measurement μ t The shortest path in the convection is from α to β. Formally, this measures the minimum of the derivative. Compared to the global correspondence in the static formula presented in Equation 1 (via π), the dynamic formula focuses on local propagation (via μ). t ).

[0063] OT can be used to define the distance between datasets (OTD). However, this is particularly important for labeled datasets. The problem is that solutions to the distance problem involve element-wise metric d, which, for labeled datasets, defines the distance between pairs of feature labels from different datasets. This is a general classification problem involving discrete sets, which is difficult. To overcome this problem, the label y can be represented as a distribution mapped to the features corresponding to the label. For a simple digit classification problem, α1 is the distribution on images with specific labels, such as y = 1 (digit = 1).

[0064] Use the metric on z as Using ground costs in Equation 1, the distance between measurements on the dataset is derived. This is called the OT dataset distance (OTDD) and is formally defined in Equation 6:

[0065]

[0066] The advantage of OTDD is that it is effective even if the label sets of two datasets do not overlap, or there is no explicit known correspondence between them (e.g., numbers to letters, cats to clothes, etc.). The distance of label overlap is independent of the geometric processing of features and labels.

[0067] Gradient flow in function The gradient flow is defined on the point x0 ∈ χ. The gradient flow is a continuous curve x(t) that evolves from x0 in the steepest descent direction of F. When χ is Hilbertian (complete, allowing calculus to be used and having an inner product structure that allows length and angle to be measured) and F is sufficiently smooth, the gradient flow can be expressed as a differentiable equation with the initial condition x(0) = x0. The solution. Some discrete gradient descent techniques include momentum and acceleration.

[0068] The formulas for OTDD and gradient descent are now used to explain optimization using the same dataset. Given a dataset whose objective is expressed as a function, the dataset objective... Determine the joint measurement ρ∈P(Z) that realizes Equation 7:

[0069]

[0070] Gradient flow method (moves along the steepest descent curve, starting at ρ0 and continuing until the solution ρ is reached) * Unfortunately, It is infinite-dimensional and non-Hilbertian. To overcome this, the concept of the derivative (variation) is provided: given a function... Consider a perturbation χ such that for at least every ε∈[0, ε0], ρ+εχ∈P(Ω). If there exists a function G such that for every such perturbation χ, ρ+εχ∈P(Ω) G is called the first variation of F at ρ, and is given by Represented. Then, the gradient flow of F (ρ) t ) t≥0 Represented as a solution to equation 8:

[0071]

[0072] Equation 8 includes a term similar to the gradient in the Wasserstein metric, i.e. Equation 8 is for ρ t and velocity field The continuity equation.

[0073] A dataset target is the Wasserstein distance to a target distribution: This can be determined using OTDD as described above. Other dataset objectives are also possible, such as those defined in Equations 9, 10, and 11.

[0074]

[0075]

[0076]

[0077] in It is convex and superlinear, and It is convex and sufficiently smooth. These terms have physical interpretations of internal energy, potential energy, and interaction energy, respectively. These dataset targets can be used with backpropagation through automatic differentiation. A combination of one or more dataset targets can be used as a dataset target. Therefore, the dataset target of interest can be as shown in Equation 12:

[0078]

[0079] Equation 13 provides the first variation of the objective for each dataset:

[0080]

[0081] Where * denotes the convolution operator between the measurable function and the measurement, and This is the Kantorovich potential in the double OT formula (Equation 3). The gradient flow of the form Equation 12 for the dataset objective is the solution to Equation 14:

[0082]

[0083] In the context of probability density and datasets, the probability density equation (PDE) can be understood as the principle of mass conservation: In solving the density sequence of this system, no probabilistic mass is generated or destroyed. For only... and The dataset objective of the form of Equation 12, defined by Equation 14, is called the diffusion-convection-interaction equation. Dataset objective and A specific choice can be made to recover a known PDE. For example, taking... And f(t) = tlogt, the gradient flow of F will be given by the Fokker-Planck equation. The solution definition is as follows. In the dataset space, this equation can be interpreted as the temporal evolution of the dataset influenced by the drift force exerted by the potential function V and the constant variance diffusion term (Δρ) generated by the entropy-induced dataset objective F. Other choices of dataset objectives provide convection equations, porous media equations, and various other diffusion-convection-interaction PDEs.

[0084] PDEs can be described using random variables. Consider a stochastic process (Zi). t ) t Each of them They all have a regular ρ t A random variable ∈ P(Z). Equation 8 and Z t The stochastic differentiable equation (SDE) on the above is associated with what is called a McKean-Vlasov process according to Equation 15:

[0085]

[0086] for Equation 15 can be interpreted as the trajectory of a particle, having an initial position ρ0 and moving according to a potential function that captures its intrinsic dynamics and interactions with other particles, all driven by F. This particle view of gradient flow is suitable for computational schemes operating on a finite number of samples. The process defined by Equation 15 is defined on Z (a finite-dimensional space) rather than on an infinite-dimensional P(Z), which allows for corrective computation.

[0087] Numerical methods for solving SDEs, such as those defined in Equation 15, can include discretization, such as the forward Euler scheme defined in Equation 16.

[0088]

[0089] Computationally, this discretization scheme is approximated by a simultaneously evolving particle system. From Initially, each particle z (i) Based on Equation 16, we obtain a system consisting of n updates, as shown in Equation 17:

[0090]

[0091] Then, ρ t It can be approximated as Where ρ N，t It is the gradient flow of F. For well-performing dataset targets, this flow inherits all the convexity and stability properties of accurate dataset targets, and ρ N，t (x)→ρ t (x).

[0092] Distance dataset target can be defined as Where β is the (fixed) target dataset distribution. Various applications (such as dataset interpolation or sample generation for transfer learning) can be modeled using this dataset objective. The distance dataset objective can be combined with other dataset objectives to "shape" the distribution ρ of interest. This shape can include the separation between labels (linear separation), the regularity of points in the distribution, the size of the distribution (e.g., the average amplitude of samples, mathematical norm, etc.), or other constraints.

[0093] Potential energy dataset target It can be used to enforce local (per instance) constraints. For example, the norm of a dataset's features can be set by establishing V(z) = V (x，y) =||x|| or more generally, for certain The constraints are V(z) = ||Ax-b||. These constraints can be enforced on a per-class basis, for example, using V(z) = |(|Ax-b||. y xb y Each class of |)| has different constraints. In another example, the linear separability between datasets can be expressed using the marginal induced potential V(z) = max{0, y(x)|. T This can be achieved by wb)}.

[0094] Interaction potential A dataset objective can be encoded that models the interactions or clustering among samples in the dataset. For example, class exclusion constraints include... Otherwise, the class exclusion constraint includes 0. This encourages class separation by penalizing (z, z′) pairs with different labels but similar features.

[0095] Internal energy dataset target An explicit density ρ(z) in closed form is required. The entropy term can be used for the dataset objective by taking f(t) = tlogt.

[0096] This section provides some practical implementation considerations regarding OTDD and gradient flow applications. When determining the distance of a differentiable dataset, such as OTDD, the difficulty in determining OTDD lies in the discreteness of the label y. Recall that each point (particle) is a pair of features x and a corresponding label y represented as (x, y), where... (continuous r-dimensional space) and y∈{c i c K}(discrete space). The OTDD discussion of Equation 6 provides an alternative representation of the feature and label pair as (x, v), where v ∈ P(χ) is a measurement on χ associated with the label y.

[0097] One way to achieve this is to backpropagate the gradient to x. (i) Gradient step size adjustment is performed on these features, and then the mean and covariance are calculated (therefore, the label v). y (The probability representation). Formally, this is expressed as:

[0098]

[0099]

[0100]

[0101]

[0102] Note that the mean and covariance evolve in response to gradient changes on the samples. In this formula, the gradient does not act directly on the mean and covariance. This type of formula is sometimes called a feature-driven method. A possible drawback of this method is that the label assignments are fixed over time. This is acceptable if the first and target datasets contain the same number of labels. A jointly driven fixed-label formula can perform independent gradient updates on the mean and covariance while keeping the label assignments fixed. Formally, the jointly driven fixed-label update is expressed as:

[0103]

[0104]

[0105]

[0106]

[0107] Relaxing the constraints on label assignment fixed by the gradient flow process can include evolving the distribution associated with each point individually. The distribution of each label can be decoupled, producing a distribution for each sample rather than for each class. Each sample (sometimes called a particle) can then evolve independently, and the labels can be recovered, such as through the evolved clustering. This is known as a jointly driven variable label formulation. Formally, the jointly driven variable label update is expressed as:

[0108]

[0109]

[0110]

[0111]

[0112] To recover the labels, clustering techniques (e.g., nonparametric clustering techniques that do not require specifying the number of clusters) can be applied to the mean and covariance pairs (μ). i , ∑ i To perform the operation.

[0113] Here are some practical implementation considerations for the dataset objective F(ρ). The first variation of the potential energy dataset objective is a scalar-valued function, so it can be computed using automatic differentiation. The first variation of the interaction energy dataset objective can be approximated as an empirical expectation over the samples. If the density is available and backpropagation is possible, the first variation of the internal energy dataset objective can be determined. In this case, automatic differentiation can be used to determine the gradient.

[0114] The step size for gradient application can be fixed or variable. To accelerate convergence and account for potential violations of the convexity assumption in the dataset objective, adaptive step-size techniques such as momentum stochastic gradient descent (SGD), adaptive moment estimation (ADAM), or adaptive gradient algorithm (ADAGARD) can be used. This adaptive step size helps avoid local minima.

[0115] Figure 5 A diagram illustrating an embodiment of a method 500 for dataset optimization is shown as an example. The method 500, as shown, includes receiving a request at operation 502 for operating on a first dataset including a first feature label pair; at operation 504, identifying a second dataset from a plurality of datasets, the second dataset including a second feature label pair; at operation 506, determining a distance between the first and second feature label pairs; and at operation 508, using a dataset target based on the determined distance to flow the first dataset to generate an optimized dataset.

[0116] Method 500 may further include, wherein each label is represented by a probability distribution of features mapped to the label in distance determination. Method 500 may further include, taking the modified first dataset as input, operating the first ML model trained on the second dataset to provide results. Method 500 may further include, keeping the first ML model static during training.

[0117] Method 500 may further include training a second ML model using at least a portion of the second dataset and the modified first dataset. Method 500 may further include providing the modified first dataset. Method 500 may further include determining a distance, wherein determining the distance includes determining a differentiable distance. Method 500 may further include determining an optimal transport distance (OTD). Method 500 may further include determining the OTD by determining the distance between corresponding labels based on features of data points associated with corresponding labels in the first and second datasets.

[0118] Method 500 may further include, wherein making the first dataset flow includes determining the gradient of the dataset objective relative to the first dataset, and making the first dataset flow further includes updating the first dataset using the determined gradient. Method 500 may further include, wherein the dataset objective includes at least one of: minimizing internal energy, minimizing potential energy, minimizing distance, or minimizing interaction energy.

[0119] Method 500 may further include, wherein the dataset objective includes minimizing the distance between feature label pairs. Method 500 may further include, wherein updating the first dataset using the determined gradients includes using feature-driven updates, jointly driven fixed-label updates, or jointly driven variable-label updates. Method 500 may further include, wherein the first and second datasets comprise image datasets, or the first and second datasets comprise natural language processing (NLP) datasets.

[0120] In this paper, "flow" refers to the iterative application and updating of features (sometimes labels) of data samples using gradient descent techniques constrained by a target. Some experimental results using examples are now provided. The experimental results consider a first dataset (D... ρ Orientation with function The second dataset (D) B )flow.

[0121] Figure 6 A series of graphs 600 are shown as examples of gradient flows at different time steps. Figure 6 The gradient flow example in Figure 600 uses a jointly driven fixed-label technique for gradient flow. In Figure 600, there are two datasets. The first dataset includes sample 660 mapped to label "0", sample 662 mapped to label "1", sample 664 mapped to label "2", sample 666 mapped to label "3", and sample 668 mapped to label "4". The second dataset includes sample 670 mapped to label "A", sample 672 mapped to label "B", sample 674 mapped to label "C", sample 676 mapped to label "D", and sample 678 mapped to label "E". Over time, sample 660 becomes closer to sample 670, sample 662 becomes closer to sample 678, sample 664 becomes closer to sample 672, sample 666 becomes closer to sample 674, and sample 668 becomes closer to sample 676.

[0122] Figure 7 Another series of figures 700 are shown as examples of gradient flows at different time steps. Figure 7 The example of gradient flow in Figure 700 uses a jointly driven variable labeling technique for gradient flow. In Figure 700, the two datasets are compared... Figure 6The datasets in the first dataset are identical, but the first dataset can be mapped to different labels during the gradient flow process. Over time, some data samples in the first dataset include labels different from those at the start of the flow. Some data samples 680 are mapped to label "5", some data samples 682 are mapped to label "6", some data samples 684 are mapped to label "7", some samples 686 are mapped to label "8", and some samples are mapped to label "9".

[0123] Figure 8 The error bar chart 800, denoted here as m, u, f, and k, illustrates different techniques for transfer learning using four image classification datasets (MNIST, USPS, fashion-MNIST, and KMNIST). Bar chart 800 comprises groups 880, 882, 884, 886, 888, and 890, representing six different transfer learning techniques. Each group of bars is described from left to right as follows: (i) no additional learning, (ii) learning on additional data without modification, (iii) learning on modified additional data using a neural network (NN) approximating a gradient flow process, (iv) learning the final state of the additional data flowing toward the target dataset, (v) learning all states of the additional data flowing toward the target dataset (the state of the additional data after each iteration of the flow), and (vi) learning on both the additional data flowing toward the target dataset and the additional data modified using the NN. Group 880 represents transfer learning from MNIST to USPS (using USPS data samples to further train the MNIST-trained model), group 882 represents transfer learning from USPS to MNIST, group 884 represents transfer learning from MNIS to KMNIST, group 886 represents transfer learning from KMNIST to MNIST, group 888 represents transfer learning from MNIST to fashion-MNIST, and group 890 represents transfer learning from fashion-MNIST to MNIST.

[0124] Figure 9 Several series of images from the embodiment are illustrated by way of example, showing images at different times during the gradient flow process. In the first series 992, MNIST data samples flow toward the USPS dataset; in the second series 994, MNIST data samples flow toward the fashion MNIST dataset; in the third series 996, MNIST data samples flow toward the KMNIST dataset; and in the fourth series 998, KMNIST data samples flow toward the MNIST dataset.

[0125] Figure 10A block diagram of an embodiment of a machine 1000 (e.g., a computer system) for implementing one or more embodiments is shown by way of example. An example machine 1000 (in the form of a computer) may include a processing unit 1002, a memory 1003, a removable storage device 1010, and a non-removable storage device 1012. Although the example computing device is shown and described as machine 1000, the computing device may take different forms in different embodiments. For example, the computing device may alternatively be a smartphone, tablet, smartwatch, or include features such as... Figure 10 Other computing devices with the same or similar elements shown and described. Devices such as smartphones, tablets, and smartwatches are generally referred to as mobile devices. In addition, although various data storage elements are shown as part of machine 1000, the storage device may also or alternatively include cloud-based storage devices accessible via a network (such as the Internet).

[0126] Memory 1003 may include volatile memory 1014 and non-volatile memory 1008. Machine 1000 may include a computing environment with or access to various computer-readable media, such as volatile memory 1014 and non-volatile memory 1008, removable storage device 1010, and non-removable storage device 1012. Computer storage devices include random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM) and electrically erasable programmable read-only memory (EEPROM), flash memory or other storage technologies, optical disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical disc storage, cassette tape, magnetic tape, disk storage devices, or other magnetic storage devices capable of storing computer-readable instructions to perform the functions described herein.

[0127] Machine 1000 may include or have access to a computing environment including input 1006, output 1004, and communication interface 1016. Output 1004 may include a display device, such as a touchscreen, which can also be used as an input device. Input 1006 may include one or more of the following: touchscreen, touchpad, mouse, keyboard, camera, one or more device-specific buttons, one or more sensors integrated into machine 1000 or coupled to machine 1000 via wired or wireless data connections, and other input devices. The computer may operate in a networked environment using communication connections to connect to one or more remote computers, such as database servers, including cloud-based servers and storage devices. Remote computers may include personal computers (PCs), servers, routers, network PCs, peer-to-peer devices, or other public network nodes. Communication connections may include local area networks (LANs), wide area networks (WANs), cellular networks, IEEE 802.11 (Wi-Fi), Bluetooth, or other networks.

[0128] Computer-readable instructions stored on a computer-readable storage device are executable by the processing unit 1002 of machine 1000. Hard disk drives, CD-ROMs, and RAM are some examples of items that include non-transitory computer-readable media such as storage devices. For example, computer program 1018 can be used to cause processing unit 1002 to execute one or more methods or algorithms described herein.

[0129] Other notes and examples:

[0130] Example 1 includes a computer-implemented method for machine learning (ML) by causing a dataset to flow toward a target dataset, the method comprising: receiving a request to operate on a first dataset including a first feature label pair; identifying a second dataset from a plurality of datasets, the second dataset including a second feature label pair; determining a distance between the first feature label pair and the second feature label pair; and using a dataset target based on the determined distance to cause the first dataset to flow, thereby generating an optimized dataset.

[0131] In Example 2, Example 1 may also include a feature in which each label is represented by a probability distribution of features mapped to the label in the distance determination.

[0132] In Example 3, at least one of Examples 1-2 may further include: operating a first ML model trained on a second dataset with the modified first dataset as input to provide results.

[0133] In Example 4, at least one of Examples 1-3 may further include: training a second ML model using at least a portion of the second dataset and the modified first dataset.

[0134] In Example 5, at least one of Examples 1-4 may also include: providing a modified first dataset.

[0135] In Example 6, at least one of Examples 3-5 may also include: keeping the first ML model static during training.

[0136] In Example 7, at least one of Examples 1-6 may further include determining the distance, wherein determining the distance includes determining the differentiable distance.

[0137] In Example 8, Example 7 may also include the distance including the optimal transmission distance (OTD).

[0138] In Example 9, Example 8 may further include determining the distance between the first corresponding labels based on features of data points associated with the corresponding labels of the first and second datasets.

[0139] In Example 10, at least one of Examples 1-9 may further include, wherein making the first dataset flow includes determining the gradient of the dataset target relative to the first dataset, and making the first dataset flow also includes updating the first dataset using the determined gradient.

[0140] In Example 11, at least one of Examples 1-10 may further include a dataset objective that includes at least one of the following: minimizing internal energy, minimizing potential energy, minimizing distance, or minimizing interaction energy.

[0141] In Example 12, Example 11 may also include a dataset objective that includes minimizing the distance between feature label pairs.

[0142] In Example 13, Example 10 may also include updating the first dataset using a defined gradient, which includes: using feature-driven updates, jointly driven fixed-label updates, or jointly driven variable-label updates.

[0143] In Example 14, at least one of Examples 1-13 may further include, wherein the first dataset and the second dataset comprise an image dataset, or the first dataset and the second dataset comprise a natural language processing (NLP) dataset.

[0144] Example 15 may include a system comprising memory and processing circuitry configured to implement the methods of any of the examples 1-14.

[0145] Example 16 may include a machine-readable medium comprising instructions that, when executed by a machine, cause the machine to perform a method of any of the examples 1-14.

[0146] While some embodiments have been described in detail above, other modifications are possible. For example, the logical flow shown in the figures does not require the indicated order or sequence to achieve the desired result. Other steps may be provided from the described flow, or steps may be deleted, and other components may be added to or removed from the described system. Other embodiments are within the scope of the following claims.

Claims

1. A computer-implemented method for machine learning (ML) by directing a dataset toward a target dataset, the method comprising: Receive a request to change the first dataset, which includes the first feature-label pairings; Identify or receive a second dataset, the second dataset comprising a second feature-label pair having one or more different features and one or more different labels compared to a first feature-label pair, wherein the first dataset and the second dataset comprise an image dataset, or the first dataset and the second dataset comprise a natural language processing (NLP) dataset; For each label in the first dataset and the second dataset, determine the feature probability distribution of the corresponding label mapped to the first feature-label pair and the second feature-label pair; The corresponding labels in the first feature-label pairing and the second feature-label pairing are replaced with the feature probability distribution mapped to the corresponding label to generate the first feature-feature probability distribution pairing and the second feature-feature probability distribution pairing; Determine the distance between the first feature probability distribution and the second feature probability distribution; as well as The characteristics of the first dataset are modified using a dataset target based on the determined distance operation to generate an optimized first dataset that flows into the domain of the second dataset.

2. The computer-implemented method according to claim 1 further includes: The first ML model trained on the second dataset is operated on using the modified first dataset as input to provide results.

3. The computer-implemented method according to claim 1 further includes: The second ML model is trained using at least a portion of the second dataset and the modified first dataset.

4. The computer-implemented method according to claim 1 further includes: Provide the modified first dataset.

5. The computer-implemented method according to claim 2, further comprising: The first ML model is kept static during training.

6. The computer-implemented method of claim 1, wherein determining the distance includes determining a differentiable distance.

7. The computer-implemented method of claim 6, wherein the distance includes the optimal transport distance (OTD).

8. The computer-implemented method of claim 7, wherein determining the OTD comprises: The distance between the labels is determined based on the characteristics of the data points associated with the corresponding labels of the first and second datasets.

9. The computer-implemented method of claim 8, wherein causing the first dataset to flow includes determining a gradient of the dataset target relative to the second dataset, and causing the first dataset to flow further includes updating the first dataset using the determined gradient.

10. The computer-implemented method of claim 9, wherein the dataset objective includes at least one of the following: minimizing internal energy, minimizing potential energy, minimizing distance, or minimizing interaction energy.

11. The computer-implemented method of claim 10, wherein the dataset objective includes minimizing the distance between feature label pairs.

12. The computer-implemented method of claim 10, wherein updating the first dataset using the determined gradient comprises: Use feature-driven updates, union-driven fixed-label updates, or union-driven variable-label updates.

13. A machine-readable medium comprising instructions that, when executed by a machine, cause the machine to perform operations for machine learning (ML) by causing a dataset to flow toward a target dataset, the operations comprising: Receive a request to change the first dataset, which includes the first feature-label pairings; Identify or receive a second dataset, the second dataset comprising a second feature-label pair having one or more different features and one or more different labels compared to a first feature-label pair, wherein the first dataset and the second dataset comprise an image dataset, or the first dataset and the second dataset comprise a natural language processing (NLP) dataset; For each label in the first dataset and the second dataset, determine the feature probability distribution of the corresponding label mapped to the first feature-label pair and the second feature-label pair; The corresponding labels in the first feature-label pairing and the second feature-label pairing are replaced with the feature probability distribution mapped to the corresponding label to generate the first feature-feature probability distribution pairing and the second feature-feature probability distribution pairing; Determine the distance between the first feature probability distribution and the second feature probability distribution; as well as The characteristics of the first dataset are modified using a dataset target based on the determined distance operation to generate an optimized first dataset that flows into the domain of the second dataset.

14. The machine-readable medium of claim 13, wherein the operation further comprises: Using the modified first dataset as input to operate a first ML model trained on the second dataset to provide results, using at least a portion of the second dataset and the modified first dataset to train a second ML model, or providing the modified first dataset.

15. A supervised machine learning (ML) system for directing a dataset toward a target dataset, the system comprising: Memory, the memory including instructions stored thereon; A processing circuit coupled to the memory, wherein the instructions, when executed by the processing circuit, configure the processing circuit: Receive a request to change the first dataset, which includes the first feature-label pairings; Identify or receive a second dataset, the second dataset comprising a second feature-label pair having one or more different features and one or more different labels compared to a first feature-label pair, wherein the first dataset and the second dataset comprise an image dataset, or the first dataset and the second dataset comprise a natural language processing (NLP) dataset; For each label in the first dataset and the second dataset, determine the feature probability distribution of the corresponding label mapped to the first feature-label pair and the second feature-label pair; The corresponding labels in the first feature-label pairing and the second feature-label pairing are replaced with the feature probability distribution mapped to the corresponding label to generate the first feature-feature probability distribution pairing and the second feature-feature probability distribution pairing; Determine the distance between the first feature probability distribution and the second feature probability distribution; as well as The characteristics of the first dataset are modified using a dataset target based on the determined distance operation to generate an optimized first dataset that flows into the domain of the second dataset.

16. The system of claim 15, wherein the processing circuitry is further configured to: operate a first ML model trained on a second dataset using the modified first dataset as input to provide results, train a second ML model using at least a portion of the second dataset and the modified first dataset, or provide the modified first dataset.

Images