Sparsity-Preserving Differentially Private Training

Abstract

Aspects of the present disclosure are directed to methods, systems, and / or non-transitory computer-readable media for implementing differentially private filtering-enabled sparse training and / or adaptive filtering-enabled sparse training. Differentially private filtering can enable sparse training, and / or adaptive filtering-enabled sparse training can maintain gradient sparsity during training of large embedding models. Training preserves data privacy and maintains accuracy levels comparable to differentially private stochastic gradient descent training while achieving significant reductions in gradient size, e.g., 10 6 can be achieved twice as much.

Description

[Technical Field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims the benefit of the filing date of U.S. Provisional Application No. 63 / 530,084, filed August 1, 2023, the disclosure of which is incorporated herein by reference in its entirety. [Background technology]

[0002] Large-scale embedding models have emerged as a fundamental tool for various applications in recommender systems and natural language processing, including digital content management (e.g., online advertising). For example, in the online advertising domain, a primary objective may be to predict whether a user will perform a useful action on an advertiser's website, e.g., whether they will purchase an advertised product after interacting with an ad on a publisher's website, such as by clicking on the ad. Large-scale embedding models enable the integration of non-numeric data into deep learning models by mapping categorical or string-valued input attributes with large vocabulary to fixed-length vector representations using embedding layers. These models have been widely deployed in personalized recommendation systems and can achieve state-of-the-art performance in linguistic tasks such as language modeling, sentiment analysis, and question answering.

[0003] However, using large-scale embedding models may involve processing user information, which can raise privacy concerns. To enable the analysis of private data, differential privacy has become a widely adopted concept, as it can ensure the privacy of individual user information while still allowing the analysis of population-level patterns. The most widely used methodology for training deep neural networks with guaranteed differential privacy is differentially private stochastic gradient descent (DP-SGD), which clips the gradient contributions of each example and adds noise to the average gradient update during each iteration of stochastic gradient descent. DP-SGD has shown effectiveness in protecting user privacy while maintaining the usefulness of the model in various applications, such as digital content management.

[0004] Nevertheless, implementing DP-SGD for training large-scale embedding models presents unique technical challenges. Large-scale embedding models typically contain non-numeric feature fields, such as product identifiers and categories, as well as words or tokens that are converted to dense vectors via an embedding layer. Due to the large vocabulary size of these features, training can require an embedding table with a significant number of parameters. In contrast to the number of parameters, gradient updates are typically sparse because each mini-batch of examples activates a subset of the embedding rows. This sparsity can be exploited in industrial applications to efficiently handle training large-scale embeddings. However, DP-SGD requires the addition of aggregated independent Gaussian noise to the coordinates, eliminating gradient sparsity. As a result, private training of large-scale embedding models is significantly less efficient than non-private training. Summary of the Invention [Means for solving the problem]

[0005] Aspects of the present disclosure are directed to methods, systems, and / or non-transitory computer-readable media for implementing differentially private filtering-enabled sparse training and / or adaptive filtering-enabled sparse training. Differentially private filtering can enable sparse training, and / or adaptive filtering-enabled sparse training can maintain gradient sparsity during training of large embedding models. Training can be performed with gradient sizes of, for example, 10, while protecting data privacy and maintaining accuracy. 6 A significant reduction of 2x can be achieved.

[0006] An aspect of the present disclosure provides a method for training a sparse machine learning model, the method including: receiving, by one or more processors, a training dataset and a plurality of model parameters; calculating, by the one or more processors, a plurality of example-specific gradient contributions from the training dataset and the plurality of model parameters based on the one or more training parameters; aggregating and adding noise to the plurality of example-specific gradient contributions based on a privacy parameter to generate noisy batchwise gradient contributions; filtering, by the one or more processors, the noisy batchwise gradient contributions based on a frequency parameter to generate filtered batchwise gradient contributions; and updating, by the one or more processors, the plurality of model parameters based on the filtered batchwise gradient contributions to generate a plurality of updated model parameters. Another aspect of the present disclosure provides a system including one or more processors and one or more storage devices coupled to the one or more processors and storing instructions that, when executed by the one or more processors, cause the one or more processors to perform the method for training a sparse machine learning model. Yet another aspect of the present disclosure provides a non-transitory computer-readable medium for storing instructions that, when executed by one or more processors, cause the one or more processors to perform a method for training a sparse machine learning model.

[0007] In an example, the one or more training parameters include at least one of a learning rate, a number of training steps, a batch size, a noise multiplier, or a clipping criterion. In another example, the method further includes clipping, by the one or more processors, the gradient contribution for each of the plurality of examples.

[0008] In yet another example, the method further includes iteratively performing the calculation, aggregation, addition, denoising, and updating for several training steps. In yet another example, the method further includes, after performing several training steps, outputting, by the one or more processors, the trained machine learning model using the trained model parameters.

[0009] In yet another example, each of the multiple per-example gradient contributions includes a per-example gradient and a gradient contribution map. In yet another example, the noisy batchwise gradient contribution includes a private contribution map.

[0010] In yet another example, filtering the noisy batchwise gradient contributions further includes removing example-by-example gradients below a threshold associated with the frequency parameter. In yet another example, the method further includes aggregating, by the one or more processors, the filtered batchwise gradient contributions and adding noise based on the first privacy parameter or the second privacy parameter.

[0011] In yet another example, the sparse machine learning model includes an embedding model. [Brief explanation of the drawings]

[0012] [Figure 1] FIG. 1 illustrates a block diagram of an example adaptive filter according to aspects of the present disclosure. [Figure 2] 1 illustrates a block diagram of an adaptive filtering differential private label training system for training machine learning models with differential private labels, according to aspects of the present disclosure. [Figure 3]1 illustrates a block diagram of an example environment for implementing an adaptive filtering training system according to aspects of the present disclosure. [Figure 4] FIG. 1 illustrates a block diagram illustrating the architecture of one or more machine learning models, according to an embodiment of the present disclosure. [Figure 5] 1 illustrates a flow diagram of an example process of a training step for training one or more sparse machine learning models while ensuring differential privacy, according to aspects of the present disclosure. [Figure 6] 1 illustrates a flow diagram of an example process for training one or more sparse machine learning models while ensuring differential privacy, according to aspects of the present disclosure. [Figure 7] 10 shows an example graph comparing the best gradient size reduction at different thresholds for utility difference, where higher curves indicate better utility-efficiency tradeoff, according to aspects of the present disclosure. [Figure 8] 10 shows an example graph comparing best gradient size reduction with different privacy parameters, according to aspects of the present disclosure. [Figure 9] 10 illustrates an example graph comparing best gradient size reduction for time series data at different streaming periods, according to aspects of the present disclosure. [Figure 10] 10 shows an exemplary graph illustrating the best gradient size reduction for time series data of the combined approach, according to aspects of the present disclosure. [Figure 11] 10 illustrates an example table showing best gradient size reduction for a language model, according to aspects of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0013] As the use of embedding models in recommendation systems and language applications increases, so too does the concern regarding data privacy. Differentially private stochastic gradient descent can be used to train models while preserving data privacy. However, a naive implementation of differentially private stochastic gradient descent for embedding models can destroy the sparsity of the gradients, reducing training efficiency.

[0014] The present technology generally relates to sparsity-preserving differential privacy training, e.g., a sparsity-preserving differential privacy system, that can train machine learning models in a sparsity-preserving manner. The sparsity-preserving differential privacy system can implement differentially private filtering-enabled sparse training and / or adaptive filtering-enabled sparse training, which allows for preserving gradient sparsity during training of large-scale embedding models. Training can be performed on gradient sizes, e.g., 10, while preserving data privacy and maintaining accuracy. 6 A significant reduction of 2x can be achieved.

[0015] Sparsity-preserving differentially private training, such as for training a sparse machine learning model such as an embedding model, includes selecting, by one or more processors, one or more buckets of categorical features for training the sparse machine learning model based on the frequency of the one or more buckets in a dataset, and adding, by one or more processors, noise to the gradients of each of the one or more buckets.

[0016] In an example, selecting one or more buckets includes selecting one or more top-k most frequent buckets for each categorical feature. In another example, selecting one or more buckets is based on historical information regarding bucket frequencies. In yet another example, selecting one or more buckets is based on differentially private top-k selection. In yet another example, the noise includes Gaussian noise.

[0017] In yet another example, selecting one or more buckets further includes computing per-example gradient contributions and a gradient contribution map for each of one or more batches of the dataset, aggregating the per-example gradient contributions across the one or more batches based on the gradient contribution map to generate batch-wise gradient contributions, adding noise to the batch-wise gradient contributions to generate noisy batch-wise gradient contributions, and thresholding the noisy batch-wise gradient contributions based on a frequency parameter to filter out insignificant gradient entries.

[0018] In yet another example, the noise added to the batchwise gradient contributions comprises Gaussian noise. In yet another example, the gradient contribution map comprises a binary vector indicating which categorical feature buckets are activated for each example of the batch. In yet another example, aggregating the per-example gradient contributions further comprises implementing per-example gradient clipping. In yet another example, the frequency parameter is configurable. In yet another example, thresholding the noisy batchwise gradient contributions further comprises retaining buckets of categorical features having noisy batchwise gradient contributions that meet or exceed a threshold based on the frequency parameter. In yet another example, the noise added to the batchwise gradient contributions and the noise added to each gradient have different scales.

[0019] FIG. 1 shows a block diagram of an example adaptive filter 100. The adaptive filter 100 can receive batchwise gradient contributions based on a machine learning model using initial model parameters 102 and filter the batchwise gradient contributions to generate filtered gradient contributions 104. The adaptive filter 100 can filter the batchwise gradient contributions based on a frequency parameter to exclude insignificant gradient entries to focus on relevant and useful features. The machine learning model generator 106 can receive the filtered gradient contributions 104 and update the initial model parameters 102 based on the filtered gradient contributions 104 to generate a machine learning model using updated model parameters 108. The adaptive filter 100 can receive updated batchwise gradient contributions based on the machine learning model using the updated model parameters 108. The adaptive filter 100 and the machine learning model generator 106 can generate the filtered gradient contributions 104 and the machine learning model using the updated model parameters 108, respectively, for several training steps. After several training steps, the machine learning model generator 106 can use the trained model parameters 110 to generate a trained machine learning model.

[0020] 2 shows a block diagram of an adaptive filtering differentially private training system 200 for training machine learning models with differentially private labels. The adaptive filtering training system 200 can be implemented on one or more computing devices at one or more locations.

[0021] The adaptive filtering training system 200 can be configured to receive input data 202. For example, the adaptive filtering training system 200 can receive the input data 202 as part of a call to an application programming interface (API) that exposes the adaptive filtering training system 200 to one or more computing devices. The input data 202 can also be provided to the adaptive filtering training system 200 via a storage medium, such as remote storage, connected to one or more computing devices via a network. The input data 202 can further be provided as input via a user interface at a client computing device coupled to the adaptive filtering training system 200.

[0022] The input data 202 may include training data for training a machine learning model having initial model parameters θ. The input data 202 may further include one or more parameters for training the machine learning model, including a learning rate η, a batch size B, a number of training steps T, one or more clipping criteria (e.g., C1, C2 frequency threshold parameter τ), and one or more noise multipliers (e.g., σ1, σ2). The learning rate may refer to a configurable parameter for controlling how quickly the machine learning model can adapt to a task. The batch size may refer to a configurable parameter for controlling the number of training samples processed during each training step. The clipping criteria may refer to a configurable parameter for controlling the amount of gradient masking during gradient clipping.

[0023] The noise multiplier can be associated with a privacy parameter, δ, which represents the scale of differential privacy implemented during training of the machine learning model. Differential privacy can refer to a framework for ensuring the privacy of individuals in a dataset, providing strong guarantees of privacy by allowing data to be analyzed without revealing information about any individuals in the dataset. Differential privacy is a method for determining the privacy of any two adjacent datasets D and

number

number

number

[0024] Differentially Private Stochastic Gradient Descent (DP-SGD) refers to a methodology for training deep learning models with differential privacy by modifying a mini-batch stochastic optimization process with example-wise gradient clipping and Gaussian noise injection. Given a dataset D, where D is the example-wise loss function

number

number

number

number

number

number

[0025] Although DP-SGD has been proven effective in training ML models using DP, DP-SGD adds noise to every coordinate of the gradient, which completely removes any sparsity structure in the original gradient. This densification of sparse gradients can be problematic for large-scale embedding models, where sparsity is heavily exploited to improve efficiency.

[0026] The input data 202 can be associated with any machine learning task, such as predictive transformation for digital content or other digital content management. The machine learning model can be a sparse model, such as a large-scale embedding model with a large-scale embedding layer for handling high-dimensional sparse input, such as words / tokens in a language model and categorical features associated with users or items in a recommendation system.

[0027] The training data may correspond to a training set D containing examples of categorical features. The training data may be split into a training set, a validation set, and / or a test set. An exemplary train / validation / test split may be an 80 / 10 / 10 split, although any other split may be possible. As an example, for a large-scale embedding model, an input x having the i-th feature may be a one-hot vector that is 1 at the i-th coordinate and 0 otherwise.

number

number

[0028] One-hot input x=e i , the embedding output can be the i-th row of the embedding table, z=W[i,:]. The gradient of the embedding table is

number

number

number

number

[0029] This structured sparsity allows both the forward and backward computations of the embedding layer to be implemented efficiently with gather and scatter, without expensive matrix multiplications, especially given the very large embedding tables, such as vocabulary sizes c in real-world applications that range from tens of thousands in language models to millions in recommendation models. Furthermore, some large-scale recommendation models have hundreds of different categorical features, each with a different embedding table. Therefore, maintaining gradient sparsity allows for a significant reduction in computational complexity, as computationally complex matrix multiplications can be replaced with gather and scatter. Gather and scatter may refer to memory addressing techniques that enable simultaneous gathering, e.g., collecting, or storing, e.g., scattering, of data into multiple arbitrary indices. While the above discussion focuses on univariate features, which activate only one feature value at a time, multivariate features, which activate multiple values, may also implement the techniques generally disclosed herein.

[0030] From the input data 202, the adaptive filtering training system 200 can be configured to output one or more results generated as output data 204. The output data 204 includes the trained model parameters θ TThe output data 204 may include a trained machine learning model having the following characteristics: (a) a trained machine learning model having the following characteristics; (b) a trained machine learning model having the following characteristics; (c) a trained machine learning model having the following characteristics; (d) a trained machine learning model having the following characteristics; (e) a trained machine learning model having the following characteristics; (f) a trained machine learning model having the following characteristics; (g) a trained machine learning model having the following characteristics; (g) a trained machine learning model having the following characteristics; (h) a trained machine learning model having the following characteristics; (i) a trained machine learning model having the following characteristics; (i) a trained machine learning model having the following characteristics; (i) a trained machine learning model having the following characteristics; (i) a trained machine learning model having the following characteristics; (ii) a trained machine learning model having the following characteristics; (iii) a trained machine learning model having the following characteristics; (iv) a trained machine learning model having the following characteristics; (v) a trained machine learning model having the following characteristics; (vi ...

[0031] The adaptive filtering training system 200 may include a gradient calculation engine 206, an aggregation engine 208, a filtering engine 210, and an optimization engine 212. The gradient calculation engine 206, the aggregation engine 208, the filtering engine 210, and the optimization engine 212 may be implemented as one or more computer programs, specially configured electronic circuits, or any combination thereof. The gradient calculation engine 206, the aggregation engine 208, the filtering engine 210, and the optimization engine 212 may perform training steps to update model parameters. During each training step, the adaptive filtering training system 200 adaptively saves the most significant gradients via a configurable frequency parameter.

[0032] For an initial training step, the gradient computation engine 206 may be configured to generate per-example gradient contributions, including per-example gradients and gradient contribution maps, from the training dataset for each mini-batch based on the initial model parameters and one or more parameters. In subsequent training steps, the gradient computation engine 206 may be configured to generate per-example gradients and gradient contribution maps from the training dataset for each mini-batch based on the updated model parameters and one or more parameters. The gradient contribution maps may be binary vectors indicating which categorical feature buckets are activated for each example in the mini-batch. For example, the gradient computation engine 206 may generate a mini-batch B of size B from the training dataset D, such as by uniform or random sampling. t For each mini-batch, e.g., i=1 to B, the gradient computation engine 206 generates the example-wise gradient g i and the gradient contribution map v i can be calculated, where v i [j]:=1[g i [j,:]≠0].

[0033] The aggregation engine 208 can be configured to generate noisy batchwise gradient contributions that ensure differential privacy, including noisy aggregate gradients and private contribution maps from the per-example gradient and gradient contribution maps by aggregating and adding noise and / or clipping. The aggregation engine 208 can aggregate the per-example gradients and gradient contribution maps, respectively, to construct aggregate gradients and aggregate contribution maps. The aggregation engine 208 can further clip the per-example gradients based on one or more clipping criteria before aggregation to ensure that any individual gradient contributions are sufficiently masked. The aggregation engine 208 can add noise, such as Gaussian noise at scale σ, to the aggregate gradients and aggregate contribution maps, respectively, to generate noisy aggregate gradients and noisy aggregate contribution maps that represent the noisy batchwise gradient contributions. For example, the aggregation engine 208 can:

number

[0034] The filtering engine 210 can be configured to threshold the noisy batchwise gradient contributions based on a frequency parameter τ to generate filtered batchwise gradient contributions that filter out insignificant gradient entries with only a few instances in the mini-batch contributions. The frequency parameter τ can be a configurable real number value, such as 1.0, 2.0, or 5.0. The filtering engine 210 can focus training on more relevant and informative features while reducing the influence of noisy or insignificant contributions. [Check whether this is reflected, page 9, bottom two lines] For example, for each mini-batch, e.g., i=1 to B, the filtering engine 210 can perform the following: g i [j,:]←0, for all j, V tNote that higher values ​​of the frequency parameter [j]<τ may reduce the gradient size, resulting in sparser gradients, while excessively high values ​​of the frequency parameter, e.g., greater than 500 for a batch size of 1024, may lead to a sharp decline in model accuracy. Thus, the optimal frequency parameter τ can be determined through empirical evaluation, and the frequency parameter can be configured to provide optimal performance in terms of the trade-off between gradient size and model accuracy depending on the batch size. Thresholding via the filtering engine 210 can reduce the gradient size, and therefore the computational cost, while maintaining accuracy by retaining only gradient entries with scores higher than the frequency parameter to update the model parameters.

[0035] The filtering engine 210 can be further configured to aggregate noise, such as Gaussian noise at scale σ, and add it to the remaining gradients of the filtered batchwise gradient contributions. Note that the scale σ of the noise added by the filtering engine 210 can be the same scale as the scale σ of the noise added by the aggregation engine 208, or can be a different scale. In particular,

number

number

[0036] Alternatively, or additionally, the filtering engine 210 can be configured to select the top-k most informative, e.g., frequent, buckets of each categorical feature and add noise, such as Gaussian noise, only to the gradients of the selected buckets during training. Top-k can be a configurable hyperparameter that refers to a configurable amount of the most frequent buckets, where the configurable amount is equal to k. This frequency filtering can significantly reduce the noise added to the gradients by limiting the noise to only a subset of the most influential features. The filtering engine 210 can select the top-k buckets based on publicly available prior information about bucket frequencies, such as token frequencies in a pre-training set of a machine learning model for a language task. Alternatively, or additionally, the filtering engine 210 can select the top-k buckets by performing differential privacy top-k selection. Differential privacy top-k selection can include counting the frequency of each bucket in a dataset, injecting noise, e.g., Gumbel noise, into the frequency counts, and calculating the top-k buckets based on the noisy frequency counts.

[0037] In the initial and subsequent training steps, the optimization engine 212 can be configured to generate updated model parameters from the initial or previously updated model parameters based on the filtered batchwise gradient contributions and the learning rate. In the final training step, the optimization engine 212 can be configured to generate updated model parameters from the trained model parameters based on the filtered batchwise gradient contributions and the learning rate. For example, the optimization engine 212 can generate updated model parameters from the trained model parameters based on the filtered batchwise gradient contributions and the learning rate. t+1 ←θ t -ηG tThe optimization engine 212 can send the updated parameters back to the gradient calculation engine 206 and output the trained parameters as output data 204.

[0038] 3 shows a block diagram of an exemplary environment 300 for implementing an adaptive filtering training system 318. The adaptive filtering training system 318 can be implemented in one or more devices having one or more processors in one or more locations, such as a server computing device 302. The client computing device 304 and the server computing device 302 can be communicatively coupled to one or more storage devices 306 via a network 308. The storage device 306 may be a combination of volatile and non-volatile memory and may be in the same or a different physical location as the computing devices 302, 304. For example, the storage device 306 may include any type of non-transitory computer-readable medium capable of storing information, such as a hard drive, solid-state drive, tape drive, optical storage, memory card, ROM, RAM, DVD, CD-ROM, writable, and read-only memory.

[0039] The server computing device 302 may include one or more processors 310 and memory 312. The memory 312 may store information accessible by the processor 310, including instructions 314 that may be executed by the processor 310. The memory 312 may also include data 316 that may be retrieved, manipulated, or stored by the processor 310. The memory 312 may be any type of transient or non-transitory computer-readable medium that may store information accessible by the processor 310, such as volatile and non-volatile memory. The processor 310 may include one or more central processing units (CPUs), graphics processing units (GPUs), field programmable gate arrays (FPGAs), and / or application-specific integrated circuits (ASICs), such as tensor processing units (TPUs).

[0040] The instructions 314 may include one or more instructions that, when executed by the processor(s) 310, cause the one or more processors 310 to perform the actions defined by the instructions 314. The instructions 314 may be stored in object code format for direct processing by the processor(s) 310, or in other formats, including interpretable scripts or collections of independent source code modules that are interpreted on-demand or pre-compiled. The instructions 314 may include instructions for implementing an adaptive filtering training system 318, which may correspond to the adaptive filtering training system 200 as shown in FIG. 2. The adaptive filtering training system 318 may be executed using the processor(s) 310 and / or using other processors located remotely from the server computing device 302.

[0041] Data 316 may be retrieved, stored, or modified by processor 310 in accordance with instructions 314. Data 316 may be stored in computer registers, in relational or non-relational databases, as tables with multiple distinct fields and records, or as JSON, YAML, proto, or XML documents. Data 316 may also be formatted in a computer-readable format, such as, but not limited to, binary values, ASCII, or Unicode. Additionally, data 316 may include sufficient information to identify related information, such as numbers, descriptive text, unique codes, pointers, references to data stored in other memory, including other network locations, or information used by a function to calculate the related data.

[0042] The client computing device 304 may also be configured similarly to the server computing device 302, with one or more processors 320, memory 322, instructions 324, and data 326. The client computing device 304 may also include a user input 328 and a user output 330. The user input 328 may include any suitable mechanism or technology for receiving input from a user, such as a keyboard, a mouse, a mechanical actuator, a soft actuator, a touchscreen, a microphone, and a sensor.

[0043] The server computing device 302 may be configured to transmit data to the client computing device 304, which may be configured to display at least a portion of the received data on a display implemented as part of the user output 330. The user output 330 may also be used to display an interface between the client computing device 304 and the server computing device 302. The user output 330 may alternatively or additionally include one or more speakers, transducers or other audio output, a haptic interface, or other haptic feedback that provides non-visual and non-auditory information to a platform user of the client computing device 304.

[0044] While FIG. 3 depicts processors 310, 320 and memories 312, 322 as being within each computing device 302, 304, the components described herein may include multiple processors and memories, which may operate in different physical locations rather than within the same computing device. For example, some of the instructions 314, 324 and data 316, 326 may be stored on a removable SD card, while others may be stored in a read-only computer chip. Some or all of the instructions 314, 324 and data 316, 326 may be stored in locations physically separate from, but still accessible by, the processors 310, 320. Similarly, the processors 310, 320 may include a collection of processors capable of performing simultaneous and / or sequential operations. The computing devices 302, 304 each may include one or more internal clocks that provide timing information that can be used to time operations and programs executed by the computing devices 302, 304.

[0045] The server computing device 302 can be connected via a network 308 to a data center 332 that houses any number of hardware accelerators 334. The data center 332 can be one of multiple data centers or other facilities in which various types of computing devices, such as hardware accelerators, are located. The computing resources housed in the data center 332 can be designated for deploying models, such as personalized recommendation systems, conversion predictions, or any other digital content management, as described herein.

[0046] The server computing device 302 can be configured to receive requests to process data from the client computing devices 304 with computing resources of the data center 332. For example, the environment 300 can be part of a computing platform configured to provide various services to users via various user interfaces and / or application programming interfaces (APIs) that expose platform services. By way of example, the various services can include predicting conversions from digital content interactions, such as whether a purchase of a good or service is completed in response to clicking an advertisement related to the good or service. The client computing device 304 can submit input data as part of a query for a particular task. The adaptive filtering training system 318 can receive the input data and, in response, generate output data including a response to the query for the particular task.

[0047] Server computing device 302 may maintain different models according to different constraints available at data center 332. For example, server computing device 302 may maintain different families for deploying models on different types of TPUs and / or GPUs housed at data center 332 or otherwise available for processing.

[0048] 4 illustrates a block diagram 400 showing one or more machine learning model architectures 402, more specifically architectures 402A-N, for deployment in a data center 404 that houses hardware accelerators 406 on which the deployed machine learning models 402 execute, such as for various services as described herein. The hardware accelerators 406 can be any type of processor, such as a CPU, GPU, FPGA, or ASIC such as a TPU.

[0049] The machine learning model architecture 402 may refer to characteristics that define the model, such as the characteristics of the model's layers, how the layers process inputs, or how the layers interact with each other. The machine learning model architecture 402 may also define the types of operations that occur within each layer. One or more machine learning model architectures 402 may be generated that can output results, such as for a recommendation system, conversion prediction, or any other form of digital content management. An exemplary model architecture 402 may correspond to a large-scale embedded model.

[0050] Machine learning models can be trained according to a variety of different learning techniques. Learning techniques for training machine learning models can include supervised learning, unsupervised learning, semi-supervised learning, and reinforcement learning techniques. For example, training data can include a plurality of training examples that can be received as input by the machine learning model. The training examples can be labeled with a desired output of the model when processing the labeled training examples. The training examples can be labeled with noisy labels that ensure label differential privacy. The noisy labels and the model output can be evaluated through a loss function to determine an error, which can be backpropagated through the machine learning model to update the model weights.

[0051] For example, supervised learning techniques can be applied to calculate the error between the output and the ground truth labels of the training examples processed by the machine learning model. Any of a variety of loss or error functions appropriate to the type of task the model is being trained for may be utilized, such as cross-entropy loss for classification tasks, or Poisson logarithmic loss or squared loss for regression tasks. The gradient of the error for different weights of the candidate model on the candidate hardware can be calculated, for example, using a backpropagation algorithm, and the model weights can be updated.

[0052] Referring back to FIG. 3 , the devices 302, 304 and the data center 332 may be capable of direct and indirect communication over the network 308. For example, the client computing device 304 may use a network socket to connect to a service running on the data center 332 through the Internet Protocol. The devices 302, 304 may set up a listening socket capable of accepting an initiating connection to send or receive information. The network 308 may include a variety of configurations and protocols, including the Internet, the World Wide Web, an intranet, a virtual private network, a wide area network, a local network, and a private network using one or more company-proprietary communication protocols. The network 308 may support a variety of short-range and long-range connections. The short-range and long-range connections may be made over different bandwidths, such as 2.402 GHz to 2.480 GHz, commonly associated with the Bluetooth® standard, 2.4 GHz and 5 GHz, commonly associated with the Wi-Fi® communication protocol, or using various communication standards, such as the LTE® standard for wireless broadband communication. The network 308 may also, or alternatively, support wired connections between the devices 302, 304 and the data center 332, including various types of Ethernet connections.

[0053] 3, it is understood that aspects of the present disclosure can be implemented according to a variety of different configurations and quantities of computing devices, including paradigms for serial or parallel processing, or a distributed network of multiple devices. In some implementations, aspects of the present disclosure can be performed on a single device connected to a hardware accelerator configured to process machine learning models, or any combination thereof.

[0054] 5 shows a flow diagram of an example process 500 of training steps for training one or more sparse machine learning models while ensuring differential privacy. The example process can be performed in a system of one or more processors in one or more locations, such as the adaptive filtering training system 200 shown in FIG.

[0055] As shown in block 510, the adaptive filtering training system 200 receives a training dataset and a plurality of model parameters representing a machine learning model. The training dataset can include a plurality of examples, each of which includes a plurality of features and corresponding labels for the features. The plurality of model parameters can be initial model parameters for a first training step and updated model parameters for subsequent training steps. The machine learning model can be a sparse machine learning model, such as a large-scale embedding model with a large-scale embedding layer for processing high-dimensional sparse inputs, such as words / tokens in a language model and / or categorical features associated with users or items in a recommendation system.

[0056] The adaptive filtering training system 200 may further receive one or more training parameters for training the machine learning model, including a learning rate, a number of training steps, a batch size, one or more noise multipliers associated with a privacy parameter, and / or one or more clipping criteria.

[0057] As shown in block 520, the adaptive filtering training system 200 calculates gradient contributions for each of a plurality of examples from the training dataset and the plurality of model parameters based on one or more training parameters. The adaptive filtering training system 200 can generate mini-batches, e.g., subsets, of the training dataset based on a batch size by uniform or random sampling, and calculate gradients and gradient contribution maps for each example in the mini-batch.

[0058] As shown in block 530, the adaptive filtering training system 200 aggregates the gradient contributions for multiple examples based on a privacy parameter and adds noise to them to generate noisy batchwise gradient contributions that ensure differential privacy. The adaptive filtering training system 200 can aggregate the gradient contribution maps and add noise to generate private contribution maps based on a clipping criterion and a noise multiplier. The adaptive filtering training system 200 can further clip the gradient contributions for each example based on the clipping criterion.

[0059] As shown in block 540, the adaptive filtering training system 200 filters the noisy batchwise gradient contributions based on the frequency parameter to generate filtered batchwise gradient contributions. The adaptive filtering training system 200 can remove example-by-example gradients below a threshold associated with the frequency parameter to focus on more relevant and informative features while reducing the impact of noisy or negligible contributions. The adaptive filtering training system 200 can further aggregate and add noise to the remaining gradients in the filtered batchwise gradient contributions based on a clipping criterion and a noise multiplier.

[0060] As shown in block 550, the adaptive filtering training system 200 updates the plurality of model parameters to generate a plurality of updated model parameters based on the filtered batchwise gradient contributions. The adaptive filtering training system 200 may update the plurality of model parameters further based on the learning rate. In the final training iteration, the adaptive filtering training system 200 may update the plurality of updated model parameters to generate a plurality of trained model parameters based on the filtered batchwise gradient contributions.

[0061] 6 shows a flow diagram of an example process 600 for training one or more sparse machine learning models while ensuring differential privacy. The example process can be performed in a system of one or more processors in one or more locations, such as the adaptive filtering training system 200 shown in FIG.

[0062] As shown in block 610, the adaptive filtering training system 200 may receive a training dataset and a number of initial model parameters associated with the sparse machine learning model to be trained. The adaptive filtering training system 200 may further receive one or more training parameters, such as a learning rate, a number of training steps, a batch size, a noise multiplier associated with a privacy parameter, and / or a clipping criterion, indicating how the sparse machine learning model is trained. Block 610 may generally correspond to block 510 shown in FIG. 5.

[0063] As shown in block 620, the adaptive filtering training system 200 may iteratively update a plurality of initial model parameters based on the training data set and the filtered batchwise gradient contributions to generate a plurality of trained model parameters. The adaptive filtering training system 200 may perform several training steps, generating updated model parameters for each training step based on the received training step number parameter, as shown in Figure 5. The final training step may generate the trained model parameters.

[0064] As indicated in block 630, the adaptive filtering training system 200 may output trained model parameters to represent a trained sparse machine learning model that satisfies differential privacy.

[0065] As shown in Figures 7-11, the adaptive filtering training system disclosed herein can achieve or improve upon alternative approaches for training sparse machine learning models while ensuring differential privacy. We compared both example adaptive filtering training systems, DP-AdaFEST associated with a frequency parameter τ and DP-FEST associated with a top-k parameter, to baseline stochastic gradient descent methods on recommendation and language understanding tasks.

[0066] The adaptive filtering training system was evaluated on Criteo's predicted click-through rate (pCTR) dataset, which contains over 4 billion ad impressions over a 24-day period. Each impression is represented by 13 numerical features and 26 categorical features. The goal is to predict the likelihood that a user will click on an ad based on these features. The pCTR model used for evaluation is a neural network that uses an embedding layer on the categorical features and a logarithmic transformation on the numerical features, followed by several fully connected layers. Binary cross-entropy loss is used as the training objective, and the area under the curve (AUC) is reported as the evaluation metric.

[0067] For the evaluation, we use two Criteo variants: Criteo-Kaggle and Criteo-time-series. Criteo-Kaggle refers to a subset of the approximately 45 million examples used in click-through rate modeling studies. Note that Criteo-Kaggle does not have a timestamp. Criteo-time-series, or Criteo-1TB, is the full Criteo dataset of over 4 billion examples, with auxiliary information indicating the days on which the data was collected. To simulate a real-world online training scenario, models are trained on the first 18 days of data and evaluated on subsequent days, e.g., days 19–24.

[0068] For the language understanding task, a language model from the BERT family, specifically the RoBERTa model, is used. This model has a vocabulary size of 50,265 subword tokens and is pre-trained on public web data. The RoBERTa model is further fine-tuned for downstream classification tasks from GLUE standards, including SST-2, QNLI, and QQP. Low-Rank Adaptation (LoRA) is employed to introduce trainable rank decomposition matrices into each transformer block of the language model. This approach significantly reduces the number of parameters required for downstream tasks and also improves the privacy-utility tradeoff. The word embedding layer is also trained with DP fine-tuning to enable bucket selection, which significantly improves the accuracy of the model.

[0069] We evaluate the performance of DP-SGD and three sparsity-preserving variants, exponentially selected DP-SGD, DP-FEST, and DP-AdaFEST, on both the click-through rate prediction and language understanding tasks. A fixed batch size of 2,048 is used for the former, and a fixed batch size of 1,024 is used for the latter. For all tasks, the privacy parameter δ is set to 1 / N, where N is the number of training examples for the respective task.

[0070] First, we consider the non-timestamped datasets Criteo-Kaggle, SST-2, QNLI, and QQP. The decision to prioritize gradient sparsity, which impacts computational efficiency, or utility in DP training depends on the task and available computational resources. However, DP-SGD can have difficulty addressing various needs for gradient sparsity or utility because it lacks a mechanism for tradeoffs between these objectives. In contrast, exponential selection DP-SGD, DP-AdaFEST, and DP-FEST all offer diverse options for balancing utility and efficiency via sparsity control parameters, but DP-AdaFEST and DP-FEST offer significantly better privacy-utility tradeoffs. While both DP-FEST and DP-AdaFEST offer ways to balance efficiency and utility in DP training, DP-AdaFEST stands out as a more versatile and customizable methodology. With three adjustable hyperparameters, DP-AdaFEST provides more diverse results compared to DP-FEST's single parameter k, which controls the number of top buckets preserved.

[0071] The effectiveness of DP-AdaFEST is evident in the graph in Figure 7, where it achieves a significantly higher gradient size reduction than DP-FEST while maintaining the same level of utility. Specifically, on the Criteo-Kaggle dataset, DP-AdaFEST reduces the gradient computation cost of DP-SGD by 5 × 10 while maintaining a comparable AUC, e.g., an AUC loss of less than 0.005. 5This reduction translates into a more efficient and cost-effective training process. The adoption of sparsity-preserving DP-SGD effectively eliminates the need for dense gradient computation. Furthermore, in line with the bias-variance tradeoff, DP-AdaFEST can also demonstrate superior utility compared to DP-SGD when the reduction in gradient size is minimal. Conversely, when incorporating sparsity, DP-SGD with its exponential mechanism faces challenges in preserving utility. It is unable to achieve an acceptable level of utility loss in all configurations.

[0072] Further performance improvements can be achieved by integrating DP-AdaFEST with DP-FEST, which involves preselecting a subset of buckets using DP-FEST and then training on this subset using DP-AdaFEST. The graph in Figure 8 shows that the combined methodology, called DP-AdaFEST+, outperforms either method alone in terms of utility / efficiency tradeoff, with the best gradient size reduction being 10 6 We show further improvements of over 1x. This can be attributed to the complementary strengths of the two approaches: DP-AdaFEST offers a more flexible approach to batch-level feature selection, while DP-FEST provides a simple yet effective means for feature selection via global frequency information. Furthermore, the combined methodology broadens the scope of choices for balancing gradient sparsity and usefulness through the combination of hyperparameters from both DP-FEST and DP-AdaFEST.

[0073] Time-series data are notoriously challenging due to non-stationarity. The performance of DP-FEST and DP-AdaFEST is evaluated in adapting to the distribution shifts characteristic of time-series data. Evaluations are performed using the Criteo-time-series dataset, which contains real-world user click data collected over a 24-day period, with 18 days for training and the remaining days for evaluation. A streaming period is introduced to simulate online data streaming scenarios. A streaming period is the time interval during which the model is updated as new data is received. The graph in Figure 9 illustrates the tradeoff between the usefulness and efficiency of DP-AdaFEST and DP-FEST for time-series data. To investigate the effectiveness of DP-FEST, various sources of lexical frequency are evaluated, including first-day, all-day, and streaming-based information, e.g., running totals updated per streaming period. The evaluation shows that using streaming-based frequency information for DP-FEST is nearly as effective as using all-day frequency information and significantly better than using only information from the first day. Furthermore, DP-AdaFEST consistently outperforms DP-FEST, achieving over 2x gradient reduction at the same level of utility. The benefits of combining DP-AdaFEST and DP-FEST are evaluated to enhance the utility / efficiency tradeoff on the Criteo-time-series dataset, where streaming period 1 and streaming-based frequency information from DP-FEST are used. The graph in Figure 10 shows that the combined approach consistently outperforms the individual methods.

[0074] The applicability of DP-AdaFEST to language models is also evaluated by comparing it with LoRA and reporting its performance on multilingual models. DP-AdaFEST is shown to be a better choice than LoRA for the embedding layer. The table in Figure 11 compares the gradient size reduction of the best embeddings achieved by DP-AdaFEST and LoRA relative to DP-SGD for SST-2 with ε = 1.0 on the RoBERTa model. The rank r of LoRA varies from {4, 8, 16, 32, 64, 128}. DP-AdaFEST consistently outperforms LoRA in gradient size reduction at similar utility levels.

[0075] Aspects of the present disclosure may be implemented in digital electronic circuitry, tangibly embodied computer software or firmware, and / or computer hardware, such as the structures disclosed herein, their structural equivalents, or combinations thereof. Aspects of the present disclosure may further be implemented as one or more computer programs, e.g., one or more modules of computer program instructions, encoded on a tangible, non-transitory computer storage medium for execution by or control the operation of one or more data processing devices. The computer storage medium may be a computer-readable storage device, a computer-readable storage substrate, or a random access memory or serial access memory device, or a combination thereof. The computer program instructions may be encoded on an artificially generated propagated signal, such as a mechanically generated electrical, optical, or electromagnetic signal, generated to encode information for transmission to a receiving device suitable for execution by a data processing device.

[0076] The term "configured" is used herein in reference to systems and computer program components. One or more computer systems configured to perform particular operations or actions means that the system has installed thereon software, firmware, hardware, or a combination thereof that causes the system to perform the operations or actions. One or more computer programs configured to perform particular operations or actions means that the one or more programs contain instructions that, when executed by one or more data processing devices, cause the devices to perform the operations or actions.

[0077] The terms "data processing apparatus" or "data processing system" refer to data processing hardware and encompass various apparatus, devices, and machines for processing data, including programmable processors, computers, or combinations thereof. A data processing apparatus may include special-purpose logic circuitry, such as a field programmable gate array (FPGA) or an application-specific integrated circuit (ASIC). A data processing apparatus may include code that creates an execution environment for a computer program, such as code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or combinations thereof.

[0078] The term "computer program" refers to a program, software, software application, app, module, software module, script, or code. A computer program can be written in any form of programming language, including compiled, interpreted, declarative, or procedural languages, or a combination thereof. A computer program can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A computer program can correspond to a file in a file system, can be stored in a portion of a file that holds other programs or data, such as one or more scripts stored in a markup language document, in a single file dedicated to the program, or in multiple cooperating files, such as files that store one or more modules, subprograms, or portions of code. A computer program can be executed on one computer or on multiple computers located at one site or distributed across multiple sites and interconnected by a data communications network.

[0079] The term "database" refers to any collection of data. The data can be unstructured or structured in any manner. The data can be stored on one or more storage devices in one or more locations. For example, an index database can contain multiple collections of data, each of which can be organized and accessed in a different way.

[0080] The term "engine" refers to a software-based system, subsystem, or process that is programmed to perform one or more specific functions. An engine can be implemented as one or more software modules or components, or can be installed on one or more computers in one or more locations. A particular engine may have one or more computers dedicated to it, or multiple engines may be installed and running on the same computer or computers.

[0081] The processes and logic flows described herein can be performed by one or more computers executing one or more computer programs to perform functions by operating on input data and generating output data. The processes and logic flows can also be performed by special purpose logic circuitry, or a combination of special purpose logic circuitry and one or more computers.

[0082] A computer or special-purpose logic circuit that executes one or more computer programs may include a central processing unit, including a general-purpose or special-purpose microprocessor, for performing or executing instructions, and one or more memory devices for storing instructions and data. The central processing unit may receive instructions and data from one or more memory devices, such as read-only memory, random-access memory, or a combination thereof, and may perform or execute the instructions. The computer or special-purpose logic circuit may also include, or be operatively coupled to, one or more storage devices for storing data, such as magnetic, magneto-optical, or optical disks, and may receive or transfer data. The computer or special-purpose logic circuit may be embedded in another device, such as a mobile phone, personal digital assistant (PDA), mobile audio or video player, game console, global positioning system (GPS), or portable storage device, such as a universal serial bus (USB) flash drive.

[0083] A computer-readable medium suitable for storing one or more computer programs may include any form of volatile or non-volatile memory, medium, or memory device, including, for example, a semiconductor memory device such as an EPROM, EEPROM, or flash memory device, a magnetic disk such as an internal hard disk or a removable disk, a magneto-optical disk, a CD-ROM disk, a DVD-ROM disk, or any combination thereof.

[0084] Aspects of the present disclosure can be implemented in a computing system that includes back-end components, e.g., data servers, middleware components, e.g., application servers, or front-end components, e.g., client computers having graphical user interfaces, web browsers, or apps, or any combination thereof. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include a local area network (LAN) and a wide area network (WAN), e.g., the Internet.

[0085] A computing system may include clients and servers. Clients and servers may be remote from each other and interact through a communications network. The relationship of client and server arises by virtue of computer programs running on the respective computers and by virtue of the client-server relationship to each other. For example, a server may send data, e.g., HTML pages, to a client device for the purpose of displaying the data to and receiving user input from a user interacting with the client device, for example. Data generated at the client device, e.g., results of user interaction, may be received from the client device by the server.

[0086] Unless otherwise stated, the foregoing alternative examples are not mutually exclusive and can be implemented in various combinations to achieve inherent advantages. Because these and other variations and combinations of the above features can be utilized without departing from the subject matter defined by the claims, the foregoing description of embodiments should be construed as illustrative, and not limiting, of the subject matter defined by the claims. Furthermore, the provision of examples described herein, as well as clauses using terms such as "such as," "including," and the like, should not be construed as limiting the subject matter of the claims to any particular example. Rather, the example is intended to illustrate only one of many possible embodiments. Furthermore, the same reference symbols on different drawings may identify the same or similar elements.

Claims

1. 1. A method for training a sparse machine learning model, comprising: receiving, by one or more processors, a training data set and a plurality of model parameters; calculating, by the one or more processors, gradient contributions for each of a plurality of examples from the training data set and the plurality of model parameters based on one or more training parameters; aggregating, by the one or more processors, the gradient contributions for each of the plurality of examples and adding noise based on a first privacy parameter to generate noisy batchwise gradient contributions; filtering, by the one or more processors, the noisy batchwise gradient contributions based on a frequency parameter to generate filtered batchwise gradient contributions; and updating, by the one or more processors, the plurality of model parameters based on the filtered batchwise gradient contributions to generate a plurality of updated model parameters.

2. The method of claim 1 , wherein the one or more training parameters include at least one of a learning rate, a number of training steps, a batch size, a noise multiplier, or a clipping criterion.

3. The method of claim 1 , further comprising clipping, by the one or more processors, gradient contributions for each of the plurality of examples.

4. The method of claim 1 , further comprising: iteratively performing the calculating, aggregating, adding, denoising, and updating for several training steps.

5. 5. The method of claim 4, further comprising, after performing the number of training steps, outputting, by the one or more processors, a trained machine learning model using trained model parameters.

6. The method of claim 1 , wherein each of the plurality of example-specific gradient contributions comprises an example-specific gradient and gradient contribution map.

7. The method of claim 1 , wherein the noisy batchwise gradient contributions comprise a private contribution map.

8. The method of claim 1 , wherein filtering the noisy batchwise gradient contributions further comprises removing example-by-example gradients below a threshold associated with the frequency parameter.

9. 2. The method of claim 1, further comprising aggregating, by the one or more processors, the filtered batchwise gradient contributions and adding noise based on the first privacy parameter or a second privacy parameter.

10. The method of claim 1 , wherein the sparse machine learning model comprises an embedding model.

11. 1. A system comprising: one or more processors; one or more storage devices coupled to the one or more processors and storing instructions that, when executed by the one or more processors, cause the one or more processors to perform operations for training a sparse machine learning model, the operations including: receiving a training data set and a plurality of model parameters; calculating a gradient contribution for each of a plurality of examples from the training data set and the plurality of model parameters based on one or more training parameters; aggregating and adding noise to the gradient contributions for the plurality of examples based on a first privacy parameter to generate noisy batchwise gradient contributions; filtering the noisy batchwise gradient contributions based on a frequency parameter to generate filtered batchwise gradient contributions; and updating the plurality of model parameters based on the filtered batchwise gradient contributions to generate a plurality of updated model parameters.

12. The system of claim 11 , wherein the one or more training parameters include at least one of a learning rate, a number of training steps, a batch size, a noise multiplier, or a clipping criterion.

13. The system of claim 11 , wherein the operations further comprise clipping the gradient contribution for each of the plurality of examples.

14. The system of claim 11 , wherein the operations further include iteratively performing the calculating, aggregating, adding, denoising, and updating for several training steps.

15. 15. The system of claim 14, wherein the operations further include, after performing the number of training steps, outputting a trained machine learning model using trained model parameters.

16. The system of claim 11 , wherein each of the plurality of example-specific gradient contributions comprises an example-specific gradient and gradient contribution map.

17. The system of claim 11 , wherein the noisy batchwise gradient contributions include a private contribution map.

18. The system of claim 11 , wherein filtering the noisy batchwise gradient contributions further comprises removing example-by-example gradients below a threshold associated with the frequency parameter.

19. The system of claim 11 , wherein the operations further comprise aggregating the filtered batchwise gradient contributions and adding noise based on the first privacy parameter or the second privacy parameter.

20. A computer program that, when executed by one or more processors, causes the one or more processors to perform operations for training a sparse machine learning model, the operations comprising: receiving a training data set and a plurality of model parameters; calculating a gradient contribution for each of a plurality of examples from the training data set and the plurality of model parameters based on one or more training parameters; aggregating and adding noise to the gradient contributions for the plurality of examples based on a privacy parameter to generate noisy batchwise gradient contributions; filtering the noisy batchwise gradient contributions based on a frequency parameter to generate filtered batchwise gradient contributions; and updating the plurality of model parameters based on the filtered batchwise gradient contributions to generate a plurality of updated model parameters.

Images