Differential private training to maintain sparsity

Abstract

Aspects of the present disclosure relate to methods, systems, and / or non-transitory computer readable media for implementing differential private filtering implemented sparse training and / or adaptive filtering implemented sparse training. Differential private filtering may enable sparse training, and / or sparse training implemented by adaptive filtering may maintain gradient sparsity during training of a large embedded model. This training can protect data privacy and achieve a significant reduction in gradient size, such as 106 times, while maintaining a level of accuracy comparable to differential private stochastic gradient descent training.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of the filing date of U.S. Provisional Patent Application No. 63 / 530,084, filed on August 1, 2023, the disclosure of which is incorporated herein by reference. Background Art

[0003] Large embedding models have become an essential tool for various applications in recommender systems and natural language processing, such as digital content management like online advertising. For example, in the field of online advertising, the main goal can be to predict whether a user will perform a useful action on an advertiser's website, such as purchasing the advertised product, after interacting with an advertisement on the publisher's website (e.g., clicking on the advertisement). Large embedding models use embedding layers to map categorical or string-valued input attributes with a large vocabulary to fixed-length vector representations, which enables the integration of non-numeric data into deep learning models. These models are widely deployed in personalized recommender systems and can achieve state-of-the-art performance in language tasks such as language modeling, sentiment analysis, and question answering.

[0004] However, the use of large embedding models can raise privacy concerns as it may involve the processing of user information. To enable private data analysis, differential privacy is a widely adopted concept that can guarantee the privacy of individual user information while still allowing analysis of group-level patterns. To train deep neural networks with differential privacy guarantees, the most widely used method is Differentially Private Stochastic Gradient Descent (DP-SGD), which clips the per-example gradient contribution during each iteration of stochastic gradient descent and adds noise to the average gradient update. DP-SGD has demonstrated its effectiveness in protecting user privacy while maintaining model utility in a variety of applications such as digital content management.

[0005] However, implementing DP-SGD when training large embedding models presents unique technical challenges. Large embedding models often contain non-numeric feature fields, such as product identifiers and categories, as well as words or tokens that are transformed into dense vectors by the embedding layer. Due to the large vocabulary size of these features, training can require embedding tables with a large number of parameters. In contrast to the number of parameters, gradient updates are typically sparse, as each mini-batch of examples activates a small fraction of the embedding rows. This sparsity can be used for industry applications to efficiently handle large-scale embedding training. However, DP-SGD removes gradient sparsity because it requires aggregation and adding independent Gaussian noise to the coordinates, resulting in private training of large embedding models with significantly reduced training efficiency compared to non-private training. Summary of the invention

[0006] Aspects of the present disclosure relate to methods, systems, and / or non-transitory computer-readable media for implementing differentially private filtering enabled sparse training and / or adaptive filtering enabled sparse training. Differentially private filtering can enable sparse training, and / or adaptive filtering enabled sparse training can maintain gradient sparsity during training of large embedding models. The training can preserve data privacy and achieve significant reductions in gradient size, e.g., 10 6 times while maintaining accuracy.

[0007] One aspect of the present disclosure provides a method for training a sparse machine learning model, the method comprising: receiving a training data set and a plurality of model parameters by one or more processors; calculating a plurality of per-sample gradient contributions from the training data set and the plurality of model parameters by the one or more processors based on one or more training parameters; aggregating and adding noise to the plurality of per-sample gradient contributions by the one or more processors based on a privacy parameter to generate a noisy batch-by-batch gradient contribution; filtering the noisy batch-by-batch gradient contribution by the one or more processors based on a frequency parameter to generate a filtered batch-by-batch gradient contribution; and updating the plurality of model parameters by the one or more processors based on the filtered batch-by-batch gradient contribution to generate a plurality of updated model parameters. Another aspect of the present disclosure provides a system, the system comprising: one or more processors; and one or more storage devices coupled to the one or more processors and storing instructions, which, when executed by the one or more processors, cause the one or more processors to execute a method for training a sparse machine learning model. Yet another aspect of the present disclosure provides a non-transitory computer-readable medium for storing instructions, which, when executed by one or more processors, cause the one or more processors to execute a method for training a sparse machine learning model.

[0008] In one example, the one or more training parameters include at least one of a learning rate, a number of training steps, a batch size, a noise multiplier, or a clipping norm. In another example, the method further includes clipping, by one or more processors, a plurality of per-example gradient contributions.

[0009] In yet another example, the method further comprises iteratively performing calculations, aggregations, additions, denoising, and updates for a plurality of training steps. In yet another example, the method further comprises outputting, by one or more processors, a trained machine learning model having trained model parameters after performing the plurality of training steps.

[0010] In yet another example, each of the plurality of per-example gradient contributions comprises a per-example gradient and a gradient contribution map. In yet another example, the noisy batch-by-batch gradient contribution comprises a private contribution map.

[0011] In yet another example, filtering the noisy batch-by-batch gradient contribution further comprises removing per-sample gradients below a threshold associated with a frequency parameter. In yet another example, the method further comprises aggregating, by one or more processors, based on the first privacy parameter or the second privacy parameter and adding noise to the filtered batch-by-batch gradient contribution.

[0012] In yet another example, the sparse machine learning model includes an embedding model. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1Depicted is a block diagram of an example adaptive filter according to aspects of the present disclosure.

[0014] Figure 2 Depicted is a block diagram of an adaptive filtered differentially private training system for training a machine learning model with differentially private labels in accordance with aspects of the present disclosure.

[0015] Figure 3 Depicted is a block diagram of an example environment for implementing an adaptive filter training system according to aspects of the present disclosure.

[0016] Figure 4 Depicted is a block diagram illustrating one or more machine learning model architectures according to aspects of the present disclosure.

[0017] Figure 5 Depicted is a flowchart of an example process for training one or more sparse machine learning models while ensuring differential privacy during training steps according to aspects of the present disclosure.

[0018] Figure 6 Depicted is a flow diagram of an example process for training one or more sparse machine learning models while ensuring differential privacy according to aspects of the present disclosure.

[0019] Figure 7 Depicted are example graphs comparing optimal gradient size reductions at different thresholds for utility difference, where higher curves indicate better utility-efficiency tradeoffs, according to aspects of the present disclosure.

[0020] Figure 8 Depicted are example graphs comparing optimal gradient size reduction under different privacy parameters in accordance with aspects of the present disclosure.

[0021] Fig. 9 Depicted is an example graph comparing optimal gradient size reduction for time series data with different streaming periods in accordance with aspects of the present disclosure.

[0022] Fig.10 Depicted is an example graph showing optimal gradient magnitude reduction for time series data for a combined approach in accordance with aspects of the present disclosure.

[0023] Fig.11 Depicted is an example table showing optimal gradient size reduction for a language model in accordance with aspects of the present disclosure. DETAILED DESCRIPTION

[0024] With the increasing use of embedding models in recommender systems and language applications, concerns about data privacy have also increased. Differentially private stochastic gradient descent can be exploited to train models while preserving data privacy. However, naively implementing differentially private stochastic gradient descent to embedding models may destroy gradient sparsity, leading to reduced training efficiency.

[0025] The present technology generally relates to sparsity-preserving differentially private training, such as for a sparsity-preserving differentially private system capable of training machine learning models in a sparsity-preserving manner. The sparsity-preserving differentially private system can enable sparse training of differentially private filtering implementations and / or sparse training of adaptive filtering implementations, thereby allowing gradient sparsity to be preserved during training of large embedding models. The training can preserve data privacy and achieve significant reductions in gradient size, such as 10 6 times while maintaining accuracy.

[0026] Sparsity-preserving differentially private training of a sparse machine learning model, such as for training a similar embedding model, includes: selecting, by one or more processors, one or more buckets of categorical features for training the sparse machine learning model based on the frequency of the one or more buckets in a dataset; and adding, by the one or more processors, noise to corresponding gradients of the one or more buckets.

[0027] In one example, selecting one or more buckets includes selecting one or more top-k most frequent buckets for each classification feature. In another example, selecting one or more buckets is based on historical information about bucket frequencies. In yet another example, selecting one or more buckets is based on differentially private top-k selections. In yet another example, the noise includes Gaussian noise.

[0028] In yet another example, selecting one or more buckets further includes: for each batch of one or more batches of the data set, calculating a per-example gradient contribution and a gradient contribution map; aggregating the per-example gradient contribution across the one or more batches based on the gradient contribution map to generate a batch-by-batch gradient contribution; adding noise to the batch-by-batch gradient contribution to generate a noisy batch-by-batch gradient contribution; and thresholding the noisy batch-by-batch gradient contribution based on a frequency parameter to exclude non-significant gradient entries.

[0029] In yet another example, the noise added to the batch-by-batch gradient contribution comprises Gaussian noise. In yet another example, the gradient contribution map comprises a binary vector indicating which categorical feature buckets are activated for each example in the batch. In yet another example, aggregating per-example gradient contributions further comprises implementing per-example gradient clipping. In yet another example, the frequency parameter is configurable. In yet another example, thresholding the noisy batch-by-batch gradient contribution further comprises retaining categorical feature buckets having noisy batch-by-batch gradient contributions that meet or exceed a threshold based on the frequency parameter. In yet another example, the noise added to the batch-by-batch gradient contribution and the noise added to the corresponding gradient have different scales.

[0030] Figure 1 A block diagram of an example adaptive filter 100 is depicted. The adaptive filter 100 may receive batch-by-batch gradient contributions 102 based on a machine learning model having initial model parameters, and filter the batch-by-batch gradient contributions to generate filtered gradient contributions 104. The adaptive filter 100 may filter the batch-by-batch gradient contributions based on a frequency parameter to exclude non-significant gradient entries for focusing on relevant and informative features. The machine learning model generator 106 may receive the filtered gradient contributions 104, and update the initial model parameters 102 based on the filtered gradient contributions 104 to generate a machine learning model having updated model parameters 108. The adaptive filter 100 may receive updated batch-by-batch gradient contributions based on the machine learning model having updated model parameters 108. The adaptive filter 100 and the machine learning model generator 106 may generate the filtered gradient contributions 104 and the machine learning model having updated model parameters 108, respectively, for a plurality of training steps. After a plurality of training steps, the machine learning model generator 106 may generate a trained machine learning model having trained model parameters 110.

[0031] Figure 2 A block diagram of an adaptive filtering differentially private training system 200 for training a machine learning model using differentially private labels is depicted. The adaptive filtering training system 200 can be implemented on one or more computing devices in one or more locations.

[0032] The adaptive filtering training system 200 can be configured to receive input data 202. For example, the adaptive filtering training system 200 can receive the input data 202 as part of a call to an application programming interface (API) that exposes the adaptive filtering training system 200 to one or more computing devices. The input data 202 can also be provided to the adaptive filtering training system 200 via a storage medium, such as a remote storage connected to the one or more computing devices via a network. The input data 202 can also be provided as input via a user interface on a client computing device coupled to the adaptive filtering training system 200.

[0033] The input data 202 may include the data for training the model with initial model parameters θ o The input data 202 may also include one or more parameters for training the machine learning model, the one or more parameters including a learning rate η, a batch size B, a number of training steps T, one or more clipping norms (e.g., C 1 , C 2 ), a frequency threshold parameter τ, and one or more noise multipliers (e.g., σ 1 , σ 2 ). A learning rate may refer to a configurable parameter for controlling how quickly a machine learning model can adapt to a task. A batch size may refer to a configurable parameter for controlling the number of training samples processed during each training step. A clipping norm may refer to a configurable parameter for controlling how much of a gradient is masked when gradient clipping.

[0034] The noise multiplier can be associated with privacy parameters ε and δ that represent the scale of differential privacy achieved during training of the machine learning model. Differential privacy can refer to a framework for ensuring the privacy of individuals in a dataset that provides strong guarantees of privacy by allowing the data to be analyzed without revealing information about any individual in the dataset. When for any two adjacent datasets D and D′, such as datasets such that one dataset can be obtained from the other by adding or removing an example, and any subset S of the outputs, the privacy parameter ε∈R ＞0 and δ∈[0,1) are applicable to the following formula: PrPr[A(D)∈S]≤e ε ·Pr Pr[A(D′)∈S]+δ, differential privacy can be satisfied for A.

[0035] Differentially private stochastic gradient descent (DP-SGD) refers to a method for training deep learning models with differential privacy by modifying the mini-batch stochastic optimization process using per-example gradient clipping and Gaussian noise injection. When a machine learning model f parameterized by θ is trained on a dataset D using a per-example loss function l(·,·), each optimization step t can involve performing a mini-batch B tPerform random sampling. It should be noted that the specific loss function depends on the specific task and model. An example loss function can be the cross entropy loss for classification. From B t First, for each (x i ,y i )∈B t Calculate the gradient for each example, where x i is a eigenvector, and y i is the corresponding label. For example, the gradient of each example is calculated as follows: Clip the per-sample gradient based on the clipping norm C. For example, clip the per-sample gradient by: By injecting Gaussian noise into the sum of clipped per-example gradients, such as by: To generate private gradients Where N(0,σ 2 C 2 I) is a function with mean 0 and covariance σ 2 C 2 I is a Gaussian distribution, and the noise multiplier σ is calculated from (ε, δ), such as by inverse privacy accounting.

[0036] While DP-SGD has been shown to be effective for training ML models with DP, DP-SGD adds noise to all coordinates of the gradient, which completely removes any sparsity structure in the original gradient. This densification of sparse gradients can be problematic for large embedding models where sparsity is heavily exploited to improve efficiency.

[0037] The input data 202 may be associated with any machine learning task such as predicting transformations of digital content or other digital content management. The machine learning model may be a sparse model, such as a large embedding model with a large embedding layer for processing high-dimensional sparse inputs such as words / tokens in a language model and categorical features associated with users or items in a recommendation system.

[0038] The training data may correspond to a training set D, which includes examples of categorical features. The training data may be split into a training set, a validation set, and / or a test set. An example training / validation / test split may be an 80 / 10 / 10 split, but any other split is possible. As an example, for a large embedding model, an input x with the i-th feature may be represented as a one-hot vector e with 1 at the i-th coordinate and 0 elsewhere. i ∈R c, where c is the number of possible values ​​of the input feature, such as the vocabulary size of a text model. Feature values ​​can also be called feature buckets, since string-valued categorical features can be preprocessed via hash map bucketing. The output of the embedding layer can be computed as a linear map z = W T x, where W∈R c×d can be a parameter of the embedding layer, such as an embedding table, and the output dimension d can be the embedding size.

[0039] For one-hot input x=e i , the embedding output can be the i-th row of the embedding table: z = W[i,:]. The gradient of the embedding table can be in can be the outer product, and It can be the partial derivative of the loss with respect to the embedding output. For a one-hot input x = e i , the result of this outer product is that its i-th row is equal to is a sparse matrix of . For mini-batch stochastic gradient descent training, the number of non-zero rows in the batch average gradient of the embedding table is upper bounded by the batch size, which is typically several orders of magnitude smaller than c. Thus, for large embedding models, gradient sparsity (e.g., the fraction of zero gradient coordinates) can be high.

[0040] Due to this structured sparsity, both the forward and backward computations of the embedding layer can be efficiently implemented using gathering and scattering without expensive matrix multiplications, especially considering that in real-world applications, the embedding tables are very large, such as the vocabulary c ranges from tens of thousands in language models to millions in recommendation models. In addition, in some large recommendation models, there are hundreds of different classification features, each with a different embedding table. Therefore, maintaining gradient sparsity allows for a significant reduction in computational complexity, as computationally complex matrix multiplications can be replaced with gathering and scattering. Gathering and scattering can refer to memory addressing techniques that enable data to be collected (e.g., gathered) or stored (e.g., scattered) to multiple arbitrary indices simultaneously. It should be noted that the above description focuses on univariate features that activate only one feature value at a time, but multivariate features that activate multiple values ​​can also implement the techniques disclosed herein as a whole.

[0041] Based on the input data 202, the adaptive filtering training system 200 can be configured to output one or more results generated as output data 204. The output data 204 may include a trained model parameter θ TThe trained machine learning model. As an example, the adaptive filtering training system 200 can be configured to send output data 204 for display on a client or user display. As another example, the adaptive filtering training system 200 can be configured to provide the output data 204 as a set of computer-readable instructions, such as one or more computer programs. The computer program can be written in any type of programming language and according to any programming paradigm (e.g., declarative, procedural, assembly, object-oriented, data-oriented, functional, or imperative). The computer program can be written to perform one or more different functions and operate within a computing environment, such as on a physical device, on a virtual machine, or across multiple devices. The computer program can also implement the functions described herein, such as by a system, engine, module, or model. The adaptive filtering training system 200 can also be configured to forward the output data 204 to one or more other devices configured to convert the output data into an executable program written in a computer programming language. The adaptive filtering training system 200 can also be configured to send the output data 204 to a storage device for storage and later retrieval.

[0042] The adaptive filtering training system 200 may include a gradient calculation engine 206, an aggregation engine 208, a filtering engine 210, and an optimization engine 212. The gradient calculation engine 206, the aggregation engine 208, the filtering engine 210, and the optimization engine 212 may be implemented as one or more computer programs, specially configured electronic circuit systems, or any combination thereof. The gradient calculation engine 206, the aggregation engine 208, the filtering engine 210, and the optimization engine 212 may perform training steps to update model parameters. In each training step, the adaptive filtering training system 200 adaptively maintains the most important gradients through a configurable frequency parameter.

[0043] For the initial training step, the gradient calculation engine 206 can be configured to generate a per-sample gradient contribution from the initial model parameters and each small batch of training data sets based on one or more parameters, and the per-sample gradient contribution includes a per-sample gradient and a gradient contribution map. For subsequent training steps, the gradient calculation engine 206 can be configured to generate a per-sample gradient and a gradient contribution map from the updated model parameters and each small batch of training data sets based on one or more parameters. The gradient contribution map can be a binary vector that indicates which classification feature buckets have been activated for each sample in the small batch. For example, the gradient calculation engine 206 can generate a small batch B of size B from the training data set D, such as by uniform or random sampling. t For each mini-batch, for example, i=1 to B, the gradient calculation engine 206 may calculate the gradient g for each example i and the gradient contribution map v i , where v i [j]:=1[gi [j,:]≠0].

[0044] The aggregation engine 208 can be configured to generate a noisy batch-by-batch gradient contribution that ensures differential privacy from each sample gradient and gradient contribution map by aggregating and adding noise and / or clipping, and the noisy batch-by-batch gradient contribution includes a noisy aggregated gradient and a private contribution map. The aggregation engine 208 can aggregate each sample gradient and the gradient contribution map separately to construct an aggregated gradient and an aggregated contribution map. The aggregation engine 208 can also clip each sample gradient based on one or more clipping norms before aggregation to ensure that the contribution of any individual gradient is fully masked. The aggregation engine 208 can separately add noise (such as noise with scale σ 1 The aggregation engine 208 may add the aggregated gradient and the aggregated contribution graph to generate a noisy aggregated gradient and a noisy aggregated contribution graph representing the noisy batch-by-batch gradient contribution. For example, the aggregation engine 208 may calculate the noisy aggregated contribution graph as This noisy aggregate contribution graph may also be referred to as a private contribution graph.

[0045] The filtering engine 210 can be configured to threshold the noisy batch-by-batch gradient contributions based on a frequency parameter τ to generate filtered batch-by-batch gradient contributions to exclude non-significant gradient entries to which only a few examples in the mini-batch contribute. The frequency parameter τ can be a configurable real value, such as 1.0, 2.0, 5.0, etc. The filtering engine 210 can focus training on more relevant and informative features while reducing the impact of noisy or non-significant contributions. For example, for each mini-batch, e.g., i=1 to B, the filtering engine 210 can perform the following operations for all j: g i [j,:]←0, where V t [j] < τ. It should be noted that higher values ​​of the frequency parameter can reduce the gradient size, resulting in sparser gradients, but too high values ​​of the frequency parameter, such as greater than 500 for a batch size of 1024, can lead to a sharp drop in model accuracy. In this way, the optimal frequency parameter τ can be determined by empirical evaluation, where the frequency parameter can be configured to provide optimal performance in terms of the trade-off between gradient size and model accuracy depending on the batch size. Thresholding via the filtering engine 210 can reduce the gradient size, and therefore reduce computational cost, while maintaining accuracy by only keeping gradient entries that score higher than the frequency parameter for updating model parameters.

[0046] The filtering engine 210 can also be configured to aggregate and convert noise (such as having a scale σ 2 The remaining gradients are added to the filtered batch-by-batch gradient contributions by Gaussian noise (σ). It should be noted that the scale of the noise added by the filtering engine 210 is 2can be a measure of the noise added by the aggregation engine 208 1 Same scale or different scale. Of particular note, A larger ratio of may result in high accuracy, but may also result in a higher gradient density. The filter engine 210 may also be configured to clip the remaining gradients based on one or more clipping norms. It should also be noted that the clipping norm C used by the filter engine 210 is 2 The clipping norm C used by the aggregation engine 208 can be 1 The same or different. For example, the filtering engine 210 can be configured to generate filtered batch-by-batch gradient contributions as follows: The filtering engine 210 allows for adaptive feature selection while maintaining differential privacy guarantees.

[0047] Alternatively or in addition, the filtering engine 210 can be configured to select the top k most informative (e.g., most frequent) buckets for each categorical feature and add noise, such as Gaussian noise, to the gradients of only the selected buckets during training. The top k can be a configurable hyperparameter that refers to a configurable amount of buckets with the highest frequency, where the configurable amount is equal to k. This frequency filtering can significantly reduce the noise added to the gradient by limiting the noise to only the most influential subset of features. The filtering engine 210 can select the top k buckets based on prior information about publicly available bucket frequencies, such as word-meta frequencies in a pre-trained set of machine learning models for language tasks. Alternatively or in addition, the filtering engine 210 can select the top k buckets by running a differentially private top k selection. The differentially private top k selection can include counting the frequency of each bucket in the data set, injecting noise (e.g., Gumbel noise) into the frequency counts, and calculating the top k buckets based on the noisy frequency counts.

[0048] For the initial and subsequent training steps, the optimization engine 212 may be configured to generate updated model parameters from the initial or previously updated model parameters based on the filtered batch-by-batch gradient contributions and the learning rate. For the final training step, the optimization engine 212 may be configured to generate trained model parameters from the updated model parameters based on the filtered batch-by-batch gradient contributions and the learning rate. For example, the optimization engine 212 may calculate θ t+1 ←θ t -ηG t The optimization engine 212 may send the updated parameters back to the gradient calculation engine 206 and may output the trained parameters as output data 204 .

[0049] Figure 3A block diagram of an example environment 300 for implementing an adaptive filter training system 318 is depicted. The adaptive filter training system 318 may be implemented on one or more devices having one or more processors in one or more locations, such as in a server computing device 302. The client computing devices 304 and the server computing devices 302 may be communicatively coupled to one or more storage devices 306 via a network 308. The storage devices 306 may be a combination of volatile and non-volatile memory and may be located in the same or different physical locations as the computing devices 302, 304. For example, the storage devices 306 may include any type of non-transitory computer-readable medium capable of storing information, such as a hard drive, a solid-state drive, a tape drive, optical storage, a memory card, ROM, RAM, DVD, CD-ROM, writable and read-only memory.

[0050] The server computing device 302 may include one or more processors 310 and memory 312. The memory 312 may store information accessible by the processor 310, including instructions 314 executable by the processor 310. The memory 312 may also include data 316 that may be retrieved, manipulated, or stored by the processor 310. The memory 312 may be a type of transient or non-transitory computer-readable medium capable of storing information accessible by the processor 310, such as volatile and non-volatile memory. The processor 310 may include one or more central processing units (CPUs), graphics processing units (GPUs), field programmable gate arrays (FPGAs), and / or application specific integrated circuits (ASICs), such as tensor processing units (TPUs).

[0051] Instructions 314 may include one or more instructions that, when executed by processor 310, cause one or more processors 310 to perform actions defined by instructions 314. Instructions 314 may be stored in an object code format for direct processing by processor 310 or in other formats including a collection of interpretable scripts or independent source code modules that may be interpreted on demand or pre-compiled. Instructions 314 may include instructions for implementing an adaptive filter training system 318 that may be used with an adaptive filter training system 318. Figure 2 The depicted adaptive filter training system 200 corresponds. The adaptive filter training system 318 may be executed using the processor 310 and / or using another processor remotely located from the server computing device 302.

[0052] Data 316 may be retrieved, stored, or modified by processor 310 in accordance with instructions 314. Data 316 may be stored in a computer register, in a relational or non-relational database as a table with a plurality of different fields and records, or as a JSON, YAML, proto, or XML document. Data 316 may also be formatted in a computer readable format, such as, but not limited to, binary values, ASCII, or Unicode. In addition, data 316 may include information sufficient to identify the relevant information, such as a number, descriptive text, proprietary code, pointer, reference to data stored in other storage (including other network locations), or information used by a function to calculate the relevant data.

[0053] The client computing device 304 may also be configured similarly to the server computing device 302, with one or more processors 320, memory 322, instructions 324, and data 326. The client computing device 304 may also include user input 328 and user output 330. The user input 328 may include any suitable mechanism or technology for receiving input from a user, such as a keyboard, a mouse, a mechanical actuator, a soft actuator, a touch screen, a microphone, and a sensor.

[0054] The server computing device 302 may be configured to transmit data to the client computing device 304, and the client computing device 304 may be configured to display at least a portion of the received data on a display implemented as part of the user output 330. The user output 330 may also be used to display an interface between the client computing device 304 and the server computing device 302. The user output 330 may alternatively or additionally include one or more speakers, transducers or other audio outputs, haptic interfaces, or other tactile feedback that provides non-visual and non-audible information to a platform user of the client computing device 304.

[0055] Although Figure 3 Processors 310, 320 and memories 312, 322 are shown as being within respective computing devices 302, 304, but the components described herein may include multiple processors and memories that may operate at different physical locations rather than within the same computing device. For example, some of instructions 314, 324 and data 316, 326 may be stored on removable SD cards and other cards within read-only computer chips. Some or all of instructions 314, 324 and data 316, 326 may be stored in a location physically away from processors 310, 320 but still accessible by the processors. Similarly, processors 310, 320 may include a collection of processors that may perform concurrent and / or sequential operations. Computing devices 302, 304 may each include one or more internal clocks that provide timing information that may be used for time measurement of operations and programs run by computing devices 302, 304.

[0056] The server computing device 302 may be connected to a data center 332 housing any number of hardware accelerators 334 via a network 308. The data center 332 may be one of a plurality of data centers or other facilities where various types of computing devices, such as hardware accelerators, are located. The computing resources housed in the data center 332 may be designated for deploying models, such as for personalized recommendation systems, conversion prediction, or any other digital content management, as described herein.

[0057] The server computing device 302 may be configured to receive a request to process data from the client computing device 304 on computing resources in the data center 332. For example, the environment 300 may be part of a computing platform configured to provide a variety of services to users through various user interfaces and / or application programming interfaces (APIs) that expose platform services. As an example, the various services may include predicting conversions based on digital content interactions, such as whether a purchase of an item or service is completed in response to clicking on an advertisement associated with an item or service. The client computing device 304 may send input data as part of a query for a particular task. The adaptive filter training system 318 may receive the input data and, in response, generate output data including a response to the query for the particular task.

[0058] The server computing device 302 may maintain a variety of models according to different constraints available at the data center 332. For example, the server computing device 302 may maintain different families for deploying models on various types of TPUs and / or GPUs housed in the data center 332 or available for processing.

[0059] Figure 4 A block diagram 400 is depicted showing one or more machine learning model architectures 402 (more specifically, 402A to 402N for each architecture) for deployment in a data center 404, the data center housing a hardware accelerator 406 on which the deployed machine learning model 402 will be executed, such as for various services as described herein. The hardware accelerator 406 can be any type of processor, such as a CPU, GPU, FPGA, or ASIC such as a TPU.

[0060] The architecture 402 of a machine learning model may refer to characteristics that define the model, such as characteristics of the layers of the model, how the layers process inputs, or how the layers interact with each other. The architecture 402 of a machine learning model may also define the types of operations performed within each layer. One or more machine learning model architectures 402 may be generated, which may output results such as for a recommendation system, conversion prediction, or any other form of digital content management. The example model architecture 402 may correspond to a large embedding model.

[0061] The machine learning model can be trained according to a variety of different learning techniques. The learning techniques used to train the machine learning model can include supervised learning, unsupervised learning, semi-supervised learning, and reinforcement learning techniques. For example, the training data can include a plurality of training examples that can be received as input by the machine learning model. The training examples can be labeled with the expected output of the model when processing the labeled training examples. The training examples can be labeled with noisy labels that ensure label differential privacy. The noisy labels and model outputs can be evaluated by a loss function to determine an error, which can be back-propagated through the machine learning model to update the weights of the model.

[0062] For example, supervised learning techniques may be applied to compute the error between outputs using ground truth labels for training examples processed by the machine learning model. Any of a variety of loss or error functions appropriate to the type of task for which the model is being trained may be utilized, such as cross entropy loss for classification tasks, or Poisson log loss or squared loss for regression tasks. Error gradients with respect to different weights of a candidate model on candidate hardware may be computed, for example using a back-propagation algorithm, and the weights of the model may be updated.

[0063] Return to reference Figure 3 , devices 302, 304 and data center 332 may be able to communicate directly and indirectly over network 308. For example, using network sockets, a client computing device 304 may connect to a service operating in data center 332 over the Internet Protocol. Devices 302, 304 may establish listening sockets that may accept initiated connections for sending and receiving information. Network 308 may include a variety of configurations and protocols, including the Internet, the World Wide Web, an intranet, a virtual private network, a wide area network, a local area network, and a private network using one or more company-proprietary communication protocols. Network 308 may support a variety of short-range and long-range connections. The short-range and long-range connections may be made over different bandwidths, such as those typically associated with a wireless network. 2.402GHz to 2.480GHz, usually associated with the 2.4 GHz and 5 GHz associated with communication protocols; or using multiple communication standards, such as for wireless broadband communication Additionally or alternatively, the network 308 may also support wired connections between the devices 302, 304 and the data center 332, including through various types of Ethernet connections.

[0064] Although in Figure 3A single server computing device 302, client computing device 304, and data center 332 are shown in FIG, but it should be understood that aspects of the present disclosure may be implemented in accordance with a variety of different configurations and numbers of computing devices (including in paradigms for sequential or parallel processing), or through a distributed network of multiple devices. In some implementations, aspects of the present disclosure may be performed on a single device or any combination thereof connected to a hardware accelerator configured for processing machine learning models.

[0065] Figure 5 A flowchart of an example process 500 for training one or more sparse machine learning models while ensuring differential privacy during training steps is depicted. The example process can be executed on a system of one or more processors in one or more locations, such as Figure 2 The adaptive filtering training system 200 depicted in .

[0066] As shown in block 510, the adaptive filtering training system 200 receives a training data set and a plurality of model parameters representing a machine learning model. The training data set may include a plurality of examples, each example including a plurality of features and corresponding labels for the plurality of features. The plurality of model parameters may be initial model parameters for a first training step and updated model parameters for subsequent training steps. The machine learning model may be a sparse machine learning model, such as a large embedding model having a large embedding layer for processing high-dimensional sparse inputs, such as words and / or tokens in a language model and / or categorical features associated with users or items in a recommendation system.

[0067] The adaptive filtering training system 200 may also receive one or more training parameters for training the machine learning model. The one or more training parameters may include a learning rate, a number of training steps, a batch size, one or more noise multipliers associated with a privacy parameter, and / or one or more clipping norms.

[0068] As shown in block 520, the adaptive filtering training system 200 calculates a plurality of per-example gradient contributions from the training dataset and the plurality of model parameters based on one or more training parameters. The adaptive filtering training system 200 may generate mini-batches (e.g., subsets) of the training dataset by uniform or random sampling based on a batch size, and calculates gradients and gradient contribution maps for each example in the mini-batch.

[0069] As shown in block 530, the adaptive filtering training system 200 aggregates and adds noise to a plurality of per-example gradient contributions based on a privacy parameter to generate noisy batch-by-batch gradient contributions that guarantee differential privacy. The adaptive filtering training system 200 may aggregate and add noise to the gradient contribution map based on a clipping norm and a noise multiplier to generate a private contribution map. The adaptive filtering training system 200 may also clip the per-example gradient contribution based on the clipping norm.

[0070] As shown in block 540, the adaptive filtering training system 200 filters the noisy batch-by-batch gradient contributions based on the frequency parameter to generate filtered batch-by-batch gradient contributions. The adaptive filtering training system 200 can remove per-example gradients below a threshold associated with the frequency parameter to focus on more relevant and informative features while reducing the impact of noisy or insignificant contributions. The adaptive filtering training system 200 can also aggregate and add noise to the remaining gradients in the filtered batch-by-batch gradient contributions based on a clipped norm and a noise multiplier.

[0071] As shown in block 550, the adaptive filtering training system 200 updates the plurality of model parameters based on the filtered batch-by-batch gradient contributions to generate a plurality of updated model parameters. The adaptive filtering training system 200 may further update the plurality of model parameters based on the learning rate. For the final training iteration, the adaptive filtering training system 200 may update the plurality of updated model parameters based on the filtered batch-by-batch gradient contributions to generate a plurality of trained model parameters.

[0072] Figure 6 A flowchart of an example process 600 for training one or more sparse machine learning models while ensuring differential privacy is depicted. The example process can be performed on a system of one or more processors in one or more locations, such as Figure 2 The adaptive filtering training system 200 depicted in .

[0073] As shown in block 610, the adaptive filtering training system 200 may receive a plurality of initial model parameters associated with a sparse machine learning model to be trained and a training data set. The adaptive filtering training system 200 may also receive one or more training parameters indicating how the sparse machine learning model is to be trained, such as a learning rate, a number of training steps, a batch size, a noise multiplier associated with a privacy parameter, and / or a clipping norm. Block 610 may generally be associated with Figure 5 The illustrated block 510 corresponds.

[0074] As shown in block 620, the adaptive filtering training system 200 may iteratively update the plurality of initial model parameters based on the training data set and the filtered batch-by-batch gradient contributions to generate a plurality of trained model parameters. The adaptive filtering training system 200 may perform the following based on the received training step number parameter: Figure 5 The depicted multiple training steps are performed to generate updated model parameters for each training step. The final training step can generate trained model parameters.

[0075] As shown in block 630 , the adaptive filtering training system 200 may output trained model parameters to represent a trained sparse machine learning model that satisfies differential privacy.

[0076] like Figures 7 to 11 As shown, the adaptive filtering training system disclosed in this article can implement or improve alternative methods for training sparse machine learning models while ensuring differential privacy. Examples of the adaptive filtering training system (both the example DP-AdaFEST associated with the frequency parameter τ and the example DP-FEST associated with the top k parameter) are compared with baseline stochastic gradient descent on recommendation and language understanding tasks.

[0077] The adaptive filtering training system was evaluated on the Criteo predicted click-through rate (pCTR) dataset, which consists of more than 4 billion ad impressions over a 24-day period. Each impression is represented by 13 numerical features and 26 categorical features. The goal is to predict the likelihood of a user clicking on an ad based on these features. The pCTR model used for evaluation is a neural network that uses an embedding layer for categorical features and a logarithmic transformation for numerical features, followed by several fully connected layers. Binary cross entropy loss is used as the training objective, and the area under the curve (AUC) is reported as the evaluation metric.

[0078] Two Criteo variants are utilized for evaluation: Criteo-Kaggle and Criteo-time-series. Criteo-Kaggle refers to a subset of examples used in the study of click-through rate modeling, totaling approximately 45 million. Note that there are no timestamps in Criteo-Kaggle. Criteo-time-series, or Criteo-1TB, is the entire Criteo dataset of over 4 billion examples, with auxiliary information indicating the day the data was collected. To simulate a real-world online training scenario, the model is trained on the first 18 days of data, and evaluated on subsequent days (e.g., days 19-24).

[0079] For language understanding tasks, a language model from the BERT family, specifically the RoBERTa model, is used. The model has a vocabulary of 50,265 subword tokens and has been pre-trained on public web data. The RoBERTa model is also fine-tuned for downstream classification tasks based on the GLUE benchmark (which includes SST-2, QNLI, and QQP). Low rank adaptation (LoRA) is used to introduce a trainable rank decomposition matrix into each transformer block of the language model. This method significantly reduces the number of parameters required for downstream tasks while also improving the privacy-utility trade-off. The word embedding layer is also trained in DP fine-tuning to achieve bucket selection. This leads to significant accuracy improvements in the model.

[0080] The performance of DP-SGD and three sparsity-preserving variants (DP-SGD with exponential selection, DP-FEST, and DP-AdaFEST) are evaluated on both the CTR prediction task and the language understanding task. A fixed batch size of 2,048 is used for the CTR prediction task and a fixed batch size of 1,024 is used for the language understanding task. For all tasks, the privacy parameter δ is set to 1 / N, where N is the number of training examples for the corresponding task.

[0081] First, consider the datasets without timestamps: Criteo-Kaggle, SST-2, QNLI, and QQP. The decision to prioritize gradient sparsity or utility that affects computational efficiency in DP training depends on the task and available computing resources. However, it may be challenging for DP-SGD to cater to different needs of gradient sparsity or utility because DP-SGD lacks a mechanism for weighing these goals. In contrast, DP-SGD, DP-AdaFEST, and DP-FEST with exponential selection can all provide various options for balancing utility and efficiency via sparsity control parameters, while DP-AdaFEST and DP-FEST provide much better privacy-utility loss. Although both DP-FEST and DP-AdaFEST provide ways to balance efficiency and utility in DP training, DP-AdaFEST stands out as a more general and customizable method. Using three adjustable hyperparameters, DP-AdaFEST provides more different results than the single parameter k of the number of the first few buckets kept by the control of DP-FEST.

[0082] The effectiveness of DP-AdaFEST is Figure 7 This is evident in the plot of , where it achieves significantly higher gradient size reduction than DP-FEST while maintaining the same level of utility. Specifically, on the Criteo-Kaggle dataset, DP-AdaFEST reduces the gradient computation cost of DP-SGD by 5×105 times while maintaining comparable AUC, e.g., AUC loss is less than 0.005. This reduction translates into a more efficient and cost-effective training process. The adoption of DP-SGD with sparsity preservation effectively avoids dense gradient computations. Moreover, consistent with the bias-variance trade-off, DP-AdaFEST can also exhibit superior utility compared to DP-SGD when the reduction in gradient magnitude is minimized. In contrast, DP-SGD with an exponential mechanism faces challenges in maintaining utility when incorporating sparsity. It fails to achieve a tolerable level of utility loss in all configurations.

[0083] The performance can also be enhanced by integrating DP-AdaFEST with DP-FEST, which involves pre-selecting a subset of buckets using DP-FEST and subsequently training on this subset using DP-AdaFEST. Figure 8 The graph in Figure 2 shows that the combined method, called DP-AdaFEST+, outperforms the individual methods in terms of utility / efficiency tradeoff, where the optimal gradient size reduction is further improved to >10 6 ×. This can be attributed to the complementary strengths of the two methods: DP-AdaFEST provides a more flexible way to perform feature selection at the batch level, while DP-FEST provides a simple and effective means of feature selection based on global frequency information. In addition, the combined method expands the range of choices for balancing gradient sparsity and utility by combining hyperparameters from both DP-FEST and DP-AdaFEST.

[0084] Time series data is notoriously challenging due to its non-stationarity. The performance of DP-FEST and DP-AdaFEST is evaluated in terms of adapting to distribution shifts, a characteristic of time series data. The evaluation is performed using the Criteo-time-series dataset, which consists of real-world user click data collected over 24 days (18 days for training and the remaining days for evaluation). To simulate an online data streaming scenario, a streaming period is introduced, i.e., the time interval at which the model is updated when new data is received. Fig. 9The graphs in represent the utility and efficiency tradeoffs of DP-AdaFEST and DP-FEST for time series data. To explore the effectiveness of DP-FEST, different sources of word frequency are evaluated, including information from the first day, all days, and streaming-based information, such as a running sum updated for each streaming period. The evaluation indicates that using streaming-based frequency information for DP-FEST is almost as effective as using frequency information for all days, and is significantly better than using information from the first day alone. In addition, DP-AdaFEST continues to outperform DP-FEST, achieving more than twice the gradient reduction at the same utility level. The advantages of combining DP-AdaFEST and DP-FEST are evaluated to enhance the utility / efficiency tradeoff for the Criteo-time-series dataset. A streaming period of 1 for DP-FEST and streaming-based frequency information are used. Fig.10 The graphs in demonstrate the continued superiority of the combined approach over the individual approaches.

[0085] The applicability of DP-AdaFEST to language models is also evaluated by comparing it with LoRA and reporting its performance utilizing multi-lingual models. DP-AdaFEST is shown to be a better choice than LoRA for the embedding layer. Fig.11 The table in compares the best embedding gradient size reduction achieved by DP-AdaFEST and LoRA for SST-2 versus DP-SGD on the RoBERTa model with ε=1.0. The rank r of LoRA varies from {4, 8, 16, 32, 64, 128}. DP-AdaFEST consistently outperforms LoRA in terms of gradient size reduction at similar utility levels.

[0086] Aspects of the present disclosure may be implemented in digital electronic circuit systems, or in tangibly embodied computer software or firmware and / or in computer hardware (such as the structures disclosed herein, their structural equivalents, or combinations thereof). Aspects of the present disclosure may also be implemented as one or more computer programs, such as one or more modules of computer program instructions encoded on a tangible, non-transitory computer storage medium, for one or more data processing devices to execute or control the operation of the one or more data processing devices. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination thereof. Computer program instructions may be encoded on an artificially generated propagation signal (such as a machine-generated electrical, optical, or electromagnetic signal), generating the artificially generated propagation signal to encode information for transmission to a suitable receiver device for execution by a data processing device.

[0087] The term "configuration" is used herein in conjunction with system and computer program components. For a system of one or more computers configured to perform specific operations or actions, it is meant that the system has installed thereon software, firmware, hardware, or a combination thereof that causes the system to perform those operations or actions. For one or more computer programs configured to perform specific operations or actions, it is meant that the one or more programs include instructions that, when executed by one or more data processing devices, cause the devices to perform those operations or actions.

[0088] The term "data processing apparatus" or "data processing system" refers to data processing hardware and encompasses a variety of devices, apparatuses, and machines for processing data, including programmable processors, computers, or combinations thereof. Data processing apparatus may include dedicated logic circuitry, such as a field programmable gate array (FPGA) or an application specific integrated circuit (ASIC). Data processing apparatus may include code that creates an execution environment for a computer program, such as code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination thereof.

[0089] The term "computer program" refers to a program, software, software application, app, module, software module, script or code. A computer program can be written in any form of programming language, including compiled, interpreted, declarative or procedural languages ​​or a combination thereof. A computer program can be deployed in any form, including as a stand-alone program or as a module, component, subroutine or other unit suitable for a computing environment. A computer program may correspond to a file in a file system and may be stored in a portion of a file that holds other programs or data (such as one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple coordinated files (such as files that store one or more modules, subroutines or code portions). A computer program may be executed on one computer, or on multiple computers located at one site or distributed over multiple sites and interconnected by a data communications network.

[0090] The term "database" refers to any collection of data. The data may be unstructured, or structured in any manner. The data may be stored on one or more storage devices in one or more locations. For example, an index database may include multiple collections of data, each of which may be organized and accessed differently.

[0091] The term "engine" refers to a software-based system, subsystem, or process that is programmed to perform one or more specific functions. An engine can be implemented as one or more software modules or components or can be installed on one or more computers in one or more locations. A specific engine can have its own dedicated computer or computers, or multiple engines can be installed and run on the same computer or computers.

[0092] The processes and logic flows described herein may be performed by one or more computers executing one or more computer programs to perform functions by operating on input data and generating output data. The processes and logic flows may also be performed by a dedicated logic circuit system or by a combination of a dedicated logic circuit system and one or more computers.

[0093] A computer or dedicated logic circuit system that executes one or more computer programs may include a central processing unit (including a general or special microprocessor) for running or executing instructions, and one or more memory devices for storing instructions and data. The central processing unit can receive instructions and data from one or more memory devices (such as read-only memory, random access memory, or a combination thereof), and can run or execute these instructions. The computer or dedicated logic circuit system may also include or be operatively coupled to one or more storage devices (such as magnetic disks, magneto-optical disks, or optical disks) for storing data to receive data from the storage device or transfer data to the storage device. The computer or dedicated logic circuit system may be embedded in another device, such as, for example, a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS), or a portable storage device, such as a universal serial bus (USB) flash drive.

[0094] Computer-readable media suitable for storing one or more computer programs may include any form of volatile or non-volatile memory, medium, or storage device. Examples include semiconductor memory devices (e.g., EPROM, EEPROM, or flash memory devices), magnetic disks (e.g., internal hard disks or removable disks), magneto-optical disks, CD-ROM disks, DVD-ROM disks, or combinations thereof.

[0095] Aspects of the invention may be implemented in a computing system that includes a back-end component (e.g., as a data server), a middleware component (e.g., an application server), or a front-end component (e.g., a client computer with a graphical user interface, a web browser or an app, or any combination thereof). The components of the system may be interconnected by any form or medium of digital data communication, such as a communication network. Examples of communication networks include local area networks (LANs) and wide area networks (WANs), such as the Internet.

[0096] A computing system may include a client and a server. The client and server may be remote from each other and interact through a communication network. The relationship of client and server arises from computer programs running on respective computers and having a client-server relationship to each other. For example, a server may transmit data (e.g., an HTML page) to a client device (e.g., for the purpose of displaying data to a user interacting with the client device and receiving user input from the user). Data generated at the client device (e.g., the result of a user interaction) may be received from the client device at the server.

[0097] Unless otherwise stated, the foregoing alternative examples are not mutually exclusive, but may be implemented in various combinations to achieve unique advantages. Since these and other variations and combinations of the features discussed above can be utilized without departing from the subject matter defined by the claims, the previous description of the embodiments should be made in an illustrative manner rather than in a manner that limits the subject matter defined by the claims. In addition, the examples described herein and the provision of clauses expressed as "such as", "including", etc. should not be interpreted as limiting the subject matter of the claims to specific examples; on the contrary, these examples are intended to illustrate only one of many possible embodiments. In addition, the same reference numerals in different figures may identify the same or similar elements.

Claims

1. A method for training a sparse machine learning model, comprising: Receiving, by one or more processors, a training data set and a plurality of model parameters; calculating, by the one or more processors, a plurality of per-example gradient contributions from the training data set and the plurality of model parameters based on one or more training parameters; aggregating and adding noise to the plurality of per-example gradient contributions based on a privacy parameter by the one or more processors to generate noisy batch-by-batch gradient contributions; filtering, by the one or more processors, the noisy batch-by-batch gradient contribution based on a frequency parameter to generate a filtered batch-by-batch gradient contribution; as well as The plurality of model parameters are updated, by the one or more processors, based on the filtered batch-by-batch gradient contributions to generate a plurality of updated model parameters.

2. The method of claim 1, wherein the one or more training parameters include at least one of a learning rate, a number of training steps, a batch size, a noise multiplier, or a clipping norm.

3. The method of claim 1, further comprising clipping, by the one or more processors, the plurality of per-example gradient contributions.

4. The method of claim 1, further comprising iteratively performing the calculating, the aggregating, the adding, the denoising, and the updating for a plurality of training steps.

5. The method of claim 4 further comprises outputting, by the one or more processors, a trained machine learning model having trained model parameters after executing the plurality of training steps.

6. The method of claim 1, wherein each of the plurality of per-sample gradient contributions comprises a per-sample gradient and a gradient contribution map.

7. The method of claim 1, wherein the noisy batch-by-batch gradient contributions comprise a private contribution map.

8. The method of claim 1, wherein filtering the noisy batch-by-batch gradient contributions further comprises removing per-example gradients below a threshold associated with the frequency parameter.

9. The method of claim 1, further comprising aggregating, by the one or more processors, and adding noise to the filtered batch-by-batch gradient contributions based on the first privacy parameter or the second privacy parameter.

10. The method of claim 1, wherein the sparse machine learning model comprises an embedding model.

11. A system comprising: one or more processors; as well as one or more storage devices coupled to the one or more processors and storing instructions that, when executed by the one or more processors, cause the one or more processors to perform operations for training a sparse machine learning model, the operations comprising: Receive a training dataset and multiple model parameters; Calculating a plurality of per-example gradient contributions from the training dataset and the plurality of model parameters based on one or more training parameters; aggregating and adding noise to the plurality of per-example gradient contributions based on a privacy parameter to generate noisy batch-by-batch gradient contributions; filtering the noisy batch-by-batch gradient contribution based on a frequency parameter to generate a filtered batch-by-batch gradient contribution; and The plurality of model parameters are updated based on the filtered batch-by-batch gradient contributions to generate a plurality of updated model parameters.

12. The system of claim 11, wherein the one or more training parameters include at least one of a learning rate, a number of training steps, a batch size, a noise multiplier, or a clipping norm.

13. The system of claim 11, wherein the operations further comprise clipping the plurality of per-example gradient contributions.

14. The system of claim 11, wherein the operations further comprise iteratively performing the calculating, the aggregating, the adding, the denoising, and the updating for a plurality of training steps.

15. The system of claim 14, wherein the operations further comprise outputting a trained machine learning model having trained model parameters after performing the plurality of training steps.

16. The system of claim 11, wherein each of the plurality of per-example gradient contributions comprises a per-example gradient and a gradient contribution map.

17. The system of claim 11, wherein the noisy batch-by-batch gradient contributions comprise a private contribution map.

18. The system of claim 11, wherein filtering the noisy batch-by-batch gradient contributions further comprises removing per-example gradients below a threshold associated with the frequency parameter.

19. The system of claim 11, wherein the operations further comprise aggregating and adding noise to the filtered batch-by-batch gradient contributions based on the first privacy parameter or the second privacy parameter.

20. A non-transitory computer-readable medium for storing instructions that, when executed by one or more processors, cause the one or more processors to perform operations for training a sparse machine learning model, the operations comprising: Receive a training dataset and multiple model parameters; Calculating a plurality of per-example gradient contributions from the training dataset and the plurality of model parameters based on one or more training parameters; aggregating and adding noise to the plurality of per-example gradient contributions based on a privacy parameter to generate noisy batch-by-batch gradient contributions; filtering the noisy batch-by-batch gradient contribution based on a frequency parameter to generate a filtered batch-by-batch gradient contribution; as well as The plurality of model parameters are updated based on the filtered batch-by-batch gradient contributions to generate a plurality of updated model parameters.

Images