Optimal correlated noise for differentially private learning

Abstract

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for calibrating correlated noise specific to a machine learning task In one aspect, there is provided a method for, at each of a plurality of training iterations, receiving a set of one or more inputs and generating a respective output for each input, determining a gradient with respect to the network parameters of an objective function for the machine learning task, calibrating a noise correlation matrix specific to the machine learning task and an index of the training iteration by generating a correlation weight matrix parameterized as a function of a tuneable noise correlation parameter and the index of the training iteration, generating a correlated-noise gradient using the calibrated noise correlation matrix, and using the correlated-noise gradient to update the values of the plurality of network parameters.

Description

OPTIMAL CORRELATED NOISE FOR DIFFERENTIALLY PRIVATE LEARNINGCROSS-REFERENCE TO RELATED APPLICATION[1] This application claims priority to U.S. Provisional Application No. 63 / 541,294, filed on September 29, 2023. The disclosure of the prior application is considered part of and is incorporated by reference in the disclosure of this application.BACKGROUND[2] This specification relates to processing data using machine learning models.[3] Machine learning models receive an input and generate an output, e.g., a predicted output, based on the received input. Some machine learning models are parametric models and generate the output based on the received input and on values of the parameters of the model.[4] Some machine learning models are deep models that employ multiple layers of models to generate an output for a received input. For example, a deep neural network is a deep machine learning model that includes an output layer and one or more hidden layers that each apply a non-linear transformation to a received input to generate an output.SUMMARY[5] This specification describes a system implemented as computer programs on one or more computers in one or more locations that can provide differentially private learning using an efficient method for calibrating correlated noise specific to a machine learning task.[6] Differential privacy involves the use of technical methods to prevent unauthorized access to information. In this specification, differentially private learning refers to a set of learning techniques that add noise, e.g. using a noise matrix, to the parameter value updates of a neural network model during each training iteration in order to provide differential privacy.[7] In this specification, differential privacy refers to providing protection for a specific entity’s data in a dataset while preserving the overall usefulness of the aggregated data. More specifically, a neural network model can be considered differentially private if the probability of an observer of the model output inferring that a particular entity’s information contributed to the output, e.g., as part of training the neural network model, is less than a, where a is the privacy level. In this framework, the smaller 8 is, the stronger the privacy guarantee. As an example, an entity can be an individual or group of related individuals. The differentially private techniques described in this specification can be used, e.g., to prevent unauthorizedaccess to the training data used to train a neural network model, especially data including information, from an individual or group of individuals.[8] In particular, this specification describes techniques for calibrating a noise correlation matrix that is specific to a machine learning task at each training iteration, e.g., as opposed to applying independent Gaussian noise at each training iteration. For example, the noise correlation matrix can be parameterized as a function of a tuneable noise correlation parameter that can be tuned as a hyperparameter during training. In some examples, the tuneable noise correlation parameter can be tuned with respect to the objective function used for training, e.g. based on the total squared error of a batch of training data for a training iteration. As an example, the tuneable noise correlation can be tuned on a tuning schedule, e.g., every iteration or every 10 training iterations.[9] The noise correlation matrix can be convolved with a sampled Gaussian noise matrix to generate a calibrated noise correlation matrix for the specific machine learning task. The calibrated noise correlation matrix can then be applied to the gradients of the parameter update for the neural network at each training iteration in order to perform a differentially private parameter update. For example, applying the calibrated noise correlation matrix can involve adding the noise matrix to the gradients calculated during the training iteration.

[0010] According to a first aspect there is provided a method for performing differentially private training of a neural network having a plurality of network parameters to perform a machine learning task, the method comprising, at each of a plurality of training iterations, receiving a set of one or more inputs, processing each of the inputs in accordance with current values of the network parameters to generate a respective output for each input, determining, using the respective outputs for the inputs, a gradient with respect to the network parameters of an objective function for the machine learning task, calibrating a noise correlation matrix specific to the machine learning task by generating a correlation weight matrix, wherein the correlation weight matrix is parameterized as a function of a tuneable noise correlation parameter, sampling a Gaussian noise matrix from a Gaussian distribution, and generating the calibrated noise correlation matrix by convolving the noise correlation matrix with the Gaussian noise matrix applying the calibrated noise correlation matrix to the gradient to generate a correlated-noise gradient, and using the correlated-noise gradient to update the values of the plurality of network parameters.

[0011] Particular embodiments of the subject matter described in this specification can be implemented so as to realize one or more of the following advantages.

[0012] Existing differentially private training techniques involve the addition of noise to one or more of the gradients, e.g., differentially private stochastic gradient descent (DP-SGD), neural network model weights, e.g., output perturbation, or objective function, e.g., objective perturbation or regularization, during the training process. However, the addition of noise can degrade the neural network model’s accuracy and robustness on unseen data, especially when training with a small privacy level to achieve a greater privacy guarantee.

[0013] In some cases, the noise added during the training process can be correlated noise, e.g., to more effectively tailor the application of noise and limit the redundancy of unnecessarily accumulating noise, which degrades data quality. In this case, careful calibration of the noise is required to maintain the balance between privacy and model utility. As an example, semi- definite programming techniques can be employed to calculate the correlated noise applied at each training iteration, e.g., to calculate the noise correlation matrix for differentially private follow-the regularized leader (DP-FTRL). However, solving a matrix factorization using semi- definite programming presents issues to scalability since adding correlated noise in a way that provides both privacy and maintains the semi-definiteness of the matrices involved can be complex and computationally-intensive.

[0014] In contrast, the techniques of this specification enable efficient differentially private learning with correlated noise that is tuned to a machine learning task during the training process and achieves near state-of-the-art privacy-utility tradeoffs. The privacy-utility tradeoff describes how minimizing privacy loss comes at a fixed expense of utility, e.g., model performance. More specifically, the privacy of the data can be fully protected, e.g., where 8 = 0 and it is impossible for an observer of the model output to infer that a particular entity’s information contributed to the output, but at the expense of applying noise that renders the training data useless.

[0015] The system of this specification can calibrate and apply noise that is specific to a particular machine learning task by tuning a noise correlation parameter as part of the neural network model training process. In particular, the system can apply a calibrated noise correlation matrix that is parameterized by the tunable noise correlation parameter. By tuning the noise correlation parameter with the neural network model weights, the system is able to tailor the addition of noise at each training iteration to the particular machine learning task such that the applied noise does not accumulate in unnecessarily redundant ways that are detrimental to the training task.

[0016] In particular, the techniques described can reduce the amount of computational resources necessary to calibrate a noise correlation matrix as part of a DP-FTRL trainingiteration. More specifically, the method for calibrating the correlated noise can circumvent the cubic complexity of state-of-the-art semi-definite programs currently used to optimize differential privacy noise correlation matrices that rely on complex matrix factorization, which scale cubically with the number of iterations in each training batch, by using an analytically derived alternating noise correlation function dependent on a single noise correlation parameter. In particular, the correlation parameter can be tuned directly as part of the neural network model training process, thereby circumventing the need for an additional layer of calculations to optimize the hyperparameter. This noise correlation matrix formulation significantly reduces the computational resources needed under a similar privacy -utility tradeoff, thereby enabling the method for calibrating correlated noise specific to a machine learning task to remain useful for training protocols with many iterations.

[0017] Moreover, the techniques described herein provably improve over the well-known DP- SGD technique that adds independent Gaussian noise to the gradients at each training iteration in expected utility on the privacy-utility tradeoff. In particular, the tuneable noise correlation parameter can allow for the tailored anti-correlation of noise across training iterations, which can improve overall model utility by subtracting out accumulating noise from previous iterations, while retaining privacy guarantees that protect information. Furthermore, when the noise correlation matrix is constrained to be a structured Toeplitz matrix, the techniques described achieve new state-of-the-art differentially private performance with respect to the privacy-utility tradeoff for standard machine learning tasks such as regression and online convex optimization.

[0018] In addition, the techniques of this specification can be integrated with privacy amplification methods to increase the robustness of protection. More specifically, privacy amplification via subsampling can be employed by the system to enable better privacy -utility tradeoffs when training batches are sampled instead of deterministically chosen. In particular, privacy amplification provides additional randomness from sampling with the randomness of the noise added to the parameter updates to provide more privacy than in the deterministic case.

[0019] Furthermore, the methods of this specification can satisfy several rigorous definitions of privacy, thereby boosting user confidence by providing for certainty and accountability in mitigating security risks when training on entity data. In particular, satisfying theoretical definitions of privacy can formally ensure that entity data is maintained in compliance with requirements. As an example, the system can satisfy epsilon-t differential privacy, e.g., approximate differential privacy, which is more practically acknowledged in real-world differential privacy machine learning settings. As another example, rho-zero concentrateddifferential privacy, which provides a more precise measure of privacy loss, can be satisfied when the values of the noise correlation matrix depend on a noise multiplier parameterized by a measure of the maximum column norm of the noise correlation matrix.

[0020] The details of one or more embodiments of the subject matter of this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, the drawings, and the claims.BRIEF DESCRIPTION OF THE DRAWINGS

[0021] FIG. 1 is a system diagram of an example correlated-noise differentially private learning system and a corresponding example correlated-noise differentially private inference system.

[0022] FIG. 2 depicts an example noise correlation engine, e.g., that can be used in the correlated-noise differentially private learning system of FIG. 1A.

[0023] FIG. 3 demonstrates example results of a neural network model trained with the example correlated-noise differentially private learning system of FIG. 1 compared with other differential privacy systems.

[0024] FIG. 4 is a flow diagram of an example process for calibrating and applying a correlated-noise gradient as part of a neural network update.

[0025] Like reference numbers and designations in the various drawings indicate like elements.DETAILED DESCRIPTION

[0026] FIG. 1 shows an example correlated-noise differentially private learning system 100 and an example correlated-noise differentially private inference system 170. Both the correlated-noise differentially private learning system 100 and the correlated-noise differentially private learning system 170 are examples of a system implemented as computer programs on one or more computers in one or more locations in which the systems, components, and techniques described below are implemented.

[0027] The system 100 performs a training process to train a neural network, also referred to herein as a neural network model 125 to perform a particular machine learning task.

[0028] In particular, the system 100 can calibrate correlated noise specific to the particular machine learning task during the training process. The system 100 can then apply the calibrated correlated noise to gradients computed as part of the training process in order to perform a differentially private parameter value update of the neural network model 125. In this context,differential privacy refers to protecting the identifiability of the training data by mitigating the risk that an entity’s information included in the training dataset can be revealed using an output of the model.

[0029] More specifically, differential privacy training techniques are useful when preserving the overall usefulness of aggregated training data is desirable, but where training examples can contain information collected from an entity. In this case, the ability to identify a particular training example using the output would expose the entity’s information. In particular, differential privacy training techniques can protect from exposing information from the training examples used to train the neural network model 125 during inference.

[0030] As an example, differential privacy can be achieved by applying DP-SGD approaches, e.g., differentially private variants of stochastic gradient descent updates that involves clipping per-example gradients to a norm, e.g., an L2 norm, and the addition of independent Gaussian noise. As another example, differential privacy can be achieved by applying DP-FTRL approaches, e.g., differentially private variants of follow-the-regularized-leader updates that involve regularization, the maintenance of a cumulative gradient, and the application of correlated Gaussian noise.

[0031] The system 100 of this specification achieves provable improvement over DP-SGD in expected accuracy by incorporating calibrated correlated noise to each model update gradient and demonstrates enhanced computational efficiency while maintaining a favorable privacyutility tradeoff with respect to other DP-FTRL approaches, e.g., as depicted in FIG. 3.

[0032] In particular, the correlated-noise differentially private learning system 100 can train the neural network model 125 on a set of inputs 105 including data from one or more entities, e.g., user groups or user devices, to generate a corresponding set of output(s) 130 that satisfy a probabilistic guarantee to not reveal any specific entities’ data with a probability 1-8, where 8 is the privacy level. The correlated-noise differentially private inference system 170 can then use the trained neural network model 180 in inference to generate a differentially private output 190.

[0033] More specifically, the system 100 can train the neural network model 125 to perform any kind of machine learning task, i.e., the system 100 can be configured to receive any kind of set of inputs 105 and to generate any kind of corresponding set of output(s) 130, e.g., a score, classification, or regression output based on the set of inputs 105 (some example machine learning tasks are described below; the training data can comprise any data suitable for the trained task). In particular, the system 100 can train the neural network model 125 using acorrelated-noise gradient that is computed using a noise correlation matrix 150 that is tuned, e.g., at each training iteration, using a noise correlation parameter 155.

[0034] For example, the system 100 can update the noise correlation parameter 155 based on a measure of error, e.g., the total error, mean squared error, etc., for each iteration. In particular, the noise correlation matrix 150 can be parameterized by the noise correlation parameter 155 and the particular iteration, and the system 100 can compute the value of the noise correlation parameter 155 by solving a single-parameter optimization problem, e.g., as a function of the noise correlation parameter 155 alone, and can minimize the objective over the noise correlation parameter 155 at each of the iterations.

[0035] By tuning the noise correlation parameter 155, the system 100 can ensure that the correlated-noise gradient 160 is tailored to the specific task, e.g., the system 100 can learn a correlation for the particular task that decreases the loss while protecting privacy along with the network parameter values.

[0036] In some cases, the neural network model 125 is a neural network that is configured to perform an image processing task, i.e., receive an input image and to process the input image, i.e., process the intensity values of the pixels of the input image, to generate a network output for the input image. For example, the task may be image classification and the output generated by the neural network model 125 for a given image may be scores or labels for each of a set of object categories, with each score representing an estimated likelihood that the image contains an image of an object belonging to the category. As another example, the task can be image embedding generation and the output generated by the neural network model 125 can be a numeric embedding of the input image. As yet another example, the task can be object detection and the output generated by the neural network model 125 can identify locations in the input image at which particular types of objects are depicted. As yet another example, the task can be image segmentation and the output generated by the neural network model 125 can assign each pixel of the input image to a category from a set of categories.

[0037] As another example, if the inputs to the neural network model 125 are Internet resources (e.g., web pages), documents, or portions of documents or features extracted from Internet resources, documents, or portions of documents, the task can be to classify the resource or document, i.e., the output generated by the neural network model 125 for a given Internet resource, document, or portion of a document may be a score for each of a set of topics, with each score representing an estimated likelihood that the Internet resource, document, or document portion is about the topic.

[0038] As another example, if each input in the set of input(s) 105 to the neural network model 125 are features of an impression context for a particular advertisement, the corresponding set of output(s) 130 generated by the neural network model 125 may be a score that represents an estimated likelihood that the particular advertisement will be clicked on.

[0039] As another example, if each input the set of input(s) 105 to the neural network model 125 are features of a personalized recommendation for a user, e.g., features characterizing the context for the recommendation, e.g., features characterizing previous actions taken by the user, the corresponding set of output(s) 130 generated by the neural network model 125 may be a score for each of a set of content items, with each score representing an estimated likelihood that the user will respond favorably to being recommended the content item.

[0040] As another example, if each of the set of input(s) 105 to the neural network model 125 is a sequence of text in one language, the corresponding set of output(s) 130 generated by the neural network model 125 may be a score for each of a set of pieces of text in another language, with each score representing an estimated likelihood that the piece of text in the other language is a proper translation of the input text into the other language.

[0041] As another example, the task may be an audio processing task. For example, if each of the set of input(s) 105 to the neural network model 125 is a sequence representing a spoken utterance, the corresponding set of output(s) 130 generated by the neural network model 125 may be a score for each of a set of pieces of text, each score representing an estimated likelihood that the piece of text is the correct transcript for the utterance. As another example, the task may be a keyword spotting task where, if each of the set of input(s) 105 to the neural network model 125 is a sequence representing a spoken utterance, the corresponding set of output(s) 130 generated by the neural network model 125 can indicate whether a particular word or phrase (“hotword”) was spoken in the utterance. As another example, if each of the set of input(s) 105 to the neural network model 125 is a sequence representing a spoken utterance, the corresponding set of output(s) 130 generated by the neural network model 125 can identify the natural language in which the utterance was spoken.

[0042] As another example, the task can be a natural language processing or understanding task, e.g., an entailment task, a paraphrase task, a textual similarity task, a sentiment task, a sentence completion task, a grammaticality task, and so on, that operates on a sequence of text in some natural language.

[0043] As another example, the task can be a text to speech task, where each of the set of input(s) 105 is text in a natural language or features of text in a natural language and thecorresponding set of output(s) 130 is a spectrogram or other data defining audio of the text being spoken in the natural language.

[0044] As another example, the task can be a health prediction task, where each of the set of input(s) 105 is electronic health record data for a patient and the corresponding set of output(s) 130 is a prediction that is relevant to the future health of the patient, e.g., a predicted treatment that should be prescribed to the patient, the likelihood that an adverse health event will occur to the patient, or a predicted diagnosis for the patient.

[0045] As another example, the task can be an agent control task, where each of the set of input(s) 105 is an observation characterizing the state of an environment and the corresponding set of output(s) defines an action to be performed by the agent in response to the observation. The agent can be, e.g., a mechanical agent such as a real-world robot or autonomous or semi- autonomous vehicle that performs actions in a real-world environment to perform a task, such as moving or manipulating an object; or a simulation of such a mechanical agent, e.g. robot (where the neural network model can be trained to perform a task in the simulation of the real world environment and afterwards used to select actions to be performed in the real world environment to accomplish the task); or a control system for an industrial facility; or a control system that controls a different kind of agent.

[0046] The neural network model 125 can have any appropriate machine learning architecture, e.g., a neural network, that can be configured to process the set of input(s) 105 in order to perform a particular machine learning task. For example, the neural network model 125 can have any appropriate number of neural network layers (e.g., 1 layer, 5 layers, or 10 layers) of any appropriate type (e.g., fully-connected layers, attention layers, convolutional layers, etc.) connected in any appropriate configuration (e.g., as a linear sequence of layers, or as a directed graph of layers).

[0047] In some examples, the neural network model 125 is a generative model, e.g., a generative-adversarial network or an autoregressive language processing network. As an example, the neural network model 125 can have a recurrent neural network architecture that is configured to sequentially process the contents of each input in the set of input(s) 105 and trained to perform next element prediction, e.g., to define a likelihood score distribution over a set of next elements. More specifically, the neural network model 125 can be a recurrent neural network (RNN), long short-term memory (LSTM), or gated-recurrent unit (GRU). As another example, the neural network model 125 can be transformer-based, e.g., an encoderdecoder transformer, an encoder-only transformer, or a decoder-only transformer, configuredto perform parallel processing of the contents of the multimodal input using a multi-headed attention mechanism.

[0048] In other examples, the neural network model 125 can be a graph neural network (GNN). In this case, the GNN can include an encoder block, a sequence of one or more message passing neural network layers, and a decoder block. The encoder block can, e.g., process the set of input(s) 105 to generate a set of node embeddings and a set of edge embeddings. Each message passing layer is configured to process the set of node embeddings and the set of edge embeddings, by neural network operations that are parametrized by a set of network parameters of the message passing layer and that are conditioned on the topology of the graph, to update the node embeddings and the edge embeddings. The decoder block can process the node and edge embeddings generated by the final message passing layer to generate the GNN output, in this case, the set of output(s) 130.

[0049] More specifically, the system 100 can use a parameter update subsystem 120 to perform a network parameter update of the neural network model 125 at each of a number of training iterations. The system 100 can train the neural network model 125 on the set of input(s) 105 by a differentially private machine learning training technique to optimize an objective function. The objective function can measure, for each training example, a discrepancy between: (i) the ground truth output(s) 110 corresponding with the set of input(s) 105, e.g., ground truth output(s) provided to the system 100 by a user, and (ii) the predicted set of output(s) 130 generated by the neural network model 125 by processing the set of input(s) 105. As an example, the objective function can measure a discrepancy between the set of output(s) 130 and the ground truth output(s) 110 in any appropriate way, e.g., using a cross-entropy loss, a mean squared error loss, a total squared error loss, etc.

[0050] In particular, the parameter update subsystem 120 can compute a gradient 135 of the network parameter values of the neural network model 125 , e.g., by computing a derivative of the objective function with respect to the network parameter values of the current iteration. The system 100 can then calibrate and apply, e.g., add, correlated noise to the gradient 135 using a noise calibration engine 140 to generate a calibrated-noise gradient 160 for the training iteration. In particular, the noise calibration engine 140 can calibrate the noise correlation matrix 150 using a method that achieves enhanced computational efficiency while maintaining a favorable privacy-utility tradeoff. An example of generating a correlated-noise gradient by leveraging a structured correlation matrix and an alternating function will be described in more detail in FIG. 2.

[0051] The system 100 can then use the calibrated-noise gradients 160 to update the network parameter values of the neural network model 125 at each of a number of training iterations, e.g., using the update rule of any appropriate gradient descent optimization algorithm, e.g., RMSprop or Adam.

[0052] In some cases, the set of input(s) 105 arrives in a data stream, e.g., a continuous or semi- continuous flow. In this case, the system 100 can train the neural network model 125 for differentially private streaming. As an example, the neural network model 125 can be trained using online learning with inputs that arrive in a data stream with a fixed order. In this case, the system 100 can train on a batch size of one with the correlated-noise gradient 160 to perform the update for the single training example, while maintaining differential privacy for the training example.

[0053] In particular, the system 100 can be implemented to train for a continual release setting, e.g., where the system 100 queries the streaming data, e.g., as a weighted sum of datapoints in the query, for information release, e.g., updating the neural network model 125, e.g., which involves aggregating knowledge from the training data, outputting an aggregation of the data, etc. For example, the system can apply the correlated-noise gradient 160 to maintain privacy when training with the query.

[0054] More specifically, rather than applying random noise, the system 100 can calibrate and apply a correlated noise matrix 150 that is specified by a noise correlation parameter 155. In particular, the system 100 can calibrate the correlated noise matrix 150 for the particular machine learning task by tuning the noise correlation parameter 145 as part of the training process. More specifically, the noise correlation parameter 145 can be tuned with the neural network model 145 directly, thereby circumventing the need for an additional layer of calculations to optimize any hyperparameters.

[0055] For example, the system 100 can calibrate the correlated noise gradient 135 using the tuning scheduler 150. As an example, the system 100 can tune the noise correlation parameter at each training iteration, every N training iterations, or based on the gradient 135 satisfying a criterion.

[0056] The noise calibration engine 140 can apply the noise correlation matrix 150 to the gradient 135 to generate the calibrated-noise gradient 160. As an example, the engine 140 can add the gradient 135 to the noise correlation matrix 150 to generate the calibrated noise gradient 160. As another example, the engine 140 can multiply the noise correlation matrix 150 with the gradient 135 to generate the calibrated-noise gradient 160.

[0057] In some cases, the noise calibration engine 135 can perform a postprocessing step before applying the noise correlation matrix 155 to the gradient 135. In particular, the system 100 can clip the gradient 135 to enhance the stability of the training process, or can apply a decay factor to the correlation weight matrix in the case that the gradient 135 satisfies a divergence criterion, e.g., when the gradient 135 is larger than a threshold. For example, the decay factor can be parameterized by the noise correlation parameter 155, the learning rate, or both.

[0058] After training, the correlated-noise differentially private inference system 170 can use the trained neural network 180 to generate differentially private outputs 190, thereby satisfying the criteria of differential privacy. The system 100 can train the neural network model 180 included in an application such that the application usability is minimally impacted while mitigating the risk of exposing the training data used for training the model 180.

[0059] FIG. 2 depicts an example noise calibration engine. As an example, the noise calibration engine 140 of the correlated-noise differentially private learning system 100 can be implemented as the noise calibration engine 200.

[0060] As depicted in FIG. 1, the noise calibration engine 200 can process the gradient 135 of the current network parameter values of the neural network model at the particular training iteration to generate the correlated-noise gradient 160 for differentially private training. In this case, the noise calibration engine 200 includes a noise correlation matrix 210 that is computed through a convolution (or product) of two matrices: a correlation weight matrix 220 and a Gaussian noise matrix 230. In the particular example depicted, the correlation weight matrix 220 is parameterized by the noise correlation parameter 225.

[0061] In particular, the system can use the noise calibration engine 200 to perform parameter updates from t = 0 to T according to:where 0i . . .0T and 0o . . .0T-I represent the sequences of neural network model parameters at time t+1 and t, respectively, J] is the learning rate, go...gT-i is the sequence gradient 135 at time t, W0...WT-1 is the sequence of Gaussian noise matrices 230 at time t, and B is the sequence of correlation weight matrices 220 which is parameterized by the noise correlation parameter 225. In this case, C is an encoder matrix, e.g., the inverse of B, such that the noise correlation matrix 210 resulting from the convolution of the correlation weight matrix 220 and the Gaussian noisematrix 230 is added to the gradient 135 to generate the correlated-noise gradient 160, e.g., as opposed to convolving the correlation weight matrix 220 directly with the gradient 135.

[0062] In this context, a convolution is a matrix operation that involves applying the correlation weight matrix 220 to the Gaussian noise matrix 230. For example, the convolution can involve using the correlation weight matrix 220 as a kernel and performing element-wise multiplication between the kernel and the overlapping submatrix of the Gaussian noise matrix 230.

[0063] In particular, the correlation weight matrix 220 can be parameterized by a tuneable noise correlation parameter 225, i.e., the value of each element in the correlation weight matrix can be determined by the tuneable noise correlation parameter. There are many ways in which values of elements in the correlation weight matrix 220 can be made dependent on such a tuneable noise correlation parameter 225; one way that works particularly well is described later.

[0064] In some implementations one datapoint, i.e., input, is processed each training iteration, and there are T datapoints indexed by t. In this case, the correlation weight matrix 220 can be a T by T matrix.

[0065] In some implementations the correlated-noise gradients are clipped using a clip norm value, i.e. a clip value that depends on a norm, e.g. an L2 norm, of the correlated-noise gradients, e.g., by multiplying the gradients by a threshold value. As an example the system can sample the Gaussian noise matrix 230, e.g., by drawing the values of the matrix from a Gaussian distribution parameterized by a standard deviation value. In some cases, the system can determine the standard deviation value of the Gaussian distribution for sampling, e.g., using a noise multiplier o and a clip norm value G, e.g., the L2 norm of the gradient 135 from the standard normal distribution N(0, G2o2I). As an example, the noise multiplier o can depend on B, the maximum column norm of B, and the privacy level 8, e.g., o2= B2 / 2e.

[0066] In the particular example depicted, the correlation weight matrix 220 is chosen as a structured matrix, e.g., a matrix whose arrangement of values relate to a particular structure. For example, the correlation weight matrix 220 can be a lower triangular matrix 240, e.g., where the entries of the matrix are below the diagonal, or a Toeplitz matrix 250, e.g., where each descending diagonal is a constant. In particular, the structured matrices can be used to enhance the efficiency of the noise correlation matrix calculation. More specifically, the structure can be exploited for representing noise correlation as an autoregressive time difference between training iterations, e.g., not their absolute position in the sequence.

[0067] In the case that the correlation weight matrix 220 is a Toeplitz matrix 250, the system can determine the values of the Toeplitz matrix 250 as a prefix sequence of correlated noise weights P = (Po, . . . , PT-I) from previous training iterations. In particular, the first row of the matrix can grow as fit is added at each iteration, and the other entries can correspond with the shared diagonals specified by the structure of the Toeplitz matrix 250. This simplifies the above equation to:where 0t+i and 0t represent the neural network model parameters at time t+1 and t, respectively, j] is the learning rate, gt is the gradient 135, wt is the Gaussian noise matrix 230, and t defines the elements of the correlation weight matrix 220 based on the Toeplitz structure, and T is an index for the terms in the sum based on the number of preceding training iterations, i.e., an index that counts training iterations.

[0068] More specifically, the effective noise that is added to the optimization of the parameters is the noise (wo, . . . ,wt-i) is linearly correlated by the first row of the correlation weight matrix 220, e.g., which increases by one value at each training iteration since Bt,t= Pt-t for all t > t'.Thus, any choice of the correlation weight matrix 220 structured as a Toeplitz matrix is particularly advantageous to use.

[0069] In some implementations the noise correlation matrix 210, which can be denoted wt=can be a matrix with only one row or column, in particular a vector with a length equal to the number of network parameters, and the correlated-noise gradient 160 can be determined as gt= gt+ wt, so that 0t+1= 0t— T]gt, where the correlation weight matrix 220 is not the identity matrix (which recovers DP-SGD). While any choice of the correlation weight matrix 220 that is not the identity matrix functions to inject noise into the learning process that is correlated across iterations, a particularly computationally -efficient example choice of values P = (Po, . . . , PT-I) for the correlation weight matrix 220 is given below.

[0070] In particular, the correlation weight matrix 220, e.g., B, can be parameterized by the noise correlation parameter 225. For example, the values P of the correlation weight matrix 220 can be determined as an alternating function with respect to the training iteration and a polynomial function of the tunable noise parameter 225, e.g., a function that depends on (-1 / . For some machine learning tasks, this alternating factor can allow for the anti-correlation ofnoise across iterations, which can improve model performance and robustness in inference by subtracting out noise from previous iterations, while retaining privacy guarantees.

[0071] For example, the values of the correlation weight matrix 220 can be parameterized by the noise correlation parameter 225, by making the elements dependent on a damping term (1- v) where 0 < v < 1. This dampens the correlations in the noise and can result in enhanced utility-privacy tradeoff performance, e.g., as shown in FIG. 3. As an example, one choice is to define values of the correlation weight matrix 220 as:-y := (-l / ffKl - «)‘ where t is the index of training iteration and v is the tuneable noise correlation parameter 225. In these examples the noise correlation parameter 225 functions as a decay parameter that enhances the stability of the training process by decreasing the noise correlation. Also, optionally, the noise correlation parameter 225 can be tuned at each of the training iterations to tailor the application of correlation noise to the specific machine learning task, e.g., the system can leam a correlation for the particular task that increases task accuracy while protecting privacy.

[0072] Defining the values of the correlation weight matrix 220 at each iteration using the above equation can decrease the computational resources necessary to calculate the noise correlation matrix 220 from O(N3) to O(N), e.g., by removing the need to rely on computationally intensive semi-definite programming to perform complex matrix factorization, e.g., multi-epoch (ME) matrix factorization, banded matrix factorization, full Honaker, etc., which will be described in more detail with respect to FIG. 3, in order to calculate a differentially private noise correlation matrix. As an example, other approaches to calculating the correlation weight matrix 220 involve calculating the correlation weight matrix 220 by minimizing the squared error in the gradient prefix sums, e.g., as is described in Choquette- Choo, et. al. “Multi-Epoch Matrix Factorization Mechanisms for Private Machine Learning” (arXiv:2211.06530v2). In some cases, these approaches can involve storing the prefix sums in a tree structure to decrease the computational resource scaling necessary to compute the calibrated-noise gradient using an updated correlation weight matrix, but are still hamstrung by solving a semidefinite program with growing matrix variables requiring cubic time.

[0073] Additionally, defining the noise correlation matrix 220 using this equation allows the matrix to be further characterized by the effective dimension and condition number of the neural network model, which are useful values for analyzing the performance of the neuralnetwork model. In particular, a measure of performance for a particular noise correlation matrix 220 for a number of machine learning tasks can be characterized by defining a lower bound that scales with the effective dimension characterizing the generalization error of the neural network and defining an upper bound that scales with the condition number characterizing sensitivity to inputs for the neural network.

[0074] As described with respect to FIG. 1, the noise correlation parameter 225 can be tuned using a tuning scheduler 145 over a number of training iterations, e.g., every N iterations, based on a measure of error for the gradient, etc. For example, the noise correlation parameter 225 can be used based on a mean square error or a total error of the gradient 135. More specifically, tuning the noise correlation parameter 225 using a tuning scheduler 145 ensures that the correlated-noise gradient 160 is tailored to the particular machine learning task, thereby augmenting the neural network model’s utility, e.g., performance in inference.

[0075] FIG. 3 demonstrates example results for a neural network model trained with a variety of differential privacy systems using different privacy level values s.

[0076] In this context, the privacy level s controls the magnitude of noise added to the training process as a means of managing the trade-off between protecting the privacy of entity data in the training data set while maintaining model performance. More specifically, a lower privacy level value provides stronger privacy protection, e.g., through the addition of more noise, whereas a higher privacy level value adds less noise, thereby improving the accuracy of the results, but potentially exposing more information.

[0077] In the particular example depicted, the performance of the example correlated-noise differentially private learning system (“correlated-noise system”) 100 of FIG. 1 is compared to other differential privacy training systems and to an absolute performance upper bound provided by a nonprivate baseline 305, e.g., at s = co, in the graph 300. The nonprivate baseline 305 represents training without the addition of any noise. In particular, the graph 300 demonstrates the test accuracy of each system at a variety of privacy levels.

[0078] More specifically, the performance of the correlated-noise system 100 is compared to neural network models trained with other DP -FTRL training systems, e.g., as depicted in Table 350, on the example-level differential privacy for image classification CIFAR-10 dataset. Table 350 depicts differential privacy training system variants including: Honaker matrix factorization 320, which involves adding different magnitudes of noise to each level of a tree, e.g., as described in Honaker, “Efficient Use of Differentially Private Binary Trees” Theory and Practice of Differential Privacy (2015), DP-SGD 330, and optimal continual counting(CC) 310, which involves matrix factorization using two lower triangular matrices, e.g., as described in Fichtenberger, H., et. al. “Constant Matters: Fine-Grained Error Bound on Differentially Private Continual Observation” (arXiv:2202.11205v6), multi-epoch (ME) matrix factorization 340, which involves Fourier transform-based online matrix factorization, e.g., as described in Choquettte-Choo, et. al. “Multi-Epoch Matrix Factorization Mechanisms for Private Machine Learning.” (arXiv:2211.06530v2). The table 350 also includes whether the system can be run online, e.g., as opposed to a system that is required to precompute the gradients, and whether the correlation matrix can be optimized agnostic to the number of training iterations.

[0079] In the case that an example differential privacy system requires the use of hyperparameters, the respective system hyperparameters were tuned with a grid search. Notably, the correlated-noise system 100 has only a single tunable parameter, e.g., the noise correlation parameter 155, which can be easily tuned by minimizing a measure of error in the gradient as part of the neural network model training process. More specifically, the system 100 can circumvent the need for an additional layer of calculations to optimize and select any hyperparameters, thereby reducing the computational resources necessary to train the neural network model.

[0080] As depicted in graph 300, the system 100 outperforms both the optimal CC 310 and Honaker 320 systems at every privacy level depicted. In particular, there is an average of three percentage point accuracy improvement that grows as s grows smaller, thereby indicating a clear accuracy advantage at lower privacy levels.

[0081] In the particular example depicted in graph 350, the DP-SGD system 330 has the additional benefit of privacy amplification by sampling, e.g., by sampling a random subset of the data for each training iteration, whereas the system 100 is considered without privacy amplification. Even so, the system 100 outperforms the DP-SGD 330 at all but the smallest privacy levels. While not depicted here, the system 100 has the flexibility to also be implemented with privacy amplification via subsampling to enable improved privacy-utility tradeoffs with respect to the results shown.

[0082] Furthermore, the system 100 makes up 30-80% of the gap between optimal CC 310, Honaker 320, and DP-SGD 330, which are considered computationally efficient differentially private learning techniques, and the more computationally-intensive and precise ME approach 340. For example, at s = 10 in the graph 350, the system 100 achieves 69.26% accuracy, nearly matching the ME system 340 at 70.83% accuracy.

[0083] FIG. 4 is a flow diagram of an example process 400 for performing differentially private training of a neural network by calibrating and applying a correlated-noise gradient as part of a neural network update. For convenience, the process 500 will be described as being performed by a system of one or more computers located in one or more locations. For example, a training system, e.g., the correlated-noise differentially private learning system 100 of FIG. 1, appropriately programmed in accordance with this specification, can perform the process 400 at each of a number of training iterations.

[0084] The system can receive a set of inputs for a machine learning task (step 410), and the system can process each input using the neural network model in accordance with current values of network parameters to generate respective outputs (step 420). As an example, the machine learning task can be an image processing task, a classification task, or a generative task. As another example, the machine learning task can be an audio processing or natural language processing task. As yet another example, the machine learning task can be an agent control task or an online convex optimization. In some cases, the inputs arrive in a data stream and the set of one or more inputs includes only one input, e.g., as part of a continual release setting that requires processing linear queries that arrive in a fixed ordering.

[0085] The system can determine a gradient with respect to the network parameters of an objective function for the machine learning task (step 430). In particular, the system can use the respective outputs for the inputs to compute the gradient of the network parameter values of the neural network model, e.g., by computing a derivative of the objective function with respect to the network parameter values of the current iteration. The objective can be any suitable objective function for the particular machine learning task to be learned.

[0086] The system can calibrate a noise correlation matrix specific to the machine learning task and the index of the training iteration (step 440). The system can generate the calibrated noise correlation matrix by convolving a correlation weight matrix with a Gaussian noise matrix. In particular, the system can sample the Gaussian noise matrix from a Gaussian distribution. As an example, the system can determine the standard deviation of the Gaussian distribution using a noise multiplier and a clip norm value, e.g., the L2 clip norm value of the gradient.

[0087] For example, correlation weight matrix can be a structured matrix, e.g., a lower triangular matrix, e.g., a Toeplitz matrix, that is parameterized as a function of a tuneable noise correlation parameter. In particular, the values of the correlation weight matrix can be determined as an alternating function with respect to the index of the training iteration and a polynomial function of the tuneable noise parameter. In the case that the correlation weightmatrix is a Toeplitz matrix, the system can determine the values of the Toeplitz matrix as a prefix sequence of correlated noise weights from previous training iterations.

[0088] In particular, the system can tune the tuneable noise correlation parameter on a tuning schedule based on the training iterations. In some cases, the system can determine the correlation parameter by tuning the tuneable noise correlation parameter with respect to a measure of error of the respective outputs for the training inputs based on the objective function computed for the machine learning task at the particular index of the training iteration. In particular, the noise correlation matrix can be parameterized by the noise correlation parameter and the particular training iteration, and the system can solve a single-parameter optimization problem over the training iterations to tune the tunable noise correlation parameter. As an example, the system can tune the tuneable noise correlation parameter with respect to the total squared error or the mean squared error of the machine learning task at the particular training iteration. In some cases, the system can tune the tuneable noise correlation parameter at each training iteration.

[0089] The system can apply the calibrated noise correlation matrix to the gradient to generate a correlated-noise gradient (step 450). For example, the system can add the calibrated noise correlation matrix to the gradient to generate the correlated-noise gradient. In some cases, the system can apply a clipping function to the gradient to generate a clipped-gradient, and can then apply the calibrated noise correlation matrix to the clipped-gradient to generate a correlated noise-clipped gradient. In another case, the system can determine whether the gradient satisfies a divergence criterion and can apply a decay factor to the gradient when the tuneable noise parameter satisfies the divergence criterion before applying the calibrated noise correlation matrix to the gradient. The divergence criterion can relate to the sensitivity of the training process to the added noise, e.g. based on a divergence between distributions of values of the network parameters during the training process.

[0090] The system can then use the correlated-noise gradient to update the values of the network parameters (step 460), e.g., at each of a number of training iterations. For example, the system can update the values of the network parameters using the update rule of any appropriate gradient descent optimization algorithm, e.g., RMSprop or Adam. In this case, the system can use the correlated-noise gradient to update the network parameters through stochastic gradient descent.

[0091] After the neural network model has been trained it can be used to perform the machine learning task by receiving a neural network input and processing the neural network input usingthe trained neural network model to generate a neural network output that performs the machine learning task.

[0092] This specification uses the term “configured” in connection with systems and computer program components. For a system of one or more computers to be configured to perform particular operations or actions means that the system has installed on it software, firmware, hardware, or a combination of them that in operation cause the system to perform the operations or actions. For one or more computer programs to be configured to perform particular operations or actions means that the one or more programs include instructions that, when executed by data processing apparatus, cause the apparatus to perform the operations or actions.

[0093] Embodiments of the subject matter and the functional operations described in this specification can be implemented in digital electronic circuitry, in tangibly-embodied computer software or firmware, in computer hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible non-transitory storage medium for execution by, or to control the operation of, data processing apparatus. The computer storage medium can be a machine- readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them. Alternatively or in addition, the program instructions can be encoded on an artificially-generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal, that is generated to encode information for transmission to suitable receiver apparatus for execution by a data processing apparatus.

[0094] The term “data processing apparatus” refers to data processing hardware and encompasses all kinds of apparatus, devices, and machines for processing data, including by way of example a programmable processor, a computer, or multiple processors or computers. The apparatus can also be, or further include, special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit). The apparatus can optionally include, in addition to hardware, code that creates an execution environment for computer programs, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them.

[0095] A computer program, which may also be referred to or described as a program, software, a software application, an app, a module, a software module, a script, or code, can bewritten in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages; and it can be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A program may, but need not, correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data, e.g., one or more scripts stored in a markup language document, in a single file dedicated to the program in question, or in multiple coordinated files, e.g., files that store one or more modules, subprograms, or portions of code. A computer program can be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and interconnected by a data communication network.

[0096] In this specification the term “engine” is used broadly to refer to a software-based system, subsystem, or process that is programmed to perform one or more specific functions. Generally, an engine will be implemented as one or more software modules or components, installed on one or more computers in one or more locations. In some cases, one or more computers will be dedicated to a particular engine; in other cases, multiple engines can be installed and running on the same computer or computers.

[0097] The processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by special purpose logic circuitry, e.g., an FPGA or an ASIC, or by a combination of special purpose logic circuitry and one or more programmed computers.

[0098] Computers suitable for the execution of a computer program can be based on general or special purpose microprocessors or both, or any other kind of central processing unit. Generally, a central processing unit will receive instructions and data from a read-only memory or a random access memory or both. The essential elements of a computer are a central processing unit for performing or executing instructions and one or more memory devices for storing instructions and data. The central processing unit and the memory can be supplemented by, or incorporated in, special purpose logic circuitry. Generally, a computer will also include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic, magneto-optical disks, or optical disks. However, a computer need not have such devices. Moreover, a computer can be embedded in another device, e.g., a mobile telephone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a Global Positioning System (GPS) receiver, or a portable storage device, e.g., a universal serial bus (USB) flash drive, to name just a few.

[0099] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto-optical disks; and CD- ROM and DVD-ROM disks.

[0100] To provide for interaction with a user, embodiments of the subject matter described in this specification can be implemented on a computer having a display device, e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user and a keyboard and a pointing device, e.g., a mouse or a trackball, by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, speech, or tactile input. In addition, a computer can interact with a user by sending documents to and receiving documents from a device that is used by the user; for example, by sending web pages to a web browser on a user’s device in response to requests received from the web browser. Also, a computer can interact with a user by sending text messages or other forms of message to a personal device, e.g., a smartphone that is running a messaging application, and receiving responsive messages from the user in return.

[0101] Data processing apparatus for implementing machine learning models can also include, for example, special-purpose hardware accelerator units for processing common and computeintensive parts of machine learning training or production, i.e., inference, workloads.

[0102] Machine learning models can be implemented and deployed using a machine learning framework, e.g., a TensorFlow framework, or a Jax framework.

[0103] Embodiments of the subject matter described in this specification can be implemented in a computing system that includes a back-end component, e.g., as a data server, or that includes a middleware component, e.g., an application server, or that includes a front-end component, e.g., a client computer having a graphical user interface, a web browser, or an app through which a user can interact with an implementation of the subject matter described in this specification, or any combination of one or more such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include a local area network (LAN) and a wide area network (WAN), e.g., the Internet.

[0104] The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. In some embodiments, a server transmits data, e.g., an HTML page, to a user device, e.g., for purposes of displaying data to and receiving user input from a user interacting with the device, which acts as a client. Data generated at the user device, e.g., a result of the user interaction, can be received at the server from the device.

[0105] While this specification contains many specific implementation details, these should not be construed as limitations on the scope of any invention or on the scope of what may be claimed, but rather as descriptions of features that may be specific to particular embodiments of particular inventions. Certain features that are described in this specification in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination. Moreover, although features may be described above as acting in certain combinations and even initially be claimed as such, one or more features from a claimed combination can in some cases be excised from the combination, and the claimed combination may be directed to a subcombination or variation of a subcombination.

[0106] Similarly, while operations are depicted in the drawings and recited in the claims in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. In certain circumstances, multitasking and parallel processing may be advantageous. Moreover, the separation of various system modules and components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0107] Particular embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve desirable results. As one example, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some cases, multitasking and parallel processing may be advantageous.

Claims

CLAIMS1. A method performed by one or more computers and for performing differentially private training of a neural network having a plurality of network parameters to perform a machine learning task, the method comprising, at each of a plurality of training iterations: receiving a set of one or more inputs; processing each of the inputs in accordance with current values of the network parameters to generate a respective output for each input; determining, using the respective outputs for the inputs, a gradient with respect to the network parameters of an objective function for the machine learning task; calibrating a noise correlation matrix specific to the machine learning task and an index of the training iteration, comprising: generating a correlation weight matrix, wherein the correlation weight matrix is parameterized as a function of a tuneable noise correlation parameter and the index of the training iteration.; sampling a Gaussian noise matrix from a Gaussian distribution; generating the calibrated noise correlation matrix by convolving the noise correlation matrix with the Gaussian noise matrix; applying the calibrated noise correlation matrix to the gradient to generate a correlated- noise gradient; and using the correlated-noise gradient to update the values of the plurality of network parameters.

2. The method of claim 1, wherein applying the calibrated noise correlation matrix to the gradient to generate the correlated-noise gradient comprises adding the calibrated noise correlation matrix to the gradient.

3. The method of any one of claims 1-2, wherein applying the calibrated noise correlation matrix to the gradient to generate a correlated-noise gradient comprises: applying a clipping function to the gradient to generate a clipped-gradient with respect to the network parameters of the objective function for the machine learning task; and applying the calibrated noise correlation matrix to the clipped-gradient to generate a correlated-noise clipped gradient.

4. The method of any one of claims 1-3, wherein the correlation weight matrix is a structured matrix.

5. The method of claim 4, wherein the structured matrix is a lower triangular matrix.

6. The method of claim 5, wherein the structured matrix is a Toeplitz matrix.

7. The method of claim 6, wherein values of the Toeplitz matrix are determined as a prefix sequence of correlated noise weights from previous training iterations.

8. The method of any one of claims 1-7, wherein the tuneable noise correlation parameter is tuned with respect to the objective function of the machine learning task for the index of the training iteration.

9. The method of claim 8, wherein the tuneable noise correlation parameter is tuned with respect to a measure of error of the respective outputs for the set of one or more inputs based on the objective function of the machine learning task for the index of the training iteration.

10. The method of claim 9, wherein the measure of error is total squared error.

11. The method of claim 9, wherein the measure of error is mean squared error.

12. The method of any one of claims 1-11, wherein values of the correlation weight matrix are determined as an alternating function with respect to the index of the training iteration and a polynomial function of the tuneable noise parameter.

13. The method of any one of claims 1-12, wherein applying the calibrated noise correlation matrix to the gradient to generate a correlated-noise gradient further comprises: determining whether the gradient satisfies a divergence criterion; and applying a decay factor to the correlation weight matrix when the gradient satisfies the divergence criterion.

14. The method of any one of claims 1-13, further comprising tuning the tuneable noise correlation parameter on a tuning schedule comprising a plurality of training iterations for tuning.

15. The method of claim 14, wherein the tuning schedule comprises tuning the tuneable noise correlation parameter each training iteration.

16. The method of any one of claims 1-15, further comprising determining a standard deviation of the Gaussian distribution used to sample the Gaussian noise matrix using a noise multiplier and a clip norm value.

17. The method of claim 16, wherein the clip norm value comprises an L2 clip norm value of the gradient at the index of the training iteration.

18. The method of any one of claims 1-17, wherein using the correlated-noise gradient to update the values of the plurality of network parameters comprises updating the network parameters through stochastic gradient descent.

19. The method of any one of the preceding claims, wherein the inputs arrive in a data stream and the set of one or more inputs includes only one input.

20. The method of any one of the preceding claims, wherein the machine learning task is an online convex optimization.

21. A computer-implemented method of performing a machine learning task, comprising: obtaining a neural network, wherein the neural network has been trained by the method of any of claims 1-20 to perform the machine learning task; receiving a neural network input; and processing the neural network input using the trained neural network to generate a neural network output that performs the machine learning task.

22. A system comprising one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform the method of any one of claims 1 to 21.

23. A computer storage medium encoded with a computer program, the program comprising instructions that are operable, when executed by data processing apparatus, to cause the data processing apparatus to perform the method of any one of claims 1 to 21.

Images