Leveraging public data to train neural networks using private image descent

Abstract

The method (300) includes obtaining a set of differential private (DP) gradients (143), each generated based on processing corresponding private data (139), and obtaining a set of public gradients (117), each generated based on processing corresponding public data (160). The method also includes applying an image descent method to the set of public gradients to learn a geometry (215) for the set of DP gradients, and reshaping the set of DP gradients based on the learned geometry. The method further includes training a machine learning model (150) based on the reshaped set of DP gradients (225).

Description

[Technical Field]

[0001] The present disclosure relates to leveraging public data in training neural networks using private image descent. [Background technology]

[0002] Differentially private (DP) training is commonly used to train private models on private data such that no sensitive information is leaked from the private data. Differentially private stochastic gradient descent (DP-SGD) has become the de facto algorithm for training private models using differential privacy. Summary of the Invention [Means for solving the problem]

[0003] One aspect of the present disclosure provides a method that includes obtaining a set of differentially private (DP) gradients, each generated based on processing corresponding private data, and obtaining a set of public gradients, each generated based on processing corresponding public data. The method also includes applying image descent to the set of public gradients to learn a geometry for the set of DP gradients, and reshaping the set of DP gradients based on the learned geometry. The method further includes training a machine learning model based on the reshaped set of DP gradients.

[0004] Implementations of the present disclosure may include one or more of the following optional features: In some examples, the private data and the public data are derived from the same source distribution. In some implementations, each DP gradient in the set of DP gradients is generated by processing corresponding private data using a machine learning model to generate a corresponding predicted private output, determining a private loss function based on the corresponding predicted private output and the corresponding private ground truth, and adding noise to the private gradient derived from the private loss function to generate the DP gradient. In some examples, the private loss function is convex and L-Lipschitz.

[0005] In some implementations, each public gradient in the set of public gradients is generated by processing corresponding public data using a machine learning model to generate a corresponding predicted public output, determining a public loss function based on the corresponding predicted public output and the corresponding public ground truth, and deriving the public gradient from the public loss function. In some examples, applying mirror descent to the set of public gradients to learn a geometry for the set of DP gradients includes applying mirror descent by using the public gradient derived from the public loss function as a mirror map to learn a geometry for the set of DP gradients. In some examples, the public loss function is strongly convex.

[0006] In some examples, the data processing hardware is located at a central server, and the set of DP gradients and the set of public gradients are stored in a central repository located at the central server. In some implementations, the data processing hardware is located at a remote system, and obtaining the set of DP gradients includes receiving the set of DP gradients from one or more client devices by federated learning without receiving any of the corresponding private data, and each DP gradient in the set of DP gradients is generated locally at a respective client device of the one or more client devices.

[0007] In some implementations, the machine learning model includes an image classification model, a language model, and / or a speech recognition model.

[0008] Another aspect of the present disclosure provides a system including data processing hardware and memory hardware in communication with the data processing hardware, the memory hardware storing instructions that, when executed on the data processing hardware, cause the data processing hardware to perform operations. The operations include obtaining a set of differential private (DP) gradients, each generated based on processing corresponding private data, and obtaining a set of public gradients, each generated based on processing corresponding public data. The method also includes applying an image descent method to the set of public gradients to learn a geometry for the set of DP gradients, and reshaping the set of DP gradients based on the learned geometry. The method further includes training a machine learning model based on the reshaped set of DP gradients.

[0009] Implementations of the present disclosure may include one or more of the following optional features: In some examples, the private data and the public data are derived from the same source distribution. In some implementations, each DP gradient in the set of DP gradients is generated by processing corresponding private data using a machine learning model to generate a corresponding predicted private output, determining a private loss function based on the corresponding predicted private output and the corresponding private ground truth, and adding noise to the private gradient derived from the private loss function to generate the DP gradient. In some examples, the private loss function is convex and L-Lipschitz.

[0010] In some implementations, each public gradient in the set of public gradients is generated by processing corresponding public data using a machine learning model to generate a corresponding predicted public output, determining a public loss function based on the corresponding predicted public output and the corresponding public ground truth, and deriving the public gradients from the public loss function. In some examples, applying mirror descent to the set of public gradients to learn a geometry for the set of DP gradients includes applying mirror descent by using the public gradients derived from the public loss function as a mirror map to learn a geometry for the set of DP gradients. In some examples, the public loss function is strongly convex.

[0011] In some examples, the data processing hardware is located at a central server, and the set of DP gradients and the set of public gradients are stored in a central repository located at the central server. In some implementations, the data processing hardware is located at a remote system, and obtaining the set of DP gradients includes receiving the set of DP gradients from one or more client devices by federated learning without receiving any of the corresponding private data, and each DP gradient in the set of DP gradients is generated locally at a respective client device of the one or more client devices.

[0012] In some implementations, the machine learning model includes an image classification model, a language model, and / or a speech recognition model.

[0013] The details of one or more implementations of the disclosure are set forth in the accompanying drawings and the description below. Other aspects, features, and advantages will be apparent from the description and drawings, and from the claims. [Brief explanation of the drawings]

[0014] [Figure 1] FIG. 1 illustrates an exemplary system of a machine learning (ML) environment that leverages public data to train neural networks using private image descent. [Figure 2] FIG. 1 is a schematic diagram of an exemplary training process that leverages public data for training a neural network using private image descent. [Figure 3] 1 is a flowchart of an exemplary arrangement of operations for a computer-implemented method for leveraging public data for training neural networks using private image descent. [Figure 4] FIG. 1 is a schematic diagram of an example computing device that can be used to implement the systems and methods described herein. DETAILED DESCRIPTION OF THE INVENTION

[0015] Like reference symbols in the various drawings indicate like elements.

[0016] Federated learning of machine learning (ML) models is an increasingly popular technique for training ML models. In traditional federated learning, a local ML model is stored locally on a user's client device, and a global ML model, which is a cloud-based counterpart of the local ML model, is stored remotely on a remote system (e.g., a cluster of servers). The client device can use the local ML model to process user input detected on the client device to generate predicted outputs and compare the predicted outputs with ground truth to generate client gradients. Furthermore, the client device can send the client gradients to the remote system. The remote system can utilize the client gradients, and possibly additional client gradients similarly generated on additional client devices, to update the weights of the global ML model. The remote system can send the global ML model or updated weights of the global ML model to the client devices. The client devices can then replace their local ML models with the global ML model or replace the weights of their local ML models with the updated weights of the global ML model, thereby updating the local ML models.

[0017] In particular, these global ML models are typically pre-trained on a remote system prior to use in federated learning, based on multiple remote gradients generated remotely on the remote system, without using client gradients generated locally on the client device. This pre-training is typically based on proxy or biased data that may not reflect the data encountered when the global ML model is deployed on the client device. Following pre-training, the weights of these global ML models are typically updated solely based on client gradients generated on the client device based on data encountered when the global ML model is deployed on the client device (e.g., private data), without using gradients generated on the remote system. However, updating the weights of these global ML models in this manner can result in catastrophic forgetting of information learned during pre-training. Furthermore, client gradients generated based on certain data (e.g., false positives, false negatives, etc.) can be difficult to obtain on the client device, thereby resulting in degraded performance of ML models trained using federated learning.

[0018] Differentially private stochastic gradient descent (DP-SGD) and its variants have become the de facto standard algorithm for training ML models using differential privacy. While DP-SGD is known to perform well in obtaining both optimal residual empirical risk and residual population risk for a convex loss, the error guarantees obtained can suffer from a clear polynomial dependence on the model dimension. This polynomial dependence can significantly affect the privacy / utility tradeoff when the model dimension is larger than the number of private training data records in the dataset. Because of this, and empirically, even when DP-SGD is used to train large-scale deep learning ML models, there can be a significant degradation in accuracy compared to non-private counterparts. Implementations herein are directed to effectively using public data (e.g., drawn from the same distribution as the original private / confidential training dataset) to improve the privacy / utility tradeoff for DP model training. Specifically, the technique provides a DP variant of mirror descent that uses a loss function generated from public data as a mirror map and DP gradients on private / confidential data as a linear term to ensure population risk guarantees for convex losses without explicit dependence on dimensionality, as long as the number of records in the public dataset exceeds the model dimension. As will become apparent, the DP variant of mirror descent, when aided by public data, can effectively reduce the variance of noise added to private gradients in DP model training. The DP model may correspond to any type of neural network model trained using DP-SGD or a variant thereof. For example, the DP neural network model may correspond to an image classification model, a language model, a speech recognition model, a speech-to-speech model, or a text-to-speech model.

[0019] 1 is a schematic diagram of an exemplary system 100 operating in an ML environment 101. In the illustrated example, the system 100 includes a remote system 110 (e.g., a central server) and one or more client devices 130 for conducting federated learning (e.g., training) of a global ML model 150 by leveraging public data 160 using private image descent. During use or inference, the ML module 132 of each client device 130 is configured to process input 133 using an on-device ML engine 134 that executes an on-device ML model 135 to generate output 136. In the illustrated example, a distribution engine 111 of the remote system 110 provides the global ML model 150 to the ML module 132, or more generally, the client devices 130, for use as the on-device ML model 135 via one or more communication networks 170 (e.g., any combination of a local area network (LAN), a wide area network (WAN), and / or other types of networks). The illustrated example includes multiple client devices 130 determining private gradients for corresponding private data, which may be used by remote system 110 to update global ML model 150 without disclosing the private data. However, in other examples, remote system 110 determines private gradients for corresponding private data such that remote system 110 performs substantially all aspects of ML training.

[0020] A client device 130 may correspond to any computing device associated with a user and capable of receiving input, processing it, and providing output. Some examples of user devices 130 include, but are not limited to, mobile devices (e.g., mobile phones, tablets, laptops, etc.), computers, wearable devices (e.g., smart watches), smart appliances, Internet of Things (IoT) devices, vehicle infotainment systems, smart displays, smart speakers, etc. Each client device 130 includes data processing hardware 137 and memory hardware 138 in communication with the data processing hardware 137. The memory hardware 138 stores instructions that, when executed by the data processing hardware 137, cause the data processing hardware 137 or, more generally, the client device 130 to perform one or more operations. Each client device 130 may include or be coupled to one or more input systems (not shown for clarity of illustration) to capture, record, receive, or possibly obtain input 133, among possibly other inputs, for the client device 130. Each client device 130 may include or be coupled to one or more output systems (not shown for clarity of illustration) to output, or possibly provide, output 136 among possibly other inputs of the client device 130. An input system may be used to obtain input from a user, another device, another system, etc. An output system may be used to provide output to a user, a device, another system, etc.

[0021] In one example, input 133 includes text, and on-device ML model 135 converts the text into synthetic speech as output 136. For example, on-device ML model 135 may convert input text into corresponding synthetic speech to provide synthetic speech as part of a verbal interactive exchange between client device 130 and a user. Additionally or alternatively, input 133 includes audio data characterizing verbal speech recorded by client device 130, and on-device ML model 135 performs speech recognition on the audio data characterizing the verbal speech to generate a transcription of the speech as output 136. For example, on-device ML model 135 employed as a speech recognition model may enable client device 130 to recognize a verbal query and then instruct a downstream application to implement the query. Additionally or alternatively, input 133 may include an image, and on-device ML model 135 may perform image classification or object recognition as output 136. In other examples, the on-device ML model 135 includes a speech-to-speech model, a language model, a language translation model, a machine translation model, or other type of neural network model trained by ML to generate an output 136 based on the received input 133.

[0022] During training, the on-device ML engine 134 processes private data 139 stored in a data store 140 (e.g., in memory hardware 138) using the on-device ML model 135 to generate one or more predicted private outputs 141. In some examples, the private data 139 and the public data 160 are derived from a common, similar, or the same source distribution.

[0023] The gradient engine 142 generates one or more differential private (DP) gradients 143 based on the predicted private outputs 141. In some implementations, the gradient engine 142 generates the DP gradients 143 based on comparing the predicted private outputs 141 to private ground truth 144 corresponding to the private data 139 using supervised learning techniques. In additional or alternative implementations, such as when the private ground truth 144 corresponding to the private data 139 is unavailable, the gradient engine 142 generates the DP gradients 143 using supervised and / or unsupervised learning techniques. The client device 130 transmits the DP gradients 143 generated / output from the gradient engine 142 of the ML module 132 to the remote system 110 over the network 170. In some examples, the client device 130 transmits the DP gradients 143 to the remote system 110 as they are generated by the gradient engine 142. Additionally or alternatively, client device 130 may store the DP gradients (e.g., in memory hardware 138) and then transmit the DP gradients 143 in batches to remote system 110. In particular, client device 130 may transmit DP gradients 143 to remote system 110 without transmitting any of private data 139, private ground truth 144, predicted private outputs 141, and / or any other personally identifiable information. In various implementations, client device 130 transmits DP gradients 143 to remote system 110 in response to determining that one or more conditions are met.Example conditions include an indication that the client device 130 is charging, the charge state of the client device 130 meets a threshold charge state, the temperature of the client device 130 (based on one or more on-device temperature sensors) is below a threshold temperature, an indication that the client device 130 is not being held by a user, a temporal condition associated with the client device 130 (e.g., for a particular time period, every N hours, where N is a positive integer, and / or other temporal condition associated with the client device 130), and / or whether a threshold number of DP gradients 143 have been generated by the client device 130.

[0024] In some examples, the gradient engine 142 determines the DP gradient 143 by determining a private loss function based on the predicted private output 141 and the corresponding private ground truth 144, derives a private gradient from the determined private loss function, and adds noise to the derived private gradient to generate the corresponding DP gradient 143. In some examples, the private loss function is convex and L-Lipschitz, where the effect of adding noise in any direction is inversely proportional to the curvature of the private loss function in that direction.

[0025] In additional or alternative implementations, the gradient engine 142 derives the DP gradient 143 from a private loss function used to train the on-device ML model 135, such that the DP gradient 143 represents the value of that private loss function (or a derivative thereof) obtained from comparing the private ground truth 144 with the predicted private output 141 (e.g., using supervised learning techniques). For example, when the private ground truth 144 and the predicted private output 141 match, the gradient engine 142 may generate a zero DP gradient 143. Also, for example, when the private ground truth 144 and the predicted private output 141 do not match, the gradient engine 142 may generate a non-zero DP gradient 143 depending on the degree of discrepancy. The gradient engine 142 may determine the degree of discrepancy based on the degree of discrepancy between the deterministic comparison of the private ground truth 144 and the predicted private output 141. In additional or alternative implementations, the gradient engine 142 can derive the DP gradient 143 from a private loss function used to train the on-device ML model 135, such that the DP gradient 143 represents the value of that private loss function (or a derivative thereof) determined based on the predicted private output 141 (e.g., using supervised or semi-supervised learning techniques).

[0026] As described in more detail below, private data 139 may include audio data generated by a microphone of client device 130, text segments provided as input by a user of client device 130 and / or stored in memory hardware 138, image data captured by an imaging device in communication with client device 130, and / or any other data captured by or generated locally to client device 130 and processed using on-device ML model 135. In some implementations, on-device ML model 135 processes private data 139 to synchronously generate DP gradients 143 as private data 139 is generated or provided to client device 130. In additional or alternative implementations, private data 139 may be stored in data store 140 as private data 139 is generated or provided to client device 130, and private data 139 may then be utilized to asynchronously generate DP gradients 143. In additional or alternative implementations, the on-device ML engine 134 processes the private data 139 to generate the predicted private outputs 141, and the client device 130 stores or caches the predicted private outputs 141, which may be stored or cached on the client device 130 (possibly in association with the private data 139 related to the predicted private outputs 141) for subsequent use by the gradient engine 142 to asynchronously generate DP gradients 143. The private data 139 (also referred to herein as on-device memory or on-device storage) may include data generated or provided to the client device 130, including, but not limited to, audio data, image data, contact lists, electronic messages (e.g., text messages, emails, social media messages, etc.) sent by or received by a user of the client device 130, and / or other client data.In particular, private data 139 corresponds to data with restricted access or that is not publicly available and / or not available to remote systems 110 .

[0027] The remote system 110 includes data processing hardware 112 and memory hardware 113 in communication with the data processing hardware 112. The memory hardware 113 stores instructions that, when executed by the data processing hardware 112, cause the data processing hardware 112 to perform one or more operations.

[0028] During training, the global ML engine 114 of the remote system 110 processes public data 160 using the global ML model 150 to generate predicted public outputs 115. The public data 160 may be obtained from a data store 121 (e.g., residing in memory hardware 113) of public data 160. In some examples, the private data 139 and the public data 160 are derived from a common, similar, or the same source distribution. Although the outputs 115 are referred to herein as predicted public outputs 115 to indicate that they are generated based on the public data 160, they are not necessarily disclosed outside the remote system 110. However, the predicted public gradients 117 may be made public. The data store 119 may include data accessible by the remote system 110, including, but not limited to, public data repositories including audio data, text data, and / or image data, as well as private data repositories. Additionally, the data store 119 may include data from different types of client devices 130 having different device characteristics or components. For example, database 119 may include audio data captured by a near-field microphone (e.g., similar to audio data captured by client device 130) and audio data captured by a far-field microphone (e.g., audio data captured by other devices). As another example, database 119 may include image data (or other visual data) captured by different visual components, such as RGB image data, RGB-D image data, CMYK image data, and / or other types of image data captured by various different visual components. Additionally, remote system 110 may employ one or more techniques on public data 160 to modify public data 160.These techniques may include filtering the audio data to add or remove noise when the public data 160 is audio data, blurring the image when the public data 160 is image data, and / or other techniques for manipulating the public data 160. This may allow the remote system 110 to better reflect the client data generated by multiple different client devices 130 and / or meet the needs of a particular type of data (e.g., inducing false positives or false negatives as described herein, ensuring sufficient diversity of the audio data as described herein, etc.).

[0029] The gradient engine 116 generates one or more public gradients 117 based on the predicted public outputs 115. Although the gradients 117 are referred to herein as public gradients 117 to indicate that they are generated based on public data 160, they are not necessarily disclosed outside the remote system 110. However, the public gradients 117 may be made public. In some implementations, the gradient engine 116 generates the public gradients 117 based on comparing the predicted public outputs 115 to public ground truth 118 corresponding to the public data 160 using supervised learning techniques. In additional or alternative implementations, such as when the public ground truth 118 corresponding to the public data 160 is unavailable, the gradient engine 116 can generate the public gradients 117 using supervised and / or unsupervised learning techniques. The public gradients 117, along with the DP gradients 143 received from the client device 130, may be stored in a gradient data store 119 stored in a central repository on the remote system 110 (e.g., long-term and / or short-term memory, such as memory hardware 113 or a buffer).

[0030] In some examples, the gradient engine 116 determines a public loss function based on the public prediction output 115 and the corresponding public ground truth 118, and determines the public gradient 117 by deriving the public gradient from the determined public loss function. In some examples, the public loss function is strongly convex.

[0031] As described above, the public and / or private gradients 117, 143 may be stored in the gradient data store 119 (or other memory (e.g., buffer)) as the gradients 117, 143 are generated and / or received. In some implementations, the gradients 117, 143 may be indexed by the type of gradient determined from among multiple different types of gradients based on the corresponding on-device ML model 135 that processed the private data 139 and / or the corresponding global ML model 150 that processed the public data 160. The multiple different types of gradients may be defined with varying degrees of granularity. For example, the types of gradients may be specifically defined as, e.g., hot-word gradients generated based on processing audio data using a hot-word model, ASR gradients generated based on processing audio data, VAD gradients generated based on processing audio data using a VAD model, continuous conversation gradients generated based on processing using a continuous conversation model, speech identification gradients generated based on processing audio data using a speech identification model, face identification gradients generated based on processing image data using a face identification model, hot-word-free gradients generated based on processing image data using a hot-word-free model, object detection gradients generated based on processing image data using an object detection model, text-to-speech (TTS) gradients generated based on processing text segments using a TTS model, and / or any other gradient that may be generated based on processing data using any other ML model. In particular, a given one of the gradients 117, 143 may belong to one of multiple different types of gradients. As another example, gradient types may be defined more generally as, for example, audio-based gradients generated based on processing audio data using one or more audio-based models, image-based gradients generated based on processing image data using one or more image-based models, or text-based gradients generated based on processing text segments using a text-based model.

[0032] The training engine 200 can utilize the DP gradients 143 and the public gradients 117 to update one or more weights of the global ML model 150. In some implementations, the remote system 110 assigns the public gradients 117 and the DP gradients 143 to a particular iteration of updating the global ML model 150 based on one or more criteria. The one or more criteria can include, for example, the types of gradients available to the training engine 200, a threshold amount of gradients available to the training engine 200, a threshold duration for updating using gradients, and / or other criteria. Specifically, the training engine 200 can identify multiple sets or subsets of DP gradients 143 and / or public gradients 117 to use to train the global ML model 150. Furthermore, the training engine 200 can update the global ML model 150 based on these sets or subsets of gradients. In some further versions of these implementations, the amount of gradients in the set of DP gradients 143 and in the set of public gradients 117 can be the same or different (e.g., proportional to each other, having more DP gradients 143 or more public gradients 117). In other implementations, the remote system 110 utilizes the DP gradients 143 and public gradients 117 to update the global ML model 150 in a first-in, first-out (FIFO) manner without allocating gradients 117, 143 to the particular iteration that updates the global ML model 150.

[0033] 2 is a schematic diagram of a training process 200 that leverages public data 160 using private mirror descent to train a global ML model 150. The training process 200 applies mirror descent to public gradients 117 to learn the geometry 215 of the public gradients 117. The training process 200 may apply mirror descent by using the public gradients 117 derived from a public loss function as a mirror map to learn the geometry 215 of a set of DP gradients.

[0034] The training process 200 uses the learned geometry 215 to reshape the DP gradients 143 so that the reshaped DP gradients 225 match the learned geometry 215. The training process 200 then trains the global ML model 150 by learning updated weights 235 of the global ML model 150. In some examples, the training process 200 updates the weights 235 using stochastic gradient descent.

[0035] The distribution engine 111 may transmit the updated global ML model 150 and / or its weights to the client device 130. In some embodiments, the distribution engine 111 transmits the updated global ML model 150 and / or its weights in response to one or more conditions being satisfied for the client device 130 and / or the remote system 110. Upon receiving the updated global ML model 150 and / or its weights, the client device 130 replaces or updates the corresponding on-device ML model 135 with the updated global ML model 150, or replaces the weights of the corresponding on-device ML model 135 with the weights of the updated global ML model 150. Furthermore, the client device 130 may subsequently use the updated on-device ML model 135 and / or its weights to make predictions based on further user inputs 133 detected at the client device 130. The client device 130 can continue to generate further DP gradients 143 in the manner described herein and transmit the further DP gradients 143 to the remote system 110. Additionally, the remote system 110 may continue to generate additional public gradients 117 in the manner described herein and update the global ML model 150 based on the additional DP gradients 143 and / or the additional public gradients 117.

[0036] 3 is a flowchart of an example configuration of operations for a computer-implemented method 300 for leveraging public data 160 in training a neural network using private mirror descent. During initial or pre-training of the machine learning model 150, the method performs operations 302 and 304. In operation 302, the method 300 includes obtaining a set of public gradients 117, each generated based on processing corresponding public data 160. In operation 304, the method 300 includes applying mirror descent to the set of public gradients 117, which may be applied to or for the set of DP gradients 143 or to learn the geometry 215 of the public gradients 117. For example, by using the derived public gradients 117 as a mirror map to learn the geometry 215 for the set of DP gradients 143.

[0037] During subsequent training or updating of the machine learning model 150, the method performs operations 306, 308, and 310. In operation 306, the method 300 includes obtaining a set of differential private (DP) gradients 143, each generated based on processing corresponding private data 139. In operation 308, the method includes reshaping the set of DP gradients 143 based on the learned geometry 215. In operation 310, the method 300 includes training or updating the machine learning model 150 based on the reshaped set of DP gradients.

[0038] 4 is a schematic diagram of an exemplary computing device 400 that may be used to implement the systems and methods described herein. Computing device 400 is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. The components shown here, their connections and relationships, and their functionality are intended to be exemplary only and are not intended to limit the implementation of the invention described and / or claimed herein.

[0039] Computing device 400 includes a processor 410 (i.e., data processing hardware) that may be used to implement data processing hardware 137 and / or 112, a memory 420 (i.e., memory hardware) that may be used to implement memory hardware 138 and / or 113, a storage device 430 (i.e., memory hardware) that may be used to implement memory hardware 138 and / or 113, a high-speed interface / controller 440 that connects to memory 420 and subsequent expansion port 450, and a low-speed interface / controller 460 that connects to a low-speed bus 470 and storage device 430. Each of components 410, 420, 430, 440, 450, and 460 are interconnected using various buses and may be mounted on a common motherboard or in other manners as appropriate. The processor 410 can process instructions for execution within the computing device 400, including instructions stored in memory 420 or on storage device 430, for displaying graphical information for a graphical user interface (GUI) on an external input / output device, such as a display 480 coupled to a high-speed interface 440. In other implementations, multiple processors and / or multiple buses may be used, along with multiple memories and multiple types of memory, as appropriate. Multiple computing devices 400 may also be connected, with each device providing a portion of the required operations (e.g., as a bank of servers, a group of blade servers, or a multiprocessor system).

[0040] The memory 420 stores information non-transiently within the computing device 400. The memory 420 may be a computer-readable medium, a volatile memory unit, or a non-volatile memory unit. The non-transient memory 420 may be a physical device used to temporarily or permanently store programs (e.g., sequences of instructions) or data (e.g., program state information) for use by the computing device 400. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electrically erasable programmable read-only memory (EEPROM) (e.g., typically used for firmware such as boot programs). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase change memory (PCM), and disk or tape.

[0041] The storage device 430 is capable of providing mass storage for the computing device 400. In some implementations, the storage device 430 is a computer-readable medium. In various different implementations, the storage device 430 may be a floppy disk device, a hard disk device, an optical disk device, or an array of devices including a tape device, a flash memory or other similar solid-state memory device, or devices in a storage area network or other configuration. In additional embodiments, a computer program product is tangibly embodied in an information carrier. The computer program product includes instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer-readable or machine-readable medium, such as the memory 420, the storage device 430, or memory on the processor 410.

[0042] The high-speed controller 440 manages bandwidth-intensive operations for the computing device 400, and the low-speed controller 460 manages lower-bandwidth-intensive operations. Such allocation of roles is merely exemplary. In some implementations, the high-speed controller 440 is coupled to memory 420, a display 480 (e.g., through a graphics processor or accelerator), and a high-speed expansion port 450 that can accept various expansion cards (not shown). In some implementations, the low-speed controller 460 is coupled to a storage device 430 and a low-speed expansion port 490. The low-speed expansion port 490, which may include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet), may be coupled, for example, through a network adapter, to one or more input / output devices such as a keyboard, pointing device, scanner, or networking device such as a switch or router.

[0043] Computing device 400 may be implemented in several different forms, as shown in the figure. For example, computing device 400 may be implemented as a standard server 400a, or multiple times in a group of such servers 400a, as a laptop computer 400b, or as part of a rack server system 400c.

[0044] Various implementations of the systems and techniques described herein may be realized in digital electronic and / or optical circuitry, integrated circuits, specially designed ASICs (application-specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include implementation in one or more computer programs executable and / or interpretable on a programmable system including at least one programmable processor, which may be special purpose or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0045] A software application (i.e., a software resource) may refer to computer software that causes a computing device to perform tasks. In some examples, a software application may be referred to as an "application," "app," or "program." Exemplary applications include, but are not limited to, system diagnostic applications, system management applications, system maintenance applications, word processing applications, spreadsheet applications, messaging applications, media streaming applications, social networking applications, and gaming applications.

[0046] These computer programs (also known as programs, software, software applications, or code) contain machine instructions for a programmable processor and may be implemented in a high-level procedural and / or object-oriented programming language and / or in an assembly / machine language. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, non-transitory computer-readable medium, apparatus, and / or device (e.g., magnetic disk, optical disk, memory, programmable logic device (PLD)) used to provide machine instructions and / or data to a programmable processor, including a machine-readable medium that receives the machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0047] The processes and logic flows described herein may be implemented by one or more programmable processors, also referred to as data processing hardware, that execute one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows may also be implemented by special purpose logic circuitry, such as FPGAs (field programmable gate arrays) and ASICs (application-specific integrated circuits). Processors suitable for executing computer programs include, by way of example, both general-purpose and special-purpose microprocessors, as well as any one or more processors of any kind of digital computer. Generally, a processor will receive instructions and data from a read-only memory or a random-access memory, or both. The essential elements of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Typically, a computer also includes one or more mass storage devices (e.g., magnetic, magneto-optical, or optical disks) for storing data, or is operably coupled to receive data from or transfer data to one or more mass storage devices, or both. A computer need not, however, have such devices. Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, by way of example, semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices, magnetic disks, e.g., internal hard disks or removable disks, magneto-optical disks, and CD-ROM and DVD-ROM disks. The processor and the memory can be supplemented by, or incorporated in, special purpose logic circuitry.

[0048] To enable interaction with a user, one or more aspects of the present disclosure may be implemented on a computer having a display device, e.g., a CRT (cathode ray tube), LCD (liquid crystal display) monitor, or touch screen, for displaying information to the user, and optionally a keyboard and pointing device, e.g., a mouse or trackball, by which the user can provide input to the computer. Other types of devices may also be used to enable interaction with the user; for example, feedback provided to the user may be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback, and input from the user may be received in any form, including acoustic, speech, or tactile input. Additionally, the computer may interact with the user by sending documents to and receiving documents from a device used by the user, e.g., by sending a web page to a web browser on the user's client device in response to a request received from the web browser.

[0049] Unless otherwise specified, "or" refers to an inclusive "or," not an exclusive "or." For example, "A, B, or C" refers to any combination or subset of A, B, and C, such as (1) A only, (2) B only, (3) C only, (4) A and B, (5) A and C, (6) B and C, and (7) A, B, and C. Similarly, the phrase "at least one of A or B" refers to any combination or subset of A and B, such as (1) at least one A, (2) at least one B, or (3) at least one A and at least one B. Furthermore, the phrase "at least one of A and B" refers to any combination or subset of A and B, such as (1) at least one A, (2) at least one B, or (3) at least one A and at least one B.

[0050] Although several implementations have been described, it will nevertheless be understood that various modifications may be made without departing from the spirit and scope of the present disclosure. Accordingly, other implementations are within the scope of the following claims. [Explanation of symbols]

[0051] 100 systems 101 Machine Learning (ML) Environment 110 Remote Systems 111 Distribution Engine 112 Data Processing Hardware 113 Memory Hardware 114 Global ML Engine 115 Predicted Public Output 116 Gradient Engine 117 Slope 118 Public Ground Truth 119 Datastore / Database 121 Datastore 130 client devices 132 ML module 133 inputs 134 On-device ML engine 135 on-device ML models 136 Output 137 Data Processing Hardware 138 Memory Hardware 139 Private Data 140 data stores 141 Predictive Private Output 142 Gradient Engine 143 Differential Private (DP) Gradients 144 Private Ground Truth 150 global ML models 160 Public Data 170 Communication Network 200 Training Engine 215 Geometry 225 Reshaped DP gradient 235 Weight 400 computing devices 410 processor 420 memory 430 Storage Devices 440 High-Speed ​​Interface / Controller 450 High-Speed ​​Expansion Port 460 Low-Speed ​​Interface / Controller 470 Slow Bus 480 display 490 Low-Speed ​​Expansion Port

Claims

1. A computer-implemented method (300) that, when executed on data processing hardware (410), causes the data processing hardware (410) to perform operations, the operations including: obtaining a set of differential private (DP) gradients (143), each generated based on processing corresponding private data (139); obtaining a set of public gradients (117), each generated based on processing corresponding public data (160); applying image descent to the set of public gradients (117) to learn a geometry (215) for the set of DP gradients (143); reshaping the set of DP gradients (143) based on the learned geometry (215); training a machine learning model (150) based on the set of reshaped DP gradients (225); A method (300) comprising:

2. Each DP gradient (143) in the set of DP gradients (143) is processing the corresponding private data (139) using the machine learning model (135) to generate corresponding predicted private outputs (141); determining a private loss function based on the corresponding predicted private output (141) and the corresponding private ground truth (144); adding noise to the private gradient derived from the private loss function to generate the DP gradient (143); The method (300) of claim 1,

3. 3. The method (300) of claim 2, wherein the private loss function is convex and L-Lipschitz.

4. 2. The method of claim 1, wherein the private data and the public data are derived from the same source distribution.

5. Each public gradient (117) in the set of public gradients (117) is processing the corresponding public data (160) using the machine learning model (150) to generate corresponding predicted public outputs (115); determining a public loss function based on the corresponding predicted public outputs (115) and the corresponding public ground truth (118); deriving the public gradient (117) from the public loss function; The method (300) of claim 1,

6. 6. The method of claim 5, wherein applying mirror descent to the set of public gradients to learn the geometry for the set of DP gradients comprises applying mirror descent by using the public gradients derived from the public loss function as a mirror map to learn the geometry for the set of DP gradients.

7. 7. The method (300) of claim 5 or 6, wherein the public loss function is strongly convex.

8. the data processing hardware (410) is located in a central server; 2. The method of claim 1, wherein the set of DP gradients and the set of public gradients are stored in a central repository located on the central server.

9. the data processing hardware (410) is in a remote system; obtaining the set of DP gradients (143) includes receiving the set of DP gradients (143) from one or more client devices (130) by federated learning without receiving any of the corresponding private data (139); 2. The method of claim 1, wherein each DP gradient in the set of DP gradients is generated locally at a respective one of the one or more client devices.

10. The method of claim 1 , wherein the machine learning model comprises an image classification model.

11. The method of claim 1 , wherein the machine learning model comprises a language model.

12. The method of claim 1 , wherein the machine learning model comprises a speech recognition model.

13. data processing hardware (410); memory hardware (420) in communication with said data processing hardware (410); the memory hardware (420) stores instructions that, when executed on the data processing hardware (410), cause the data processing hardware (410) to perform operations, the operations comprising: obtaining a set of differential private (DP) gradients (143), each generated based on processing corresponding private data (139); obtaining a set of public gradients (117), each generated based on processing corresponding public data (160); applying image descent to the set of public gradients (117) to learn a geometry (215) for the set of DP gradients (143); reshaping the set of DP gradients (143) based on the learned geometry (215); training a machine learning model (150) based on the set of reshaped DP gradients (225); Including, the system.

14. Each DP gradient (143) in the set of DP gradients (143) is processing the corresponding private data (139) using the machine learning model (135) to generate corresponding predicted private outputs (141); determining a private loss function based on the corresponding predicted private output (141) and the corresponding private ground truth (144); adding noise to the private gradient derived from the private loss function to generate the DP gradient (143); The system of claim 13, wherein the system is generated by

15. 15. The system of claim 14, wherein the private loss function is convex and L-Lipschitz.

16. The system of claim 13 , wherein the private data (139) and the public data (160) are derived from the same source distribution.

17. Each public gradient (117) in the set of public gradients (117) is processing the corresponding public data (160) using the machine learning model (150) to generate corresponding predicted public outputs (115); determining a public loss function based on the corresponding predicted public outputs (115) and the corresponding public ground truth (118); deriving the public gradient (117) from the public loss function; The system of claim 13, wherein the system is generated by

18. 18. The system of claim 17, wherein applying mirror descent to the set of public gradients to learn the geometry for the set of DP gradients comprises applying mirror descent by using the public gradients derived from the public loss function as a mirror map to learn the geometry for the set of DP gradients.

19. 19. The system of claim 17 or 18, wherein the public loss function is strongly convex.

20. the data processing hardware (410) is located in a central server; The system of claim 13, wherein the set of DP gradients (143) and the set of public gradients (117) are stored in a central repository (119) on the central server.

21. the data processing hardware (410) is in a remote system; obtaining the set of DP gradients (143) includes receiving the set of DP gradients (143) from one or more client devices (130) by federated learning without receiving any of the corresponding private data (139); The system of claim 13 , wherein each DP gradient (143) in the set of DP gradients (143) is generated locally at a respective client device of the one or more client devices (130).

22. The system of claim 13 , wherein the machine learning model (150) comprises an image classification model.

23. The system of claim 13 , wherein the machine learning model (150) comprises a language model.

24. The system of claim 13 , wherein the machine learning model (150) comprises a speech recognition model.

Images