Systems and methods for efficient differentially private set union

Abstract

Scalable private set union adaptively reroutes weights from items with weights above a privacy threshold to items with smaller weights, thus increasing the probability to output less frequent items. Scalable private set union can output larger subsets of items while maintaining the same level of differential privacy when compared to alternative techniques. The larger subsets of items can also be output with less processing cost and memory usage compared to alternative techniques.

Description

SYSTEMS AND METHODS FOR EFFICIENT DIFFERENTIALLY PRIVATE SET UNIONCROSS-REFERENCE TO RELATED APPLICATIONS

[0001] The present application claims priority from European Patent Application No. 24306024.1, filed June 25, 2024, the disclosure of which is hereby incorporated herein by reference.BACKGROUND

[0002] In private set union, each user has a private subset of items from an unknown and potentially infinite universe of items, and the goal is to output as many of the items which appear in the union of the sets of the users as possible while respecting the privacy of the users. For example, the goal may be to extract as many words used in emails of users as possible without leaking information about any of the users. Private set union may also be referred to as private partition selection or private key selection. Other applications for private set union include privately outputting queries asked to a search engine, discovering words and phrases used in emails or text messages, analyzing data streams, sparse histograms, and / or structured query language operations, as examples.

[0003] Within privacy preserving machine learning, private set union can be used to address a preliminary step of vocabulary, tokens, and n-gram extraction. Identifying which tokens or n- grams are used in a private corpus is utilized in many privacy-preserving natural language processing algorithms, including training language models for sentence completion and response generation for emails. Moreover, machine learning techniques often handle categorical features by creating a large embedding table with a separate embedding for each category. Identifying relevant features or categories to learn embeddings is useful to improve privacy-preserving machine learning techniques. Private set union also has applications in the sparsification of gradients in the differential private standard gradient descent technique for training machine learning models. Real-world datasets for these applications can be massive, leading to higher processing costs and memory usage that preclude solutions where the input in-memory is stored on a single physical machine or the dataset is processed sequentially.

[0004] Differential privacy allows for, depending on how privacy parameters are configured, the output for private set union to not depend significantly on any particular user set. For example, differential privacy guarantees that a private set union cannot output items which appear in only a single set but can output items which appear in many different sets. Typically, private set union starts by subsampling each user set to bound a maximum number of items per user. Then, privateset union assigns a weight to each item in the union, adds configurable noise to the accumulated weight of each item, and outputs all items with noised weight above a configurable threshold. Example noise can be Gaussian or Laplace noise. The amount of noise and value of the threshold is based on how the privacy parameters are configured as well as how sensitive the weighting assignment is to the addition or removal of any particular user set. Overall, the contribution of each user to the item weights are bounded to achieve differential privacy, with a goal of limiting the sensitivity of the weighting assignment.

[0005] For example, each user can contribute equal weight to each of the items in their set in order to bound the sensitivity for differential privacy. However, this weight contribution is lossy in that it can over-allocate weights to high frequency items, missing the opportunity to boost the weight of items closer to the configurable threshold. Other known weighting contributions that attempt to more accurately distribute weights become greedy and inherently sequential, resulting in increased processing cost and memory usage such that the weighting contribution cannot be implemented in a distributed framework. Further, while re-assigning equal weight contributions multiple times may improve the overall weights of items, this still requires increased processing cost and memory usage to run the private set union technique through numerous iterations.BRIEF SUMMARY

[0006] Aspects of the disclosure are directed to scalable private set union. In private set union, users contribute sets of items as input with a goal to increase, e.g., maximize, output of a subset of the union of all items while maintaining user-level differential privacy. Private set union can be utilized in many privacy-preserving machine learning applications, such as vocabulary extraction or identification of categorical data. Scalable private set union adaptively reroutes weights from items with weights above a privacy threshold to items with smaller weights, thus increasing the probability to output less frequent items. Scalable private set union can output larger subsets of items while maintaining the same level of differential privacy when compared to alternative techniques. Further, scalable private set union can output the larger subsets of items with less processing cost and memory usage, since, for example, weight reassignment does not need to be performed as frequently.

[0007] An aspect of the disclosure provides for a method for generating a private set union including: receiving, by one or more processors, a plurality of data sets, each data set of the plurality of data sets including a plurality of items, and corresponding weights for each item of each of the plurality of items; determining, by the one or more processors, that the weights for some items of the plurality of items are above a reweighting threshold; truncating, by the one or more processors, the weights for the items that are above the reweighting threshold, whereinexcess weight that was truncated is to be rerouted to other items of the plurality of items that are not above the reweighting threshold; adding, by the one or more processors, noise to the weights for each item of the plurality of items to generate noisy weights; determining, by the one or more processors, that the noisy weights for some items of the plurality of items are above an output threshold; and combining, by the one or more processors, the items with noisy weights above the output threshold to form a data set corresponding to the private set union.

[0008] Another aspect of the disclosure provides for a system including: one or more processors; and one or more storage devices coupled to the one or more processors and storing instructions that, when executed by the one or more processors, cause the one or more processors to perform operations for the method for generating a private set union. Yet another aspect of the disclosure provides for a non-transitory computer readable medium storing instructions that, when executed by one or more processors, causes the one or more processors to perform operations for the method for generating a private set union. Yet another aspect of the disclosure provides for a computer program including instructions that, when executed by one or more processors, cause the one or more processors to perform operations for the method for generating a private set union.

[0009] In some examples, the method further includes subsampling, by the one or more processors, each data set of the plurality of data sets based on a maximum degree cap. In some examples, the method further includes determining, by the one or more processors, that a cardinality of some data sets of the plurality of data sets are above a cardinality threshold, where uniform weights are contributed to the items in the data sets determined to be above the cardinality threshold. In some examples, the cardinality threshold is a number between 1 and a maximum degree cap.

[0010] In some examples, the reweighting threshold is based on the output threshold, a standard deviation value determined by one or more privacy parameters, and a free parameter greater than or equal to 0. In some examples, the reweighting threshold is greater than the output threshold.

[0011] In some examples, the method further includes outputting, by the one or more processors, the excess weight for rerouting in proportion to amounts contributed. In some examples, the method further includes receiving, by the one or more processors, the excess weight rerouted to the other items of each of the plurality of items not above the reweighting threshold.

[0012] In some examples, the noise includes at least one of Gaussian or Laplace noise based on one or more privacy parameters.

[0013] In some examples, the method further includes receiving, by the one or more processors, one or more privacy parameters for setting a level of differential privacy, a maximum degree cap for determining a maximum number of items in a data set, a cardinality threshold for determining whether to participate in adaptive reweighting, the output threshold for determining whether anitem is to be included in the private set union, and a free parameter for determining whether an item is to have its weight truncated.

[0014] In some examples, the data sets are associated with at least one of outputting queries asked to a search engine, discovering words or phrases used in messages, analyzing data streams, outputting sparse histograms, or structured query language operations. In some examples, the data sets are associated with privacy preserving machine learning applications. In some examples, the privacy preserving machine learning applications include at least one of vocabulary, token, or n- gram extraction, identifying relevant features or categories to learn embeddings, or sparsification of gradients in differential privacy standard gradient descent.

[0015] In some examples, the method further includes training, by the one or more processors, a machine learning model using the data set corresponding to the private set union. In some examples, the plurality of data sets include sets of emails from respective users and the data set corresponding to the private set union includes a data set of emails from the respective users that preserves differential privacy. In some examples, training the machine learning model further includes training the machine learning model to generate email drafts in response to prompts to generate the email drafts.BRIEF DESCRIPTION OF THE DRAWINGS

[0016] FIG. 1 depicts a block diagram of an example scalable private set union according to aspects of the disclosure.

[0017] FIG. 2 depicts a block diagram of a scalable private set union system according to aspects of the disclosure.

[0018] FIG. 3 depicts a block diagram of an example environment for implementing a scalable private set union system according to aspects of the disclosure.

[0019] FIG. 4 depicts a block diagram illustrating one or more machine learning model architectures according to aspects of the disclosure.

[0020] FIG. 5 depicts a flow diagram of an example process for generating a private set union according to aspects of the disclosure.

[0021] FIG. 6 depicts a flow diagram of an example process for training a machine learning model according to aspects of the disclosure.DETAILED DESCRIPTION

[0022] The technology generally relates to an adaptive, non-uniform private set union, also referred to herein as scalable private set union, in which over-allocated weights are re-routed to less frequent items. This private set union can be implemented in a distributed system, since itsprocessing cost and memory usage is significantly lower than alternative approaches while still maintaining the same level of differential privacy.

[0023] A system including one or more processors implementing scalable private set union determines whether each user set has a cardinality, e.g., number of elements in their set, that is equal to or above a threshold. Cardinality may also be referred to as degree. Scalable private set union labels any user sets with a cardinality above this threshold as non-adaptive. User sets that are non-adaptive have weights distributed uniformly across the items in their set. The remaining user sets with cardinality below the threshold may be labeled as adaptive to allow for participation in the adaptive reweighting that accounts for differential privacy.

[0024] For adaptive reweighting, a small amount of weights is initially assigned uniformly among items per set, where the total amount of weight per user set is bounded by a predetermined total weight value, e.g., 1. Then, a scalable private set union determines whether any items have weights significantly above a threshold used to determine output for the scalable private set union. The scalable private union reduces, e.g., truncates, any of the items with weights significantly above this threshold to be only slightly above this threshold. The excess weight removed via truncation is returned, proportional to the initial weight contributions. A fraction of this excess weight is reassigned among the items per set and additional uniform weight is added to the items to make up for the smaller amount of weight initially assigned.

[0025] For scalable private union, items have their weight truncated if the weight exceeds an adaptive threshold T. The adaptive threshold T can be set to be ft standard deviations of the noise above the threshold used to determine output for the scalable private set union, where ? > 0 is a free parameter. By free parameter, / 3 can be any number greater than or equal to 0. For example, / 3 can be set to any number from 1 to 2. A system implementing scalable private set union increases outputs while guaranteeing differential privacy with the same privacy parameters as alternative approaches, but with lower processing cost and memory usage. For example, a system may implement a scalable private set union to pre-process more data as part of a machine learning training pipeline, ensuring differential privacy relative to the same set of privacy parameters used in a uniform weighting process or mechanism.

[0026] In private set union, there are n users with each user u having a set Suof items from an unknown and possibly infinite universe of items, where input may be defined as S = {(it, Su)}ue[n]. The goal is to output a subset of the union of the user sets U =UME[n] Suof maximum cardinality while maintaining user- level differential privacy. Two input datasets S and S' are neighboring if one can be obtained by removing a single user set from the other, e.g., S' = S U {( , Sv)} for some new user v. A randomized function M is (f, 8 ) -differentially private if, forany two neighboring datasets S and S' and for any possible subset of outputs 0 {IP. U £ [ / }, then Pr (MCS) G O) < e£-Pr (M(S') G O) + S, where E and S are configurable privacy parameters that indicate the amount of allowable differential privacy loss. <£: / ? -> / ? is the standard Gaussian cumulative density function and / : D -> Rdis a function with l2sensitivity d2. For any E > 0 and 6 E [0,1], the mechanismdifferentially private if ~ ~) < 8. Scalable private set union can satisfythese features of differential privacy while providing larger outputs with less processing cost and memory usage.

[0027] The following is an example weight and threshold methodology according to aspects of the disclosure. Inputs include a set of user sets S = {(it, Su)}ue[n] , privacy parameters E and 8, a maximum degree cap d0, and a weighting technique “WeightingAlgo”. Inputs may also include a maximum adaptive degree dmax> 1 and an adaptive threshold excess parameter / ? > 0. The output of this example process can include a subset of the union of user sets U = U[]=1SU. The methodology may include the following steps, as examples:

[0028] Select (5 corresponding to a Gaussian mechanism for with A2= 1.

[0029] Set threshold p = — 0() to cap user degrees.

[0030] For all it G [n] do:

[0031] If |SJ > Aothen:

[0032] Randomly subsample Suto Aoitems.

[0033] If WeightingAlgo is adaptive weighting technique, then:

[0034] w <- Weighting Algols, p + (3a, dmax).

[0035] Else:

[0036] w <- Weighting Algols').

[0037] w «- w + 1V(O, <J2 / ) .

[0038] Return {t G U: w(i') > p.

[0039] Each user set is randomly subsampled so that size of each resulting set is at most the maximum degree cap 40. Then, the system implementing the example weight and thresholding process above receives the cardinality-capped sets and outputs a set of weights over all items in the union of the cardinality-capped sets. Noise, e.g., independent Gaussian noise, with standard deviation a is added to each coordinate of the weights. Items with a weight above a threshold p are output. The system outputs items which belong to the true union U with the size of the output depending on the number of items with noised weight above the threshold p. By true union, the methodology may omit outputting an item but may not generate an item that was not in any inputuser sets.

[0040] The privacy of this methodology depends on sensitivity properties of the weighting technique as well as the choice for standard deviation a and threshold p, as these parameters are configurable. For any pair of neighboring inputs 5 and S' = 5 U {(v,5v)}, corresponding unions U and U', and items weights w and w' assigned by the weighting technique on the neighboring inputs, the l2sensitivity of the weighting technique is defined as the smallest value d2such that for all possible inputs {Su}”=1and a new user set Sv, d2= .Given bounded l2sensitivity, choosing the scale of noise a appropriately ensures that outputting the noised weights on items in U satisfy (e, 3 / 2) -differential privacy. Therefore, knowing U allows for the output of the methodology after thresholding to be private via post-processing. The challenge is that there may be items in U' which do not appear in U. These items may be referred to as T = U' \ U where t = |T|. As long as the probability that any of these items are output by the methodology is at most then (e, S) -differential privacy can be maintained. For a single itemi G T which has zero probability of being output by a weight and threshold methodology run on 5 but is given some weightwhen the weighting technique is run on S', this item i will be output if after adding the noise with standard deviation cr, the noised weight exceeds p. The probability that any item in T is output follows from a union bound. Union bounding only over finitely many events is based on t < d0.

[0041] The Zco sensitivity of a weighting technique is parametrized by the number of items t = | T | which are unique to a new user. Thesensitivity may be defined as the smallest value(t) such that for all possible inputs {5U}”=1and a new user set Sv, dco(t) > w'(t) . Then, the calculation of p to obtain (f, 3) -differential privacy is obtained based on thesensitivity, 3, cr, x and d0. If a weighting technique has bounded l2andsensitivities d2< 1 and d^-Tt) < — , then vt the methodology run with that weighting technique satisfies (e, 3) -differential privacy.

[0042] The following is an example adaptive weighting technique according to aspects of the disclosure. The adaptive weighting technique is amenable to distributed implementations even with the same l2andsensitivities as alternative techniques, allowing for lower processing cost and memory usage while maintaining the same level of differential privacy. Inputs include user sets 5 = {(it, 5M)}Me[n] , threshold T > 0, maximum adaptive degree dmax> 1. Output is w: U -> R weighting of the items. The adaptive weighting technique may include the following steps, as examples:

[0043] Initialize weight vectors w, winit, wtrunc, wreroutewith zeros.1 ?

[0044] Set reroute discount factor a = (1 - , ) .Zy d-max

[0045] Iadapt = {w G [n]: |SU| < dmax} where users with a small enough cardinality participate in adaptivity.

[0046] For all u G Iadaptdo:contribution.

[0052] wreroute+= aeul dmax for i 6 Suwhere excess weight is rerouted to items, discounted by cc I dmax.

[0053] w <- wtrunc+ wreroutewhich is the total Zxbounded adaptive weights.

[0054] For all u G Iadaptdo:

[0055] for i G Suwhere l2bounded weight is added and initial weights aresubtracted.

[0056] For all u G [n]\ / adaptdo:

[0057] w(i)+= -j== for i G Suwhere basic l2bounded weight is added for non-adaptive items. V l^u I

[0058] Return w.

[0059] The adaptive weighting technique receives two additional parameters: a maximum adaptive degree dmaxG [l,40] and an adaptive threshold T = p + flo for a free parameter ft >0. User sets with cardinalities greater than dmaxare set aside and contribute uniform weights to their items at the end of the weighting technique. The rest of the user sets participate in the adaptive reweighting. This adaptive reweighting starts from a uniform weighting where, for example, each 1 user set sends ■ — ■ weight to each of their items. Items have their weights truncated to T and any |SUI excess weight is sent back to the user sets proportional to the amount they contributed. User sets then reroute a fraction of this excess weight across their items. This fraction is based on dmaxand a discount factor a. The discount factor can be a Then, each user set adds1 1-=== — : — : to the weight of each of their items.V'TM |SU|&

[0060] Each of these stages allows for linear processing in the size of the input, e.g., the sum ofthe sizes of the user sets. Further, each stage can be implemented in a distributed framework, as the processing cost and memory usage is significantly lower than alternative approaches given that there are a constant number of stages and constant number of rounds.

[0061] The methodology executed with the weighting technique is (e, 3) -differentially private x because the weighting technique hasbounded by (t) < — and l2sensitivity upper bounded V t by 1. Further, for any data, the performance of the weighting technique can be significantly improved over alternative examples while maintaining the same differential privacy. For example, consider n users each with degree 3 as well as a single heavy item i* and m light items, where z* >> m. Each user set includes i* as well as two random light items. Under uniform weighting, each user contributes weights to each of their items. Therefore, the weights for uniform weightingare w(t) * <1'^notherwise. Under the weighting technique,assuming n » T, the weighting technique reroutes almost all of the initial weight on the heavy item i* back the users, so each user has excess weight approximately equal to 1 / 3. For dmax= 3, there is a discount factor a > 0.5, so each user sends approximately 1 / 18 weight to each of its items. The weights under the weighting technique are thus w(t) = T + — if i = i* and w(i) = 18 2n 1 1 1 27n— (-= -I - ) < - — otherwise. Here, the weighting technique allows for assigning close to 10%TH v 3 18 THTi more weight to the light items compared to uniform weighting. If — is close to the threshold p. this gap can have a large effect on the final output size. For example, for n = 15,000, m = 1000, E = 1, and 3 = 10-5, the methodology with weighting technique returns 610 items on average while the methodology with uniform weighting returns 519 items. Therefore, the adaptive weighting technique allows for large set unions while maintaining the same level of differential privacy.

[0062] FIG. 1 depicts a block diagram of an example scalable private set union 100 according to aspects of the disclosure. The scalable private set union 100 can be implemented on one or more computing devices in one or more locations, such as a distributed computing system.

[0063] The scalable private set union 100 can receive respective data sets 102, such as from various users, a central repository, and any other data source. While FIG. 1 depicts three user sets 102A, 102B, 102C, the scalable private set union 100 can receive any number of user sets 102. The data sets 102 can include data associated with various applications incorporating privacy, such as privately outputting queries asked to a search engine, discovering words or phrases used in emails or text messages, analyzing data streams, outputting sparse histograms, and / or structured query language operations. Alternatively, or additionally, the data sets 102 can include data associated with various privacy preserving machine learning applications, such as vocabulary,token, and / or n-gram extraction, identifying relevant features or categories to learn embeddings, and / or sparsification of gradients in differential privacy standard gradient descent.

[0064] In response to receiving the data sets 102 along with initial weights 108 for respective items in the data sets and various configurable parameters, the scalable private set union 100 can generate and output a set union 104 of the data sets 102. The set union 104 includes as many items of the data sets 102 as possible which would appear in the union of the data sets 102 while maintaining differential privacy for the items in the data sets 102. The level of differential privacy to be maintained is based on some of the configurable parameters, such as differential privacy parameters E and 8. Items with a weight above a threshold p after noise is added are included in the set union 104, where the threshold p is one of the configurable parameters. Prior to outputting the generated set union 104, the scalable private set union 100 can truncate the weight for some items of the data sets 102, where the weight is above an adaptive threshold T, and output that excess weight 106 to be redistributed to other items in the respective data sets 102 proportional to the amount initially contributed. The scalable private set union 100 can receive the rerouted weights 108 for generating the set union 104. The adaptive threshold T = p + flo where ft is a free parameter ? > 0 and a is a standard deviation of the added noise.

[0065] FIG. 2 depicts a block diagram of a scalable private set union system 200 according to aspects of the disclosure. The scalable private set union system 200 can be implemented on one or more computing devices in one or more locations. The scalable private set union system 200 can correspond to or be a part of the scalable private set union 100 as depicted in FIG. 1.

[0066] The scalable private set union system 200 can be configured to receive input data 202. For example, the scalable private set union system 200 can receive the input data 202 as part of a call to an application programming interface (API) exposing the scalable private set union system 200 to one or more computing devices. The input data 202 can also be provided to the scalable private set union system 200 through a storage medium, such as remote storage connected to the one or more computing devices over a network. The input data 202 can further be provided as input through a user interface on a client computing device coupled to the scalable private set union system 200. The user interface can include a natural language interface, such as one or more text boxes, and / or a graphical interface, such as one or more sliders, checkboxes, and / or templates.

[0067] The input data 202 can include a plurality of data sets associated with various applications incorporating privacy, such as privately outputting queries asked to a search engine, discovering words or phrases used in emails or text messages, analyzing data streams, outputting sparse histograms, and / or structured query language operations. Alternatively, or additionally, the plurality of data sets can include data associated with various privacy preserving machine learning applications, such as vocabulary, token, and / or n-gram extraction, identifying relevant features orcategories to learn embeddings, and / or sparsification of gradients in differential privacy standard gradient descent.

[0068] The input data 202 can further include weights for items in each of the plurality of data sets as well as configurable parameters. The configurable parameters can include parameters for setting the level of differential privacy, a cardinality cap for determining the maximum number of items in a data set, an adaptive cardinality cap for determining whether adaptive reweighting can be applied, a threshold for determining whether an item is to be output in the set union, and / or an adaptive threshold for determining whether an item should have its weight truncated.

[0069] From the input data 202, the scalable private set union system 200 can be configured to output one or more results generated as output data 204. As an example, the scalable private set union system 200 can be configured to send the output data 204 for display on a client or user display. As another example, the scalable private set union system 200 can be configured to provide the output data 204 as a set of computer-readable instructions, such as one or more computer programs. The computer programs can be written in any type of programming language, and according to any programming paradigm, e.g., declarative, procedural, assembly, object- oriented, data-oriented, functional, or imperative. The computer programs can be written to perform one or more different functions and to operate within a computing environment, e.g., on a physical device, virtual machine, or across multiple devices. The computer programs can also implement functionality described herein, for example, as performed by a system, engine, module, or model. The scalable private set union system 200 can further be configured to forward the output data 204 to one or more other devices configured for translating the output data into an executable program written in a computer programming language. The scalable private set union system 200 can also be configured to send the output data 204 to a storage device for storage and later retrieval. The output data 204 can be used as part of a training pipeline, e.g., for training a machine learning model to perform a privacy preserving machine learning task.

[0070] The output data 204 can include a set union of the plurality of data sets, where the set union includes as many items of the plurality of data sets as possible that appear in the union of the data sets while maintaining differential privacy for the items in the plurality of data sets. The output data 204 can further include excess weight that was truncated from items in the plurality of data sets, where the excess weight is to be rerouted to other items in the plurality of data sets proportional to the amount contributed prior to truncation.

[0071] The scalable private set union system 200 can include a cardinality engine 206, an adaptive reweighting engine 208, a noise engine 210, and a thresholding engine 212. The cardinality engine 206, adaptive reweighting engine 208, noise engine 210, and thresholding engine 212 can be implemented as one or more computer programs, specially configured electronic circuitry, or anycombination thereof.

[0072] The cardinality engine 206 can be configured to determine whether data sets of the plurality of data sets participate in the adaptive reweighting. The cardinality engine 206 can compare the cardinality of each data set with a cardinality threshold. The cardinality threshold may be based on the adaptive cardinality cap, which may be a number between 1 and a cardinality maximum. The cardinality engine 206 can determine that data sets with a cardinality above the cardinality threshold do not participate in adaptive reweighting while data sets with a cardinality that is equal to or below the cardinality threshold participate in adaptive reweighting. Data sets that do not participate in adaptive reweighting can have uniform weights contributed to their items. The cardinality engine 206 can send those data sets and weights to the noise engine 210, skipping the adaptive reweighting engine 208. The cardinality engine 206 can send the data sets participating in adaptive reweighting to the adaptive reweighting engine 208.

[0073] The adaptive reweighting engine 208 can be configured to determine whether to truncate any item weights for redistribution among the other items of respective data sets. The adaptive reweighting engine 208 can compare the weight of each item of each data set participating in adaptive reweighting to a reweighting threshold. The reweighting threshold can be based on an output threshold, a standard deviation determined by the privacy parameters, and a free parameter greater than or equal to 0. The adaptive reweighting engine 208 can determine that items with weights above the reweighting threshold should have their weight truncated while items with weights that are equal to or below the reweighting threshold can have their weight remain as is. For items with weights above the reweighting threshold, the adaptive reweighting engine 208 can truncate those weights and output the excess weight proportional to the amount contributed. This excess weight is then rerouted across the items of respective data sets. Once all items have weights equal to or below the reweighting threshold, the adaptive reweighting engine 208 can send the data sets and weights to the noise engine 210. The adaptive reweighting engine 208 can perform these steps once or iteratively.

[0074] The noise engine 210 can be configured to add noise to the weights of the items in the plurality of data sets. The amount of noise can be based on the privacy parameters. Example noise can be Gaussian noise and / or Laplace noise. The noise engine 210 can send the data sets with noisy weights to the thresholding engine 212.

[0075] The thresholding engine 212 can be configured to determine whether to include items from the data sets in the set union of the plurality of data sets. The thresholding engine 212 can compare the noisy weight of each item of each data set to an output threshold less than the reweighting threshold. The output threshold p can be p = — -)t). The thresholdingengine 212 can determine that items with noisy weights above the output threshold are to be included in the set union while items with noisy weights equal to or below the output threshold are to be excluded from the set union. The thresholding engine 212 can generate the set union by combining the items with noisy weights above the output threshold into a single data set. The thresholding engine 212 can output that data set as the private set union.

[0076] FIG. 3 depicts a block diagram of an example environment 300 for implementing a scalable private set union system 318 according to aspects of the disclosure. The scalable private set union system 318 can be implemented on one or more devices having one or more processors in one or more locations, such as in server computing device 302. A client computing device 304 and the server computing device 302 can be communicatively coupled to one or more storage devices 306 over a network 308. The storage devices 306 can be a combination of volatile and non-volatile memory and can be at the same or different physical locations than the computing devices 302, 304. For example, the storage devices 306 can include any type of non-transitory computer readable medium capable of storing information, such as a hard-drive, solid state drive, tape drive, optical storage, memory card, ROM, RAM, DVD, CD-ROM, write-capable, and readonly memories.

[0077] The server computing device 302 can include one or more processors 310 and memory 312. The memory 312 can store information accessible by the processors 310, including instructions 314 that can be executed by the processors 310. The memory 312 can also include data 316 that can be retrieved, manipulated, or stored by the processors 310. The memory 312 can be a type of transitory or non-transitory computer readable medium capable of storing information accessible by the processors 310, such as volatile and non-volatile memory. The processors 310 can include one or more central processing units (CPUs), graphic processing units (GPUs), field- programmable gate arrays (FPGAs), and / or application- specific integrated circuits (ASICs), such as tensor processing units (TPUs).

[0078] The instructions 314 can include one or more instructions that, when executed by the processors 310, cause the one or more processors 310 to perform actions defined by the instructions 314. The instructions 314 can be stored in object code format for direct processing by the processors 310, or in other formats including interpretable scripts or collections of independent source code modules that are interpreted on demand or compiled in advance. The instructions 314 can include instructions for implementing a scalable private set union system 318, which can correspond to the scalable private set union system 200 as depicted in FIG. 2. The scalable private set union system 318 can be executed using the processors 310, and / or using other processors remotely located from the server computing device 302.

[0079] The data 316 can be retrieved, stored, or modified by the processors 310 in accordancewith the instructions 314. The data 316 can be stored in computer registers, in a relational or nonrelational database as a table having a plurality of different fields and records, or as JSON, YAML, proto, or XML documents. The data 316 can also be formatted in a computer-readable format such as, but not limited to, binary values, ASCII, or Unicode. Moreover, the data 316 can include information sufficient to identify relevant information, such as numbers, descriptive text, proprietary codes, pointers, references to data stored in other memories, including other network locations, or information that is used by a function to calculate relevant data.

[0080] The client computing device 304 can also be configured similarly to the server computing device 302, with one or more processors 320, memory 322, instructions 324, and data 326. The client computing device 304 can also include a user input 328 and a user output 330. The user input 328 can include any appropriate mechanism or technique for receiving input from a user, such as keyboard, mouse, mechanical actuators, soft actuators, touchscreens, microphones, and sensors.

[0081] The server computing device 302 can be configured to transmit data to the client computing device 304, and the client computing device 304 can be configured to display at least a portion of the received data on a display implemented as part of the user output 330. The user output 330 can also be used for displaying an interface between the client computing device 304 and the server computing device 302. The user output 330 can alternatively or additionally include one or more speakers, transducers or other audio outputs, a haptic interface or other tactile feedback that provides non- visual and non-audible information to the platform user of the client computing device 304.

[0082] Although FIG. 3 illustrates the processors 310, 320 and the memories 312, 322 as being within the respective computing devices 302, 304, components described herein can include multiple processors and memories that can operate in different physical locations and not within the same computing device. For example, some of the instructions 314, 324 and the data 316, 326 can be stored on a removable SD card and others within a read-only computer chip. Some or all of the instructions 314, 324 and data 316, 326 can be stored in a location physically remote from, yet still accessible by, the processors 310, 320. Similarly, the processors 310, 320 can include a collection of processors that can perform concurrent and / or sequential operation. The computing devices 302, 304 can each include one or more internal clocks providing timing information, which can be used for time measurement for operations and programs run by the computing devices 302, 304.

[0083] The server computing device 302 can be connected over the network 308 to a data center 332 housing any number of hardware accelerators 334. The data center 332 can be one of multiple data centers or other facilities in which various types of computing devices, such as hardwareaccelerators, are located. Computing resources housed in the data center 332 can be specified for deploying models, such as for scalable private set union, as described herein.

[0084] The server computing device 302 can be configured to receive requests to process data from the client computing device 304 on computing resources in the data center 332. For example, the environment 300 can be part of a computing platform configured to provide a variety of services to users, through various user interfaces and / or application programming interfaces (APIs) exposing the platform services. As an example, the variety of services can include generating set unions for various applications incorporating privacy, such as privately outputting queries asked to a search engine, discovering words or phrases used in emails or text messages, analyzing data streams, outputting sparse histograms, and / or structured query language operations. Alternatively, or additionally, the various applications can include privacy preserving machine learning applications, such as vocabulary, token, and / or n-gram extraction, identifying relevant features or categories to learn embeddings, and / or sparsification of gradients in differential privacy standard gradient descent. The client computing device 304 can transmit input data as part of a query for a task involving generating a set union that maintains differential privacy. The scalable private set union system 318 can receive the input data, and in response, generate output data including a response to the query including the set union.

[0085] The server computing device 302 can maintain a variety of models in accordance with different constraints available at the data center 332. For example, the server computing device 302 can maintain different families for deploying models on various types of TPUs and / or GPUs housed in the data center 332 or otherwise available for processing.

[0086] FIG. 4 depicts a block diagram 400 illustrating one or more machine learning model architectures 402 according to aspects of the disclosure. More specifically, FIG. 4 depicts architectures 402A-N for deployment in a datacenter 404 housing a hardware accelerator 406 on which the deployed machine learning models 402 will execute, such as for the variety of services as described herein. The hardware accelerator 406 can be any type of processor, such as a CPU, GPU, FPGA, or ASIC such as a TPU.

[0087] An architecture 402 of a machine learning model can refer to characteristics defining the model, such as characteristics of layers for the model, how the layers process input, or how the layers interact with one another. The architecture 402 of the machine learning model can also define types of operations performed within each layer. One or more machine learning model architectures 402 can be generated that can output results involving private set unions. Example model architectures 402 can correspond to generative models, such as language models, foundation models, and / or graphical models.

[0088] The machine learning models can be trained according to a variety of different learningtechniques. Learning techniques for training the machine learning models can include supervised learning, unsupervised learning, semi-supervised learning, and reinforcement learning techniques. For example, training data can include multiple training examples that can be received as input by a model. The training examples can be labeled with a desired output for the model when processing the labeled training examples. The label and the model output can be evaluated through a loss function to determine an error, which can be back propagated through the model to update weights for the model. For example, a supervised learning technique can be applied to calculate an error between outputs, with a ground-truth label of a training example processed by the model. Any of a variety of loss or error functions appropriate for the type of the task the model is being trained for can be utilized, such as cross-entropy loss for classification tasks, or mean square error for regression tasks. The gradient of the error with respect to the different weights of the candidate model on candidate hardware can be calculated, for example using a backpropagation algorithm, and the weights for the model can be updated. The model can be trained until stopping criteria are met, such as a number of iterations for training, a maximum period of time, a convergence, or when a minimum accuracy threshold is met.

[0089] Referring back to FIG. 3, the devices 302, 304 and the data center 332 can be capable of direct and indirect communication over the network 308. For example, using a network socket, the client computing device 304 can connect to a service operating in the data center 332 through an Internet protocol. The devices 302, 304 can set up listening sockets that may accept an initiating connection for sending and receiving information. The network 308 can include various configurations and protocols including the Internet, World Wide Web, intranets, virtual private networks, wide area networks, local networks, and private networks using communication protocols proprietary to one or more companies. The network 308 can support a variety of short- and long-range connections. The short- and long-range connections may be made over different bandwidths, such as 2.402 GHz to 2.480 GHz, commonly associated with the Bluetooth® standard, 2.4 GHz and 5 GHz, commonly associated with the Wi-Fi® communication protocol; or with a variety of communication standards, such as the LTE® standard for wireless broadband communication. The network 308, in addition or alternatively, can also support wired connections between the devices 302, 304 and the data center 332, including over various types of Ethernet connection.

[0090] Although a single server computing device 302, client computing device 304, storage device 306, and data center 332 are shown in FIG. 5, it is understood that aspects of the disclosure can be implemented according to a variety of different configurations and quantities of computing devices, including in paradigms for sequential or parallel processing, or over a distributed network of multiple devices. In some implementations, aspects of the disclosure can be performed on asingle device connected to hardware accelerators configured for processing machine learning models, or any combination thereof.

[0091] FIG. 5 depicts a flow diagram of an example process 500 for generating a private set union according to aspects of the disclosure. For example, the private set union can be utilized in training a machine learning model. The example process 500 can be performed on a system of one or more processors in one or more locations, such as the scalable private set union system 200 as depicted in FIG. 2.

[0092] As shown in block 510, the scalable private set union system 200 receives a plurality of data sets, where respective data sets include a plurality of items. The scalable private set union system 200 also receives corresponding weights for respective items of the plurality of items of the plurality of data sets. The data sets can be associated with various privacy based applications, such as outputting queries asked to a search engine, discovering words or phrases used in messages, analyzing data streams, outputting sparse histograms, or structured query language operations. The data sets can also be associated with various privacy preserving machine learning applications, such as vocabulary, token, or n-gram extraction, identifying relevant features or categories to learn embeddings, or sparsification of gradients in differential privacy standard gradient descent. For example, the plurality of data sets can include sets of emails from respective users.

[0093] The scalable private set union system 200 can further receive one or more privacy parameters for setting a level of differential privacy, a maximum degree cap for determining a maximum number of items in a data set, a cardinality threshold for determining whether to participate in adaptive reweighting, an output threshold for determining whether an item is to be included in the private set union, and a free parameter for determining whether an item is to have its weight truncated.

[0094] The scalable private set union system 200 can further subsample respective data sets of the plurality of data sets based on the maximum degree cap such that the number of items in the respective data sets are not greater than the maximum degree cap.

[0095] The scalable private set union system 200 can further determine that a cardinality of some data sets of the plurality of data sets are above the cardinality threshold. Uniform weights can be contributed to the items in the data sets determined to be above the cardinality threshold. The cardinality threshold can be a number between 1 and the maximum degree cap.

[0096] As shown in block 520, the scalable private set union system 200 determines that the weights for some items of the pluralities of items of the plurality of data sets are above a reweighting threshold. The reweighting threshold can be based on the output threshold, a standard deviation determined by the one or more privacy parameters, and the free parameter greater thanor equal to 0. The reweighting threshold can be greater than the output threshold.

[0097] As shown in block 530, the scalable private set union system 200 truncates the weights for the items that are above the reweighting threshold. Excess weight that was truncated is to be rerouted to other items of the pluralities of items of the plurality of data sets that are not above the reweighting threshold. The scalable private set union system 200 can further output the excess weight for rerouting in proportion to the amounts contributed. The scalable private set union system 200 can further receive the excess weight rerouted to the other items of the pluralities of items of the plurality of data sets not above the reweighting threshold.

[0098] As shown in block 540, the scalable private set union system 200 adds noise to the weights for respective items of the pluralities of items of the plurality of data sets to generate noisy weights. The noise can include Gaussian and / or Laplace noise based on the one or more privacy parameters.

[0099] As shown in block 550, the scalable private set union system 200 determines that the noisy weights for some items of the pluralities of items of the plurality of data sets are above the output threshold.

[0100] As shown in block 560, the scalable private set union system 200 combines the items with noisy weights above the output threshold to form a data set corresponding to the private set union. For example, the data set corresponding to the private set union can include a data set of emails from respective users that preserves differential privacy for those users.

[0101] The scalable private set union system 200 can further output the data set corresponding to the private set union, such as for training a machine learning model. Alternatively, or additionally, the scalable private set union system 200 can train the machine learning model using the data set corresponding to the private set union. For example, the scalable private set union system 200 can train the machine learning model to generate email drafts in response to prompts to generate the email drafts using the data set of emails from respective users that preserves differential privacy for those users.

[0102] FIG. 6 depicts a flow diagram of an example process 600 for training a machine learning model using data corresponding to a private set union. The machine learning model can be a generative model, such as a large language model, large foundation model, and / or a large graphical model. The machine learning model can be a general usage model or a model fine-tuned for a particular task, such as email drafting. The example process 600 can be performed on a system of one or more processors in one or more locations, such as the scalable private set union system 200 as depicted in FIG. 2.

[0103] As shown in block 610, the scalable private set union system 200 receives a plurality of data sets for training the machine learning model. Block 610 can correspond to block 510 of the process 500 for generating a private set union. The data sets can be associated with various privacypreserving machine learning applications, such as vocabulary, token, or n-gram extraction, identifying relevant features or categories to learn embeddings, or sparsification of gradients in differential privacy standard gradient descent. For example, the plurality of data sets can include sets of emails from respective users.

[0104] As shown in block 620, the scalable private set union system 200 generates a data set corresponding to a private set union based on the plurality of data sets. Block 620 can correspond to the blocks depicted in the process 500 for generating the private set union, including determining that the weights for some items of each of the plurality of items are above a reweighting threshold, truncating the weights for the items that are above the reweighting threshold, adding noise to the weights for each item of each plurality of items to generate noisy weights, determining that the noisy weights for some items of each of the plurality of items are above an output threshold, and combining the items with noisy weights above the output threshold to form a data set corresponding to the private set union.

[0105] As shown in block 630, the scalable private set union system 200 trains the machine learning model using the data set corresponding to the private set union. Example training techniques include supervised learning, unsupervised learning, semi- supervised learning, and reinforcement learning techniques.

[0106] Aspects of this disclosure can be implemented in digital electronic circuitry, in tangibly embodied computer software or firmware, and / or in computer hardware, such as the structure disclosed herein, their structural equivalents, or combinations thereof. Aspects of this disclosure can further be implemented as one or more computer programs, such as one or more modules of computer program instructions encoded on a tangible non-transitory computer storage medium for execution by, or to control the operation of, one or more data processing apparatus. The computer storage medium can be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or combinations thereof. The computer program instructions can be encoded on an artificially generated propagated signal, such as a machine-generated electrical, optical, or electromagnetic signal, which is generated to encode information for transmission to suitable receiver apparatus for execution by a data processing apparatus.

[0107] The term “configured” is used herein in connection with systems and computer program components. For a system of one or more computers to be configured to perform particular operations or actions means that the system has installed thereon software, firmware, hardware, or a combination thereof that cause the system to perform the operations or actions. For one or more computer programs to be configured to perform particular operations or actions means that the one or more programs include instructions that, when executed by one or more data processingapparatus, cause the apparatus to perform the operations or actions.

[0108] The term “data processing apparatus” or “data processing system” refers to data processing hardware and encompasses various apparatus, devices, and machines for processing data, including programmable processors, computers, or combinations thereof. The data processing apparatus can include special purpose logic circuitry, such as a field programmable gate array (FPGA) or an application specific integrated circuit (ASIC). The data processing apparatus can include code that creates an execution environment for computer programs, such as code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or combinations thereof.

[0109] The term “computer program” refers to a program, software, a software application, an app, a module, a software module, a script, or code. The computer program can be written in any form of programming language, including compiled, interpreted, declarative, or procedural languages, or combinations thereof. The computer program can be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. The computer program can correspond to a file in a file system and can be stored in a portion of a file that holds other programs or data, such as one or more scripts stored in a markup language document, in a single file dedicated to the program in question, or in multiple coordinated files, such as files that store one or more modules, sub programs, or portions of code. The computer program can be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and interconnected by a data communication network.

[0110] The term “database” refers to any collection of data. The data can be unstructured or structured in any manner. The data can be stored on one or more storage devices in one or more locations. For example, an index database can include multiple collections of data, each of which may be organized and accessed differently.

[0111] The term “engine” refers to a software-based system, subsystem, or process that is programmed to perform one or more specific functions. The engine can be implemented as one or more software modules or components or can be installed on one or more computers in one or more locations. A particular engine can have one or more computers dedicated thereto, or multiple engines can be installed and running on the same computer or computers.

[0112] The processes and logic flows described herein can be performed by one or more computers executing one or more computer programs to perform functions by operating on input data and generating output data. The processes and logic flows can also be performed by special purpose logic circuitry, or by a combination of special purpose logic circuitry and one or more computers.

[0113] A computer or special purpose logic circuitry executing the one or more computer programs can include a central processing unit, including general or special purpose microprocessors, for performing or executing instructions and one or more memory devices for storing the instructions and data. The central processing unit can receive instructions and data from the one or more memory devices, such as read only memory, random access memory, or combinations thereof, and can perform or execute the instructions. The computer or special purpose logic circuitry can also include, or be operatively coupled to, one or more storage devices for storing data, such as magnetic, magneto optical disks, or optical disks, for receiving data from or transferring data to. The computer or special purpose logic circuitry can be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a Global Positioning System (GPS), or a portable storage device, e.g., a universal serial bus (USB) flash drive, as examples.

[0114] Computer readable media suitable for storing the one or more computer programs can include any form of volatile or non-volatile memory, media, or memory devices. Examples include semiconductor memory devices, e.g., EPROM, EEPROM, or flash memory devices, magnetic disks, e.g., internal hard disks or removable disks, magneto optical disks, CD-ROM disks, DVD-ROM disks, or combinations thereof.

[0115] Aspects of the disclosure can be implemented in a computing system that includes a back end component, e.g., as a data server, a middleware component, e.g., an application server, or a front end component, e.g., a client computer having a graphical user interface, a web browser, or an app, or any combination thereof. The components of the system can be interconnected by any form or medium of digital data communication, such as a communication network. Examples of communication networks include a local area network (LAN) and a wide area network (WAN), e.g., the Internet.

[0116] The computing system can include clients and servers. A client and server can be remote from each other and interact through a communication network. The relationship of client and server arises by virtue of the computer programs running on the respective computers and having a client-server relationship to each other. For example, a server can transmit data, e.g., an HTML page, to a client device, e.g., for purposes of displaying data to and receiving user input from a user interacting with the client device. Data generated at the client device, e.g., a result of the user interaction, can be received at the server from the client device.

[0117] Unless otherwise stated, the foregoing alternative examples are not mutually exclusive, but may be implemented in various combinations to achieve unique advantages. As these and other variations and combinations of the features discussed above can be utilized without departing from the subject matter defined by the claims, the foregoing description of theembodiments should be taken by way of illustration rather than by way of limitation of the subject matter defined by the claims. In addition, the provision of the examples described herein, as well as clauses phrased as "such as," "including" and the like, should not be interpreted as limiting the subject matter of the claims to the specific examples; rather, the examples are intended to illustrate only one of many possible embodiments. Further, the same reference numbers in different drawings can identify the same or similar elements.

Claims

CLAIMS1. A method for generating a private set union comprising: receiving, by one or more processors, a plurality of data sets, each data set of the plurality of data sets comprising a plurality of items, and corresponding weights for each item of each of the plurality of items; determining, by the one or more processors, that the weights for some items of the plurality of items are above a reweighting threshold; truncating, by the one or more processors, the weights for the items that are above the reweighting threshold, wherein excess weight that was truncated is to be rerouted to other items of the plurality of items that are not above the reweighting threshold; adding, by the one or more processors, noise to the weights for each item of the plurality of items to generate noisy weights; determining, by the one or more processors, that the noisy weights for some items of the plurality of items are above an output threshold; and combining, by the one or more processors, the items with noisy weights above the output threshold to form a data set corresponding to the private set union.

2. The method of claim 1, further comprising subsampling, by the one or more processors, each data set of the plurality of data sets based on a maximum degree cap.

3. The method of claim 1, further comprising determining, by the one or more processors, that a cardinality of some data sets of the plurality of data sets are above a cardinality threshold, wherein uniform weights are contributed to the items in the data sets determined to be above the cardinality threshold.

4. The method of claim 3, wherein the cardinality threshold is a number between 1 and a maximum degree cap.

5. The method of claim 1, wherein the reweighting threshold is based on the output threshold, a standard deviation value determined by one or more privacy parameters, and a free parameter greater than or equal to 0.

6. The method of claim 1, wherein the reweighting threshold is greater than the output threshold.

7. The method of claim 1, further comprising outputting, by the one or more processors, the excessweight for rerouting in proportion to amounts contributed.

8. The method of claim 1, further comprising receiving, by the one or more processors, the excess weight rerouted to the other items of each of the plurality of items not above the reweighting threshold.

9. The method of claim 1, wherein the noise comprises at least one of Gaussian or Laplace noise based on one or more privacy parameters.

10. The method of claim 1, further comprising receiving, by the one or more processors, one or more privacy parameters for setting a level of differential privacy, a maximum degree cap for determining a maximum number of items in a data set, a cardinality threshold for determining whether to participate in adaptive reweighting, the output threshold for determining whether an item is to be included in the private set union, and a free parameter for determining whether an item is to have its weight truncated.

11. The method of claim 1, wherein the data sets are associated with at least one of outputting queries asked to a search engine, discovering words or phrases used in messages, analyzing data streams, outputting sparse histograms, or structured query language operations.

12. The method of claim 1, wherein the data sets are associated with privacy preserving machine learning applications.

13. The method of claim 12, wherein the privacy preserving machine learning applications comprise at least one of vocabulary, token, or n-gram extraction, identifying relevant features or categories to learn embeddings, or sparsification of gradients in differential privacy standard gradient descent.

14. The method of claim 1, further comprising training, by the one or more processors, a machine learning model using the data set corresponding to the private set union.

15. The method of claim 14, wherein the plurality of data sets comprises sets of emails from respective users and the data set corresponding to the private set union comprises a data set of emails from the respective users that preserves differential privacy.

16. The method of claim 15, wherein training the machine learning model further comprises training the machine learning model to generate email drafts in response to prompts to generate the email drafts.

17. A system comprising: one or more processors; and one or more storage devices coupled to the one or more processors and storing instructions that, when executed by the one or more processors, cause the one or more processors to perform operations for generating a private set union, the operations comprising: receiving a plurality of data sets, each data set of the plurality of data sets comprising a plurality of items, and corresponding weights for each item of each of the plurality of items; determining that the weights for some items of the plurality of items are above a reweighting threshold; truncating the weights for the items that are above the reweighting threshold, wherein excess weight that was truncated is to be rerouted to other items of the plurality of items that are not above the reweighting threshold; adding noise to the weights for each item of the plurality of items to generate noisy weights; determining that the noisy weights for some items of the plurality of items are above an output threshold; and combining the items with noisy weights above the output threshold to form a data set corresponding to the private set union.

18. The system of claim 17, wherein the operations further comprise training a machine learning model using the data set corresponding to the private set union.

19. The system of claim 18, wherein: the plurality of data sets comprises sets of emails from respective users and the data set corresponding to the private set union comprises a data set of emails from the respective users that preserves differential privacy; and training the machine learning model further comprises training the machine learning model to generate email drafts in response to prompts to generate the email drafts.

20. A non-transitory computer readable medium storing instructions that, when executed by oneor more processors, causes the one or more processors to perform operations for generating a private set union, the operations comprising: receiving a plurality of data sets, each data set of the plurality of data sets comprising a plurality of items, and corresponding weights for each item of each of the plurality of items; determining that the weights for some items of the plurality of items are above a reweighting threshold; truncating the weights for the items that are above the reweighting threshold, wherein excess weight that was truncated is to be rerouted to other items of the plurality of items that are not above the reweighting threshold; adding noise to the weights for each item of the plurality of items to generate noisy weights; determining that the noisy weights for some items of the plurality of items are above an output threshold; and combining the items with noisy weights above the output threshold to form a data set corresponding to the private set union.

Images