Group-by-group distributed privacy budget

Abstract

One or more servers implement a method of managing privacy budget. The method includes receiving a plurality of data sets, each of the data sets including respective encrypted data and respective metadata; classifying the plurality of data sets into one or more groups of data sets based on the respective metadata; and performing a query to determine whether there is a sufficient privacy budget for a group included in the one or more groups to store a result of analyzing the group, wherein the privacy budget for the group corresponds to a number of times the group can be analyzed.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to and the benefit of the filing date of provisional U.S. patent application No. 63 / 478,140, ​​entitled “Distributed Privacy Budget Service,” filed on December 31, 2022. The entire contents of the provisional application are hereby expressly incorporated herein by reference. Technical Field

[0003] The present disclosure relates to implementing privacy budgets, and more particularly, to techniques for distributing privacy budget monitoring to improve security and enhance differential privacy. Background Art

[0004] The background description provided herein is for the purpose of generally presenting the context of the present disclosure. The work of the presently named inventors (to the extent that it is described in this background section) and aspects of this description that may not be identified as prior art at the time of filing are neither explicitly nor implicitly admitted to be prior art to the present disclosure.

[0005] Increasingly, data is collected about user behavior and user interactions with electronic resources, which may contain personally identifiable information (PII). Certain services or applications may analyze such data to generate reports and metrics. For example, a service may aggregate records containing data from multiple users and analyze the aggregated records to generate a general report. Even if the data is edited to remove PII, this type of analysis may still be susceptible to security risks.

[0006] For example, if a service that analyzes user input data to produce output does not maintain differential privacy, a malicious actor can retrieve the user input data based on the output. Differential privacy refers to a mathematically provable guarantee that an individual user's information is protected. A set of analysis results can be considered differentially private if it is impossible to trace the analysis results (i.e., output data) back to a single user's data (i.e., input data).

[0007] There is a class of differential privacy attacks that craft batches of records, where each new batch is generated by adding or removing a small number of records, and sends these batches to a service for processing. By knowing some metadata about the records (e.g., the Internet Protocol (IP) address from which the records originated) and understanding the aggregation algorithm, the reports generated by these crafted batches can be analyzed to reveal the private information contained in some records. This class of differential privacy attacks relies on reusing the same records over and over to generate multiple insights that can then be compared and analyzed.

[0008] Therefore, there is a need for improved techniques for analyzing records in a manner that enforces differential privacy on the outputs. Summary of the invention

[0009] An example embodiment of these techniques is a method for managing a privacy budget in one or more servers. The method may include: receiving a request to analyze a data set, the data set being associated with a privacy budget, the privacy budget indicating a number of times the data set can be analyzed; sending a first request to a first server implementing a first privacy budget service to verify whether there is sufficient privacy budget to analyze the data set, the first privacy budget service maintaining a first instance of a privacy budget associated with the data set; sending a second request to a second server implementing a second privacy budget service to verify whether there is sufficient privacy budget to analyze the data set, the second privacy budget service being independent of the first privacy budget service and the second privacy budget service maintaining a second instance of a privacy budget associated with the data set; receiving a first response from the first server indicating whether there is sufficient privacy budget according to the first privacy budget service; receiving a second response from the second server indicating whether there is sufficient privacy budget according to the second privacy budget service; and processing the data set based on the first response and the second response.

[0010] Another example embodiment is a method for managing a privacy budget in one or more servers. The method may include receiving a plurality of data sets, each of the plurality of data sets including encrypted data and metadata of the data set; classifying the plurality of data sets into one or more groups based on corresponding metadata included in the plurality of data sets; and for one of the one or more groups, querying whether the group has enough privacy budget to store analysis results of the group, the privacy budget of the group indicating the number of times the group can be analyzed.

[0011] A further example embodiment is a computing system comprising one or more servers and a non-transitory computer-readable medium storing instructions thereon that, when executed by the one or more servers, cause the computing system to implement any of the above methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 is a block diagram of an example computing system in which the techniques of this disclosure may be implemented;

[0013] Figure 2A is a block diagram illustrating an example computing architecture including a security control plane and a data plane of the present disclosure;

[0014] Figure 2B is shown with Figure 2A A block diagram of another example computing architecture is similar, except that: Figure 2BAdditional infrastructure for managing cryptographic keys and privacy budgets is shown;

[0015] Figure 3 It shows that FIG. 2A to FIG. 2B A message passing diagram for an example scenario in which a control plane receives a request to perform a computation using logic executed on a data plane;

[0016] Figure 4 is a block diagram illustrating an example technique for grouping records and allocating privacy budgets by group;

[0017] Figure 5 is a message passing diagram illustrating an example scenario of grouping records, allocating privacy budgets by group, and verifying the privacy budgets through a distributed privacy budget service;

[0018] Figure 6 is a flow chart illustrating an example method for managing privacy budgets by group; and

[0019] Figure 7 is a flow chart illustrating an example method for managing a privacy budget using a distributed privacy budget service. DETAILED DESCRIPTION

[0020] The service of the present disclosure implements techniques for mitigating differential privacy attacks. According to an example technique, the service prevents records from being reused for multiple batches. The service accomplishes this task by monitoring used records and allocating a budget (called a privacy budget) for each of these records. Each time the record is used in a processed batch, the service decrements the privacy budget. For example, if the privacy budget of record N is '2', record N can be used in a batch that is only processed twice. After a record is used in a batch and processed by the service, the service decrements the budget for the record by 1. The service will then refuse to process any batches containing record N.

[0021] In some implementations, the service further optimizes the technique by tracking privacy budgets by batch rather than by record. For example, a service may process trillions of events per day. Therefore, tracking privacy budgets by record may be inefficient. To reduce the computational resources required to monitor the privacy budget for a set of records, the service may divide a set of records into "buckets" or groups and assign a privacy budget to each group. For example, such groups may be defined by time range and sender. An example group may include "all records sent by sender X from 9 a.m. to 10 a.m." The service may then assign a privacy budget to the example group and decrement the budget each time the example group is processed, including decrementing the budget each time a batch of records including the group is processed. This optimization efficiently processes large numbers of records with minimal overhead while maintaining differential privacy.

[0022] The second example technique is to distribute the privacy budget operation. A service that analyzes records may call an external privacy budget service operated by a trusted party to verify that there is a privacy budget available to process a record or a set of records. However, if a single trusted party operates the privacy budget service, the trusted party has the opportunity to tamper with the privacy budget. To address this potential vulnerability, trust can be expanded by distributing the privacy budget service among more than one operator. Instead of a single trusted party, there may be N independent co-owners of the privacy budget (i.e., multiple trusted parties). Each trusted party may implement an instance of the privacy budget service, where each of these instances has a copy of the entire privacy budget. In order to consume the budget, the service that analyzes the record may be required to perform atomic transactions in all of these instances, where "atomic" means that all instances must verify the conditions of the privacy budget, otherwise the process cannot proceed. The transaction can only proceed if the privacy budgets in all of these instances are consistent. Therefore, if a trusted party tampers with its privacy budget, the transaction will fail. Therefore, the distributed privacy budget service reduces the likelihood that the privacy budget will be tampered with because all trusted parties that implement instances of the privacy budget must collude to modify the privacy budget.

[0023] Sample computing environment

[0024] In some embodiments, the techniques discussed above may be implemented using a security control plane (SCP), which in turn provides an isolated secure execution environment for the data plane (DP). Any arbitrary business logic (e.g., any logic for analyzing batch records) may be executed within the DP, and all sensitive data traversing the SCP and entering the DP is encrypted. It should be understood that while the examples provided in this disclosure primarily describe techniques for managing privacy budgets as performed using an SCP architecture, any suitable computing environment may be used to implement the techniques described herein.

[0025] The SCP described herein provides an unobservable secure execution environment in which services can be deployed. Specifically, any business logic (e.g., application code) that provides a service can be executed within the secure execution environment to provide the security and privacy guarantees required by the workflow, while no party can observe the runtime computations. The state of the environment is opaque even to the administrator of the service, and the service can be deployed on any supported cloud.

[0026] As an example, two clients generating data (i.e., Client 1 and Client 2), may wish to combine the data streams they receive from their respective customers so that the clients can generate quantitative metrics related to those customers that cannot be derived from their separate data sets. As a more specific example, for example, Client 1 may be a retailer having data indicative of customer transactions, and Client 2 may be an analytics engine capable of measuring the effectiveness of an advertising campaign for products offered by the retailer.

[0027] Client 2 may offer a service with an algorithm that Client 2 claims will securely perform data analysis. However, Client 1 may not wish to expose its customer data to Client 2 in a manner that would potentially allow the data to be leaked or used in a manner that does not comply with Client 1's privacy and security guarantees. Therefore, Client 1 wants to ensure that (1) its customer data cannot be leaked by Client 2 or any other party, and (2) the logic used to analyze the customer data complies with Client 2's security requirements. The technology disclosed herein provides a secure execution environment in which business logic executes, such that sensitive data analyzed by the business logic remains encrypted everywhere except within the secure execution environment, and provides proof so that any party can ensure that logic running within the secure execution environment executed as guaranteed.

[0028] Generally speaking, services that perform computations (i.e., use business logic to process events or requests) are split between a data plane (DP) and a security control plane (SCP). Computation-specific business logic is hosted within a DP, which is located within a trusted execution environment (TEE), also referred to herein as an enclave. Business logic can be provided to the DP as a container, wherein a container is a software package containing all necessary elements for running business logic in any environment. For example, the container can be provided to the SCP by the business logic owner. Functionally, the SCP provides a secure execution environment and facilities to deploy and operate the DP on a large scale, including managing cryptographic keys, buffering requests, tracking privacy budgets, accessing storage, and orchestrating policy-based horizontal automatic expansion. The SCP execution environment isolates the DP from the details of the cloud environment, allowing the service to be deployed on any supported cloud provider without making changes to the DP. Both DP and SCP communicate through an input / output (I / O) application programming interface (API) (also referred to herein as a control plane I / O API or CPIO API) to work together.

[0029] In one example implementation, all data traversing the SCP is always encrypted, and only the DP has access to the decryption keys. For example, the SCP can facilitate trusted data exchange, where data from multiple parties that may not trust each other can be combined, but none of these parties can access the keys used to decrypt the data. Further, when outside the DP, the decryption key can be bit-split so that only the DP can assemble the decryption key within the TEE. Depending on the desired application, the output from the DP can be edited or aggregated in a way that it can be shared and the data of any individual user cannot be identified or leaked.

[0030] SCP provides several privacy, trust, and security guarantees. With respect to privacy, services using SCP can provide assurances that no stakeholder (e.g., devices operated by the client, cloud platform, third party) can act alone to access or leak plaintext (i.e., non-encrypted) sensitive information, including administrators of SCP deployments. Further, with respect to trust, DP runs in a trusted state in a secure execution environment when the enclave is booted. For example, SCP can be implemented using techniques to ensure process isolation in hardware, including memory encryption and / or memory address space segmentation and a trust chain from boot, using a trusted platform module (TPM) or virtual trusted platform module (vTPM) according to secure boot standards, and / or using a trusted and / or certified operating system (OS). Starting with an audited code base and reproducible builds, cryptographic proofs are used to prove the DP binary identity and provenance at runtime to a key management service (KMS), which is configured to release cryptographic keys only to verified enclaves. Therefore, any tampering with the DP image will result in the system being unable to decrypt any data. Cloud providers are absolutely trustworthy given the strong incentive they have to guarantee their Terms of Service (ToS) guarantees. Regarding security, the secure execution environment is unobservable. The secure execution environment's memory is encrypted or otherwise hardware-protected to prevent access from other processes. In an example implementation, core dumps are not possible. All data is encrypted in transit and at rest, and all I / O to / from the DP is encrypted. No one has access to the private key in plaintext (e.g., the KMS is locked, the key is split, and the key is only available within the DP located within the secure execution environment.

[0031] SCP distributes trust in a way that three stakeholders need to cooperate in order to leak plaintext user event data. SCP also uses a distributed trust model to ensure that two stakeholders need to cooperate in order to tamper with the privacy budget service (see Figure 2B and Figure 5Described in more detail). Distributed trust is used for both event decryption and privacy budget services. With respect to event decryption, the private key required to decrypt events received at the SCP is generated in a secure environment and bit-split between at least two KMSs, each of which is under the control of an independent trusted party. For example, each trusted party can further encrypt its corresponding key split using a KMS key owned by the trusted party in the cloud provider's KMS. The KMS is configured to release key material only to DPs that match a specific hash. If the DP is tampered with, the key split will not be released. In this case, the service can be started but no events can be decrypted. Similarly, the privacy budget service can be distributed between two independent trusted parties, and transaction semantics can be used to ensure that the budgets of the two trusted parties match, so that budget tampering can be detected.

[0032] If you will refer to Figure 2B As discussed, SCP also provides a mechanism for proving that any business logic running on DP corresponds to publicly released code, allowing other parties to verify the business logic used to analyze sensitive data. In general, the complete code base of the business logic is available for inspection and audit by all stakeholders. The build is reproducible, and any stakeholder can build the DP container. Building the DP container generates a set of cryptographic hashes (e.g., platform configuration registers (PCRs)). Therefore, all parties can verify whether the deployed product matches the released code base by comparing the hashes. The trusted party publishes the hash to the parties requesting verification of the build logic. For example, the KMS is configured to release key material only to images that match the hash generated from the logic published by the build. This ensures that the private key used to decrypt sensitive information is only available to images corresponding to specific submissions to specific repositories.

[0033] Turning to an example computing system in which the SCP of the present disclosure may be implemented, Figure 1 An example computing system 100 is shown. Computing system 100 includes a client computing device 102 (also referred to herein as client device 102) coupled to a cloud platform 122 (also referred to herein as cloud 122) via a network 120. Network 120 may typically include one or more wired and / or wireless communication links, and may include, for example, a wide area network (WAN) (such as the Internet), a local area network (LAN), a cellular telephone network, or another suitable type of network or combination of networks. Although examples of the present disclosure are primarily directed to cloud-implemented architectures, it should be understood that the techniques disclosed herein, including techniques for providing a secure execution environment in which sensitive data is processed, techniques for generating, partitioning, and distributing cryptographic keys, techniques for managing privacy budgets, and techniques for providing mechanisms for authenticating proprietary business logic, may also be applied in non-cloud systems.

[0034] For example, the client device 102 may be a portable device such as a smart phone or a tablet computer. The client device 102 may also be a laptop computer, a desktop computer, a personal digital assistant (PDA), a wearable device such as smart glasses, or other suitable computing devices. The client device 102 may include a memory 106, one or more processors (CPUs) 104, a network interface 114, a user interface 116, and an input / output (I / O) interface 118. The client device 102 may also include Figure 1 Components not shown in the diagram, such as a graphics processing unit (GPU). The client device 102 may be associated with a service user, which is an end user of the services provided by the SCP, as described below. The end user operates a client device 102 (or more specifically, a browser or application on the client device 102), which sends a request / event to the service. In order to send a request or event to the service, the client device 102 encrypts the request / event using a public key, which the client device 102 can retrieve from a public key repository (e.g., a public key repository server 178). The client device 102 is exemplary only. As described below, the cloud platform 122 can receive incoming events and / or requests from the client device 102, from a browser / application / client process executing on the client device 102, or from another computing device that issues a request on behalf of the client device 102 or forwards a request from the client device 102. Further, although Figure 1 Only one client device is shown in FIG. 1 , but computing system 100 may include multiple client devices capable of communicating with cloud platform 122 .

[0035] The network interface 114 may include one or more communication interfaces, such as hardware, software, and / or firmware, for enabling communication via a cellular network, a WiFi network, or any other suitable network, such as the network 120. The user interface 116 may be configured to provide information to a user, such as a response to a request / event received from the cloud platform 122. The I / O interface 118 may include various I / O components (e.g., ports, capacitive or resistive touch-sensitive input panels, keys, buttons, lights, LEDs). For example, the I / O interface 118 may be a touch screen.

[0036] The memory 106 may be a non-temporary memory and may include one or more suitable memory modules, such as a random access memory (RAM), a read-only memory (ROM), a flash memory, other types of persistent memory, etc. The memory 106 may store machine-readable instructions that may be executed on one or more processors 104 and / or special processing units of the client device 102. The memory 106 also stores an operating system (OS) 110, which may be any suitable mobile or general-purpose OS. In addition, the memory 106 may store one or more applications that transmit data with the cloud platform 122 via the network 120. Transmitting data may include sending data, receiving data, or both. For example, the memory 106 may store instructions for implementing a browser, an online service, or an application that requests data from an application implemented on a DP of a secure execution environment on the cloud platform 122 / sends data (i.e., business logic) to the application, as described below.

[0037] The cloud platform 122 may include multiple servers associated with a cloud provider to provide cloud services via the network 120. The cloud provider is the owner of the cloud platform 122 on which the SCP 126 is deployed. Figure 1 Only one cloud platform is shown in FIG. 1 , but SCP 126 may be deployed on multiple cloud platforms, even if the cloud platforms are operated by different cloud providers. The servers providing cloud platform 122 may be distributed across multiple sites to improve reliability and reduce latency. Individual servers or groups of servers within cloud platform 122 may communicate with client devices 102 and with each other via network 120. Example servers that may be included in cloud platform 122 are discussed in further detail below. Although not specifically described herein, Figure 1 106. Each server in the cloud platform 122 is shown, but each server included in the cloud platform 122 may include one or more processors similar to processor 104, which are suitable and configured to execute various software stored in one or more memories similar to memory 106. The server may further include a database, which may be a local database stored in the memory of a particular server or a network database stored in a network-connected memory (e.g., in a storage area network). The server may also include a network interface and an I / O interface similar to interfaces 114 and 118, respectively. Further, it should be understood that although certain components are described as separate servers, in general, the term "server" may refer to one or more servers. In addition, although functions are generally described as being performed by separate servers, some functions described herein may be performed by the same server.

[0038] The cloud platform 122 includes an SCP 126, which includes a TEE 124. TEE 124 is a secure execution environment in which DP 128 is isolated. A TEE (such as TEE 124) is an environment that provides execution isolation and provides a higher level of security than a conventional system. TEE 124 can use hardware to implement isolation (known as confidential computing). The cloud provider is considered to be the root of trust for SCP 126 and complies with the terms of service (ToS) agreement of the cloud platform 122. The hardware manufacturer of the server that provides TEE 124 also has a ToS guarantee and therefore also provides an additional layer of trust. SCP 126 also uses technology to ensure that the state at startup is secure, including using a minimized OS image recommended by the cloud provider, and using a secure boot sequence based on TPM / vTPM to enter the OS image.

[0039] One or more servers of the cloud platform 122 perform control plane (CP) functions (i.e., support SCP 126), and one or more servers perform data plane (DP) functions. For example, CP functions including key management and privacy budget services can be distributed across multiple trusted parties. All functions of DP 128 are performed by processes within TEE 124. Depending on the implementation, each DP server may have more than one TEE. TEE 124 may be deployed and operated by an administrator. The administrator may audit the logic to be implemented on DP128 and verify against the hash of the binary image to deploy logic 142. On the CP, there may be a front-end server or process 134 that receives external request / event indications (e.g., from the client device 102), buffers requests / events until they can be processed by DP 128, and forwards the received request to DP 128. Generally speaking, as used herein, unless otherwise specified, a request may also refer to an event or record, or may include one or more events or records. Unless otherwise specified, the terms "record", "request" and "event" may be used interchangeably herein. The request may include encrypted data (e.g., encrypted data representing an interaction of the client device 102 with a website, application, or other online resource) and metadata for the request. The metadata may include a timestamp indicating the date and / or time the request was generated. The metadata may also include other information depending on the nature of the data, such as an indication of the source of the request or an indication of where the results of data processing are to be output. For example, if the request is generated based on an interaction of the client device 102 with an advertiser's online advertisement placed on an online resource published by a publisher, the metadata may include the advertiser's domain and / or the publisher's domain, or other indications of the advertiser and / or publisher.

[0040] In some implementations, there is a third-party server 136 between the client device 102 and the SCP 126. The third-party server 136 (which may include one or more servers and may or may not be hosted on the cloud platform 122) may be responsible for receiving the request (encrypted by the client device 102) from the client device 102 and then dispatching the encrypted request to the SCP 126. In some cases, the third party is the administrator of the service. The third-party server 136 does not have the key for decrypting the request. The third-party server 136 may, for example, aggregate the requests into batches and store the batches (e.g., on the cloud storage 160). The third-party server 136 or the cloud storage server 160 may notify the front-end server 134 that the request is ready to be processed, and / or the front-end server 134 may subscribe to notifications that are pushed to the front-end server 134 when a batch is added to the cloud storage 160.

[0041] DP 128 includes a server (which may include one or more servers) including one or more processors 138 (similar to processor 104) and one or more memories 140 (similar to memory 106). Memory 140 includes business logic 142 (also referred to as logic 142), which can be executed by processor 138. Business logic 142 is used to implement any application or service deployed on TEE 124. Memory 140 can also store key cache 146, which stores cryptographic keys for encrypting and decrypting communications. Further, memory 140 includes CPIO API 144, which includes a function library for communicating with other elements of cloud platform 122 (including components on CP of SCP 126). CPIOAPI 144 can be configured to interface with any cloud platform provided by a cloud provider. For example, in a first deployment, SCP 126 can be deployed to a first cloud platform provided by a first cloud provider. DP 128 hosts specific business logic 142, and CPIO API 144 facilitates communication between logic 142 and the first cloud platform. In a second deployment, SCP 126 may be deployed to a second cloud platform provided by a second cloud provider. DP 128 may host the same business logic 142 as the first deployment, and CPIO API 144 is configured to facilitate communication between logic 142 and the second cloud platform. Thus, SCP 126 may be deployed to a different cloud platform without editing the underlying business logic 142, and only CPIO API 144 is configured to interface with a specific cloud platform.

[0042] There may be additional CP-level services provided by servers of the cloud platform 122 that support the SCP 126. For example, the validator server 148 may provide a validator module that can verify whether the business logic 142 complies with the security policy. Figure 1 124, but the verifier module may operate within the TEE 124. As another example, the privacy budget service server 152 may implement a privacy budget service that verifies whether a privacy budget of a user or device has been exhausted. Figure 2B As discussed, one or more privacy budget services may additionally or alternatively be implemented by a trusted party.

[0043] In addition, the cloud platform 122 may include other servers and databases that communicate with the SCP 126, as described in the following paragraphs. These servers may facilitate the CP functions of the SCP 126. Specifically, the CP functions may be distributed across several servers, as will be discussed below. However, the processes of the DP 128 remain within the TEE 124 and are not distributed outside the TEE 124.

[0044] As described above, cloud storage 160 may store encrypted batches of requests before the encrypted batches are received by front-end server 134. Cloud storage 160 may also be used to store responses after DP 128 has processed received requests, or to perform storage functions for other components of cloud platform 122. Queue 162 may be used by front-end server 134 to store pending requests before they can be analyzed by DP 128. For example, after receiving a request from client device 102, front-end server 134 may receive the request and temporarily store the pending request in queue 162 until DP 128 is ready to process the request. As another example, after receiving notification from third-party server 136 that a batch of requests is stored in cloud storage 160, front-end 134 may retrieve the batch of requests and place the batch of requests in queue 162, where the batch of requests awaits analysis by DP 128.

[0045] KMS service 164 provides a KMS that generates, deletes, distributes, replaces, rotates, and otherwise manages cryptographic keys. The functions of KMS 164 may be performed by one or more servers. Thus, KMS 164 may be a cloud KMS. Trusted party 1 server 166 and trusted party 2 server 172 are servers associated with trusted party 1 and trusted party 2, respectively, that provide the functions of each trusted party. Although Figure 1 Only two trusted parties are shown, but the cloud platform 122 may include multiple trusted parties. Each trusted party may manage instances of a privacy budget (such as reference Figure 2B and Figure 5), and the logic 142 implemented on DP128 can also be audited to verify the built product against the hash of the published logic. The trusted party owns the creation and management of asymmetric keys used to encrypt and decrypt user data. The trusted party can securely generate keys and publish public keys to the world. The private key can be bit-split into two parts (one split under the control of each trusted party, but any number N of splits can also be supported, such as when there are N trusted parties). Envelope encryption technology can be used, where each trusted party encrypts each of its key splits with the KMS's symmetric key and stores the encrypted splits in its repository. Envelope encryption allows rotation of envelopes without rotating the keys within the envelopes. Public keys can be stored and managed by the public key repository server 178. In addition or alternatively, the KMS server 164 can manage the public keys.

[0046] The computing system 100 may also include a public security policy storage 180 that may be located on or off the cloud platform 122. The public security policy storage 180 stores security policies so that the security policies are accessible to the public (e.g., by the client device 102, by components of the cloud platform 122). A security policy (also referred to herein as a policy) describes what actions or fields are allowed for the output of a composition service. A policy may also be described as a machine-readable and machine-implementable privacy design document (PDD).

[0047] Next reference Figure 2A , example architecture 200A illustrates connections between components and software elements of computing system 100. Client device 102 may retrieve a public key (e.g., from public key repository server 178) in order to address a request to a service being implemented on DP 128 (i.e., by business logic 142). For example, client device 102 may initiate a request to access content provided by a service, or may emit an event including user behavior data.

[0048] The encrypted request from the client device 102 is first received by the front-end module 234 (i.e., a module implemented by the front-end server 134) of the SCP 126. In some implementations, the request is first received by a third party that batches the request before notifying the front-end 234 (or causing the front-end 234 to be notified). The notification to the front-end 234 may include the location in the cloud storage 160 where the encrypted request resides (e.g., the location of the cloud storage bucket), and may include an indication of where the output of the DP 128 should be output (e.g., by including metadata indicating such information). In such a case, the front-end 234 may retrieve the encrypted request from the cloud storage 160. In any case, the front-end 234 passes the encrypted request to the DP 128 using the functions defined by the CPIO API 144. The front-end 234 may store the encrypted request in the queue 162 until the DP 128 is ready to process the request and retrieve the request from the queue 162. The DP 128 decrypts the request and processes the request according to the business logic 142. Decrypting the request may include communicating with KMS 164 (e.g., a cloud KMS implemented by a distributed server) to retrieve and assemble a private key for decrypting the request, and / or communicating with a trusted party, such as Figure 2B These are examples of cloud native services integrated with SCP 126, but the concept can be extended to other cloud infrastructure and services, where SCP 126 mediates between these services and business logic 142 by using CPIO API 144 to transform semantics so that business logic 142 is agnostic to a specific cloud environment.

[0049] Processing the request may include communicating with a privacy budget service 252 (e.g., implemented by a privacy budget service server 152) using the CPIO API 144 functionality to check the privacy budget and ensure compliance with the privacy budget. The privacy budget tracks processed requests and events. For example, there may be a maximum number of requests originating from a particular user that may be processed during a particular computation or time period. Ensuring compliance with the privacy budget prevents parties analyzing the output from the DP 128 from extracting information about the particular user. By checking whether the privacy budget is being complied with, the DP 128 provides a differentially private output. As will be described with reference to Figures 4 to 5 As discussed, a privacy budget can be allocated to a group of requests so that the group of requests can only be processed a maximum number of times.

[0050] The results from processing these requests may be encrypted by DP 128 and may be edited and / or aggregated so that the output does not reveal information about a particular user. DP 128 may store the results, for example, in cloud storage 160, where the results may be retrieved by parties having a decryption key for the results. As an example, if the results are processed for a third-party server 136, DP 128 may encrypt the results using a key that the third-party server 136 may decrypt.

[0051] Go to Figure 2B , architecture 200B is similar to architecture 200A, except that additional details regarding key management and privacy budget are shown. Figure 2A compared to, Figure 2B Also shown are trusted party 1 server 166 (referred to herein as trusted party 1 166 for simplicity), trusted party 2 server 172 (referred to herein as trusted party 2 172 for simplicity), and public key distribution service 278. Public key distribution service 278 provides public keys to client devices 102, which client devices 102 can use to address requests to DP 128, front end 234, or third-party servers 136 that aggregate requests ( Figure 2B 164). The public key distribution service 278 may be operated by the public key repository server 178 or by the KMS server 164. Trusted party 1 166 includes a key cache 268 containing an encrypted split 1 key (i.e., an encrypted first portion of a private key), and trusted party 2 172 includes a key cache 274 containing an encrypted split 2 key (i.e., an encrypted second portion of a private key). Each of the trusted parties 166, 172 may also provide a privacy budget service 270, 276, and may each manage an instance of a privacy budget. Distributing management of the privacy budget to the two trusted parties helps ensure that no one trusted party can tamper with the privacy budget. Both privacy budget services 270, 276 should implement the same privacy budget; therefore, if the two services return different outputs, the SCP 126 can identify that one of the trusted parties 166, 172 has tampered with the privacy budget. Figure 2B The architecture shown prevents any one trusted party from having full control over the private decryption keys or the privacy budget. A single trusted party cannot act alone to provide an unlimited budget to any user, and therefore a single trusted party cannot repeatedly aggregate the same batch of data.

[0052] The elements shown in architecture 200B can be implemented Figure 3 and Figure 5 The actions shown in .

[0053] Example scenario including split key

[0054] exist Figure 3During the illustrated example scenario 300, the client device 102 retrieves 302 a public key from a public key distribution service 278 (which may be implemented by a public key repository server 178 or may be provided by a KMS 164). The client device 102 encrypts 304 a request for a service implemented on a TEE 124 using the public key associated with the service. The client device 102 then sends 306 the encrypted request to a front-end server 134. As previously described, in some implementations, a third-party server 136 receives the encrypted request before the request reaches the front-end server 134 and stores the encrypted request in a cloud storage 160, where the encrypted request may be retrieved by the front-end server 134. The front-end server 134 passes 308 the encrypted request to the DP 128. The business logic 142 retrieves the encrypted request for processing and attempts to retrieve a decryption key from the key cache 146. If the business logic 142 retrieves the decryption key, the scenario continues from event 308 to event 320.

[0055] As applicable to Figure 3 and Figure 5 In a more specific example of the illustrated scenario, the service implemented on the TEE 124 may be an aggregation service. In this case, the encrypted request may include events detected by an API implemented on the client device 102. Each event may be a conversion, where a conversion may correspond to an interaction of the client device 102 with an advertisement provided by a publisher. The aggregation service may be a service configured to provide an aggregate report that provides aggregate statistics of events from client software (e.g., a browser, an application, or a process implemented by the client device 102). Thus, in this example, the business logic 142 may include performing event aggregation and outputting an output report.

[0056] Return to reference Figure 3 , if the decryption key is not present in the key cache 146, the business logic 142 uses the CPIO API 144 to request the decryption key. More specifically, the DP 128 sends 310 a request for key split 1 to the trusted party 1 166 and sends 312 a request for key split 2 to the trusted party 2 172. In response, the trusted party 1 166 sends 314 key split 1 to the DP 128, and the trusted party 2 172 sends 316 key split 2 to the DP 128. When the key splits 1 and 2 are received 314, 316 by the DP 128, the key splits 1 and 2 may be encrypted using the symmetric keys generated by the trusted party 1 166 and the trusted party 2 172, respectively. Therefore, the DP 128 may also need to request the key splits to be decrypted by, for example, the KMS 164 (i.e., the cloud KMS), which may store the symmetric keys operated by the trusted parties 166, 172.

[0057] Before sending the key split 314 and 316 to DP 128, the trusted parties 166, 172 can first verify that the business logic 142 corresponds to the code publicly released when the code repository is submitted. This can be achieved through proof. The code base of the business logic 142 is available for inspection and audit by all stakeholders (client devices 102, trusted parties 166, 172, cloud platforms 122, administrators, third parties, etc.). As described above, any stakeholder can build a DP container including the business logic 142 and generate PCRs for the published logic. Therefore, any party can verify that the business logic 142 built and deployed on DP 128 matches the published code base by comparing the PCR of the deployed business logic 142 with the PCR of the published code base. The CPIO API 144 can transmit the PCR of the deployed business logic 142 to other parties (i.e., client devices 102, other components of the cloud platform 122, or computing systems 100) to prove that the deployed business logic 142 corresponds to the published code base and has not been changed.

[0058] Thus, in the request 310, 312 sent by the DP 128, the business logic 142 may include the PCRs of the deployed business logic 142 using the CPIO API 144. Alternatively or in addition, the trusted party 166, 172 may request the PCRs. The trusted party 166, 172 may then confirm that the deployed image matches the PCRs of the published code base. After performing this verification process, the trusted party 166, 172 may release 314, 316 the key split to the DP 128. Likewise, the KMS 164 may also verify that the binary image deployed on the DP 128 is certified and only release symmetric decryption keys to certified DP binaries running in the TEE 124.

[0059] The business logic 142 then assembles 318 the decrypted private key from the key splits and may store the assembled private key in the key cache 146. The assembly of the private key occurs only in the TEE 124, and the CPIO API 144 uses secure channels and authentication when communicating with the trusted parties 166, 172. Since each trusted party 166, 172 contains only a portion of each key, the entire private key is present within the secure TEE 124 only after the key splits are combined. Thus, this prevents either party from leaking plaintext data or the private key. The business logic 142 retrieves 320 the private key from the key cache 146 and uses the private key to decrypt the request. Before processing the request, the business logic 142 may verify 322 that there is a privacy budget available to process the request. The business logic 142 may perform such verification by communicating with the privacy budget service 270, 272, and / or 154 according to the CPIO API 144. Verifying that sufficient privacy budget is available to process the request may include sending the request to trusted party 1 166 and trusted party 2 172, each of which manages an instance of a privacy budget, as will be referenced in Figure 5 for discussion.

[0060] If there is sufficient privacy budget, the business logic 142 may process 324 the request. Before storing 326 the processing result, the business logic 142 may in some cases check again whether the privacy budget is still available. Depending on the implementation, the privacy budget may be checked before processing the request, after processing the request and before storing the processing result, or before processing the request and before storing the processing result (e.g., Figure 3 ). Business logic 142 may then encrypt and store 326 the result for later retrieval. For encryption, business logic 142 may perform static encryption using a system-provided or customer-managed encryption key (CMEK). The result may then be retrieved and used by any party (e.g., an administrator, client device 102, third-party server 136, etc.) that possesses the CMEK key.

[0061] Example scenarios including privacy budgets and distributed trust by group

[0062] Go to Figure 4 , a block diagram illustrating an example technique for grouping records and allocating privacy budgets by group. TEE 124 may receive several data sets 404A-404E, either individually or in one or more batches. TEE 124 may receive data sets (e.g., Figure 3), or receiving a data set from a third-party server 136 via the front end 134. For example, each data set may include the encrypted request discussed above with reference to event 306. Therefore, each data set may include encrypted data (e.g., representing the interaction of the client device 102 with the online resource) and metadata associated with the encrypted data. Metadata may also be included in the encrypted data. As used herein, a "data set" may correspond to the "request", "report", and "event" discussed above. Further, as described above, the metadata may be a timestamp indicating the date and / or time when the data set was generated. The metadata may also include other information depending on the nature of the data, such as an indication of the source of the data set or an indication of where the results of processing the data are output. For example, if the request is generated based on the interaction of the client device 102 with an advertiser's online advertisement placed on an online resource published by a publisher, the metadata may include the advertiser's domain and / or the publisher's domain, or other indications to the advertiser and / or the publisher. The metadata or a portion of the metadata can be included in plain text (i.e., unencrypted) form.

[0063] In order to reduce the computing resources required to monitor the privacy budget of each data set 404A-E, the TEE 124 can classify the data sets 404A-404E into groups and allocate the privacy budget to each group instead of to each individual data set 404A-E. The metric used to classify the data sets 404A-E may vary depending on the implementation. In general, the classification can be based on the metadata included in each data set 404A-E. In one example, the classification can be based on the timestamp included in the metadata. The example timestamp may indicate both time and data, such as 2021-01-01 12:29:33AM. In order to group the data sets, the timestamp can be rounded down to the hour (e.g., rounding the example timestamp 2021-01-01 12:29:33AM down to 2021-01-01 12:00:00AM). As another example, the classification can be based on the publisher domain and / or advertiser domain included in the metadata. In a further example, the classification may be based on both timestamp and publisher domain and / or advertiser domain. In this example, each group may include data sets from the same advertiser domain that are included within the same rounded-down hour time window.

[0064] exist Figure 4In the example of , five data sets 404A-404E are classified into two groups, namely groups 408A and 408B, based on metadata included in each of the data sets 404A-E. For example, if the data sets 404A-E are classified based on timestamps and advertiser domains, then data sets 404A-404C may have been generated from the same advertiser domain during the same time window, and data sets 404D and 404E may have been generated from the same advertiser domain during subsequent time windows. A first privacy budget 411A is then assigned to group 408A including data sets 404A-404C, and a second privacy budget 411B is assigned to group 408B including data sets 404D and 404E. The first privacy budget 411A and the second privacy budget 411B may be equal in size. As will be seen in reference Figure 5 As further discussed, when group 408A is analyzed or included in a processed batch, privacy budget 411A is decremented by a privacy budget service, which may be a distributed privacy budget service. Similarly, when group 408B is analyzed, privacy budget 411B is decremented. When group 408A is analyzed (e.g., processed using business logic 142, such as in Figure 3 Before the analysis of group 408A is stored, TEE 124 can verify whether there is sufficient privacy budget to analyze and / or store the analysis of group 408A. This verification can be performed on a group basis rather than on a record basis, thereby reducing the computational resources required to maintain differential privacy for multiple records.

[0065] In order to track the privacy budget of each group, the TEE 124 can generate a privacy budget key for each group. The privacy budget key can be generated based on information included in the metadata, such as by hashing the information included in the metadata. More specifically, the privacy budget key can be generated based on information used to classify the data set into groups so that a unique privacy budget key is assigned to each group. For example, an example privacy budget key can be generated by hashing the advertiser domain, the publisher domain, or both the advertiser domain and the publisher domain. The example privacy budget key can also be generated based on a rounded-down time window corresponding to the group (e.g., an hourly window on a particular day), or the privacy budget of the group can be tracked using a privacy budget key and an indication of the rounded-down time window. In some implementations, the TEE 124 generates a privacy budget key for a group. In other implementations, the privacy budget key can be generated by the creator of the data set and included in the data set. For example, if a data set is generated based on the client device 102 interacting with a publisher, when the publisher generates the data set for transmission to the TEE 124 for analysis, the publisher can hash information such as the advertiser domain and / or the publisher domain to generate a privacy budget key and include the privacy budget key in the data set. When the TEE 124 queries a privacy budget service (e.g., trusted party 1 166 and trusted party 2 172) to determine whether there is sufficient privacy budget to process the data set, the TEE 124 can include the privacy budget key in the request and, if not indicated by the privacy budget key itself, the time window.

[0066] Go to Figure 5 , scenario 500 depicts two techniques of the present disclosure: allocating privacy budgets by group, and verifying privacy budgets using a distributed privacy budget service. It should be understood that these techniques can be implemented independently of each other. For example, in one implementation, TEE 124 can allocate privacy budgets by group, but rely only on a single trusted party to verify the privacy budget. In another implementation, TEE 124 can allocate privacy budgets by record, but rely on multiple trusted parties to verify the privacy budget. In a further implementation, such as Figure 4 As shown, TEE 124 can either allocate privacy budgets by group or rely on multiple trusted parties to verify the privacy budgets.

[0067] although Figure 5 300 (i.e., by including events similar to events 302, 304, 306, and 308). DP 128 may receive multiple encrypted requests 508A-508N (i.e., data sets, such as Figure 4DP 128 may then analyze metadata included in each of the encrypted requests to classify the encrypted requests into groups (e.g., similar to groups 408A, 408B). DP 128 allocates 530 a privacy budget to each group. The privacy budgets of different groups may be the same, so that each group has the same initial privacy budget. The privacy budget of each group may be a number N, where N may be an integer equal to 1 or greater. If the privacy budget is equal to N = 1, the group can only be analyzed once before the privacy budget of the group is completely consumed.

[0068] Although Figure 5 Although not shown in FIG. 5 , scene 500 may also include events 310, 312, 314, 316, 318, and 320, such as Figure 3 shown.

[0069] Next, DP 128 verifies 522 that there is sufficient privacy budget to handle the group. Each of the trusted parties 166, 172 may implement a privacy budget service that maintains a corresponding instance of the privacy budget for the group. The first instance of the privacy budget maintained by trusted party 1 166 should be the same as the second instance of the privacy budget maintained by trusted party 2 172, because each of trusted parties 166, 172 should implement the same privacy budget for a given group. Therefore, trusted party 1 166 and trusted party 2 172 can be referred to as implementing a distributed privacy budget service. Each of the trusted parties 166, 172 is independent of each other (i.e., implemented on independent servers). Further, it is not required that the trusted parties 166, 172 be implemented on the same cloud platform; each trusted party may be implemented on a different cloud platform.

[0070] Verifying 522 the privacy budget may include sending 532 to trusted party 1 166 to verify that there is sufficient privacy budget to process the group's request according to trusted party 1 166, and sending 534 to trusted party 2 172 to verify that there is sufficient privacy budget to process the group's request according to trusted party 2 172. Figure 5166, 172 as occurring sequentially (for clarity), but DP 128 may send 532, 434 the first request and the second request simultaneously. Each request may include a privacy budget key (and a time window rounded down if not indicated by a privacy budget key) so that each trusted party can track which group the request belongs to. Each of trusted parties 166, 172 may then check whether there is enough privacy budget to process the group (i.e., check whether the privacy budget is at least 1) based on its corresponding instance of the privacy budget. If there is enough privacy budget according to trusted party 1 166, then trusted party 1 166 decrements its instance of the privacy budget. Similarly, if there is enough privacy budget according to trusted party 2 172, then trusted party 2 172 decrements its instance of the privacy budget. Trusted party 1 166 sends 536 a response to DP 128 indicating whether there is sufficient privacy budget according to trusted party 1 166 , and trusted party 2 172 sends 538 a response to DP 128 indicating whether there is sufficient privacy budget according to trusted party 2 172 .

[0071] Based on the two responses, DP 128 verifies 540 whether there is sufficient privacy budget to process the group. The two responses must match and indicate that there is sufficient privacy budget for the process to proceed. If the responses do not match, or one or both of the trusted parties 166, 172 indicate that there is insufficient privacy budget to process the group, the process is aborted and the group is not analyzed (i.e., the process does not continue to event 524 or 526). Verifying 522 that there is sufficient privacy budget to process the group can include events 532, 534, 536, 538, and 540. DP 128 can communicate 532, 534, 536, 538 with the trusted parties 166, 172 according to the CPIO API 144.

[0072] In scenario 500, after verifying 522 that there is an available privacy budget to process the group, DP 128 may then process 524 the group (e.g., similar to event 324, by analyzing the group using business logic 142). DP 128 may analyze the group within a large batch of requests, but enforces that each request included in the group is analyzed within the same group and, if applicable, within the same batch. Then, similar to event 326, DP 128 may encrypt and store 526 the results of the analysis at event 524 for subsequent retrieval. In scenario 500, DP 128 verifies 522 the privacy budget before analyzing 524 the group, but in other scenarios, DP 128 may analyze 524 the group and verify 522 the privacy budget before storing 526 the results. If there is not enough privacy budget, DP 128 discards the results of the analysis at event 524 and does not store 526 the results.

[0073] DP 128 and trusted parties 166 and 172 may perform Figure 5 524 to ensure that the consumption of the privacy budget of the group occurs atomically, i.e., the privacy budget of all instances is either consumed or not consumed. For example, after verifying 522 that there is available privacy budget to process the group, DP 128 may send a commit message to each of the trusted parties 166, 172 (e.g., simultaneously) to cause each of the trusted parties 166, 172 to simultaneously decrement its corresponding instance of the privacy budget. The trusted parties 166, 172 may then send a notification to DP 128 informing each trusted party 166, 172 that the privacy budget instance has been successfully consumed. Then, after confirming that all instances of the privacy budget have been successfully consumed, DP 128 may continue to process 524 the group or store 526 the results of any analysis. If such confirmation is not received from any trusted party 166, 172, the process may be aborted.

[0074] Example approach to privacy budget and distributed trust by group

[0075] Figure 6 6 is a flow chart illustrating an example method 600 for managing privacy budgets by group. The method 600 may be implemented by one or more servers (e.g., servers supporting SCP 126). The method 600 may begin at block 602, where the server receives a plurality of data sets (e.g., events 508A-508N). Each data set may include encrypted data and metadata for the data set. For example, the encrypted data may represent an interaction of a user with an online resource (e.g., an advertisement or other interactive element of a browser or application) via a client device 102.

[0076] At block 604, the server classifies the plurality of data sets into one or more groups (e.g., event 528) based on metadata included in each data set. For each group, the server may assign a privacy budget representing the number of times the group may be analyzed, such that the privacy budget for the one or more groups is performed on a per-group basis rather than on a per-request basis. The privacy budget for each group may be the same initial privacy budget. Classifying the data sets may include classifying based on a timestamp included in the metadata of the data sets. For example, the timestamp may be rounded down to the hour so that each group corresponds to data received or generated during a particular hour window on a date.

[0077] At block 606, the server queries, for one of the one or more groups, whether the group has sufficient privacy budget to store the analysis results of the group. In some implementations, the server analyzes the data sets included in the group to produce an output (e.g., event 524) before the query. If, based on the query, the group has sufficient privacy budget, the server stores the output (e.g., event 526). If the group does not have sufficient privacy budget, the server avoids storing the output and instead discards the output. In other implementations, the server queries whether there is sufficient privacy budget before analyzing the data sets in the group. If there is sufficient privacy budget, the server analyzes the data sets included in the group and stores the output of the analysis. If there is not sufficient privacy budget, the server avoids analyzing the data sets in the group. While or after analyzing the data sets, the server decrements the privacy budget of the group, or decrements the privacy budget by notifying a privacy budget service (e.g., trusted party 1 166 and / or trusted party 2 172) to decrement the privacy budget. In some implementations, the privacy budget service decrements the privacy budget in response to receiving a request to verify whether the group has an available privacy budget.

[0078] Querying whether a group has sufficient privacy budget may include sending (e.g., via an API call) a request to consume the privacy budget of the group to a privacy budget service (e.g., to trusted party 1 166, trusted party 2 172, or to both trusted parties 166, 172 in a scenario involving a distributed privacy budget service) to consume the privacy budget of the group (e.g., event 322, event 522, 532, 534). The server may then receive a response from the privacy budget service indicating whether there is sufficient privacy budget to consume the privacy budget of the group. In a scenario involving a distributed privacy budget service, the query may include sending a first request to consume the privacy budget of the group to a first privacy budget service (e.g., trusted party 1 166), sending a second request to consume the privacy budget of the group to a second privacy budget service (e.g., trusted party 2 172), and receiving a first response and a second response from the first privacy budget service and the second privacy budget service, respectively (e.g., events 536, 538). The server may determine whether there is sufficient privacy budget based on the two responses. The two responses must match and indicate that there is sufficient privacy budget for the server to continue analyzing or storing the results of any analysis, of the datasets included in the set.

[0079] Further, the query may include sending a request to the privacy budget service, wherein the request includes a privacy budget key representing the group. The privacy budget key may be generated by the server based on metadata common to the group (e.g., advertiser domain, publisher domain, timestamp, depending on how the data sets are categorized into groups), such as by hashing at least a portion of the metadata. For example, the key may be generated based on hashing an indication in metadata of a domain from which a set of data sets was received (e.g., a publisher domain that published an advertisement and / or an advertiser domain of the advertisement).

[0080] Figure 7 7 is a flow chart illustrating an example method 700 for managing a privacy budget using a distributed privacy budget service. The method 700 may be implemented by one or more servers (e.g., servers supporting SCP 126). The method 700 may begin at block 702, where a server receives a request to analyze a data set (e.g., events 308, 508A-508N). The data set may include encrypted data and metadata for the data set. For example, the encrypted data may represent an interaction of a user with an online resource (e.g., an advertisement or other interactive element of a browser or application) via a client device 102. Depending on the implementation, the server may receive multiple data sets, and the request may be to analyze multiple data sets. The data set may be associated with a privacy budget representing the number of times the data set may be analyzed.

[0081] At block 704, the server sends (e.g., via an API call) a first request to verify whether there is sufficient privacy budget to analyze the data set (e.g., event 532) to a first server (e.g., trusted party 1 166) that implements a first privacy budget service. The first privacy budget service maintains a first instance of a privacy budget associated with the data set. Similarly, at block 706, the server sends a second request to verify whether there is sufficient privacy budget to analyze the data set (e.g., event 534) to a second server (e.g., trusted party 2 172) that implements a second privacy budget service. The second privacy budget service is independent of the first privacy budget service (i.e., implemented on a server independent of the server that implements the first privacy budget service) and maintains a second instance of a privacy budget associated with the data set.

[0082] At block 708, the server receives a first response from the first server according to the first privacy budget service indicating whether there is sufficient privacy budget (e.g., event 536). Similarly, at block 710, the server receives a second response from the second server according to the second privacy budget service indicating whether there is sufficient privacy budget (e.g., event 538). Based on the first response and the second response, at block 712, the server processes the data set. If both responses indicate that there is sufficient privacy budget, the processing at block 712 includes analyzing the data set (e.g., event 524) and storing the results of the analysis (e.g., event 526). In some implementations, the server may first analyze the data set and, if both responses indicate that there is sufficient privacy budget, store the results of the analysis. If there is sufficient privacy budget to continue the analysis and store the results of the analysis, the server causes the first privacy budget service and the second privacy budget service to each decrement their respective instances of the privacy budget (e.g., as described with reference to block 606). The first privacy budget service and the second privacy budget service decrement their budgets atomically (i.e., either both services decrement their respective instances of the privacy budget because there is sufficient privacy budget to consume, or neither service decrements their respective instances of the privacy budget). If one or both responses indicate that there is not sufficient privacy budget, the server refrains from analyzing the data set, or, if the data set was analyzed before determining that there is not sufficient privacy budget, refrains from storing the results of the analysis.

[0083] In some implementations, method 700 can be combined with aspects of method 600 such that the data set can be part of a group of data sets to which a privacy budget is allocated. In such implementations, the first request and the second request to the privacy budget service are requests to verify the privacy budget of the group.

[0084] Example

[0085] The following example list reflects various embodiments explicitly contemplated by the present disclosure. Those skilled in the art will readily appreciate that the following examples neither limit the embodiments disclosed herein nor exhaustively describe all embodiments that can be conceived from the above disclosure, but are exemplary in nature.

[0086] Example 1. A method for managing a privacy budget in one or more servers, the method comprising: receiving a request to analyze a data set, the data set being associated with a privacy budget, the privacy budget indicating the number of times the data set can be analyzed; sending a first request to a first server implementing a first privacy budget service to verify whether there is sufficient privacy budget to analyze the data set, the first privacy budget service maintaining a first instance of the privacy budget associated with the data set; sending a second request to a second server implementing a second privacy budget service to verify whether there is sufficient privacy budget to analyze the data set, the second privacy budget service being independent of the first privacy budget service and the second privacy budget service maintaining a second instance of the privacy budget associated with the data set; receiving a first response from the first server indicating whether there is sufficient privacy budget according to the first privacy budget service; receiving a second response from the second server indicating whether there is sufficient privacy budget according to the second privacy budget service; and processing the data set based on the first response and the second response.

[0087] Example 2. The method of Example 1, wherein processing includes: determining that both the first response and the second response indicate sufficient privacy budget; and in response to the determination, analyzing the data set and storing results of the analysis.

[0088] Example 3. The method of Example 1, wherein processing comprises: analyzing the data set; determining that both the first response and the second response indicate that there is sufficient privacy budget; and responsive to the determination, storing a result of the analysis.

[0089] Example 4. The method of Example 2 or 3, further comprising causing the first privacy budget service and the second privacy budget service to atomically decrement the first instance of the privacy budget and the second instance of the privacy budget.

[0090] Example 5. The method of example 1, wherein processing comprises: determining that the first response indicates sufficient privacy budget and the second response indicates insufficient privacy budget; and responsive to the determination, refraining from analyzing the data set.

[0091] Example 6. The method of example 1, wherein processing comprises: analyzing the data set; determining that the first response indicates sufficient privacy budget and the second response indicates insufficient privacy budget; and responsive to the determination, avoiding storing results of the analysis.

[0092] Example 7. The method of example 1, wherein processing includes: determining that both the first response and the second response indicate an insufficient privacy budget; and responsive to the determination, refraining from analyzing the data set.

[0093] Example 8. The method of Example 1, wherein processing includes: analyzing the data set; determining that both the first response and the second response indicate that there is insufficient privacy budget; and responsive to the determination, avoiding storing results of the analysis.

[0094] Example 9. A method as described in any of the preceding examples, wherein: the data set is included in a set of data sets, the privacy budget is defined for the set of data sets, and receiving the request to process the data set includes receiving the request to process the set of data sets.

[0095] Example 10. The method of any of the preceding examples, wherein sending the first request comprises sending the first request to a first privacy budget service via an application programming interface (API) call.

[0096] Example 11. A method for managing a privacy budget in one or more servers, the method comprising: receiving multiple data sets, each of the multiple data sets including encrypted data and metadata of the data set; classifying the multiple data sets into one or more groups based on the corresponding multiple metadata included in the multiple data sets; and for one of the one or more groups, querying whether the group has sufficient privacy budget to store analysis results of the group, the privacy budget of the group representing the number of times the group can be analyzed.

[0097] Example 12. The method of Example 11 further comprises: before querying whether the group has sufficient privacy budget, analyzing the data set included in the group to generate an output; and in response to determining that the group has sufficient privacy budget based on the query, storing the output.

[0098] Example 13. The method of Example 11 further comprises: before querying whether the group has sufficient privacy budget, analyzing the data set included in the group to generate an output; and in response to determining that the group does not have sufficient privacy budget based on the query, avoiding storing the output.

[0099] Example 14. The method as described in Example 11 further includes: before analyzing the data sets included in the group, querying whether the group has sufficient privacy budget; and in response to determining that the group has sufficient privacy budget based on the query: analyzing the data sets included in the group to generate an output, and storing the output.

[0100] Example 15. The method of Example 11 further comprises: before analyzing the data sets included in the group, querying whether the group has sufficient privacy budget; and in response to determining that the group does not have sufficient privacy budget based on the query, avoiding analyzing the data sets included in the group.

[0101] Example 16. The method of Example 12 or 14, further comprising: decrementing the privacy budget of the group.

[0102] Example 17. The method of any of Examples 11-16, wherein: for each data set, the metadata of the data set includes a timestamp; and classifying the plurality of data sets into one or more groups includes classifying the plurality of data sets based on a respective plurality of timestamps corresponding to the plurality of data sets.

[0103] Example 18. The method of Example 17, wherein classifying the plurality of data sets based on the corresponding plurality of timestamps comprises classifying the plurality of data sets into a plurality of groups such that each of the one or more groups corresponds to an hour window of a date.

[0104] Example 19. A method as described in any of Examples 11-18, wherein each group in the one or more groups has the same initial privacy budget.

[0105] Example 20. A method as described in any of Examples 11-19, wherein querying whether the group has sufficient privacy budget includes: sending a request to a privacy budget service to consume the privacy budget of the group; and receiving a response from the privacy budget service indicating whether there is sufficient privacy budget to consume the privacy budget of the group.

[0106] Example 21. A method as described in Example 20, wherein the privacy budget service is a first privacy budget service, the request is a first request, and the response is a first response, and wherein the query further includes: sending a second request to a second privacy budget service to consume the privacy budget of the group; receiving a second response from the second privacy budget service, the second response indicating that there is sufficient privacy budget to consume the privacy budget of the group; and determining whether there is sufficient privacy budget based on whether both the first response and the second response indicate that there is sufficient privacy budget.

[0107] Example 22. A method as described in any of Examples 11-21, wherein the group includes a set of data sets, the method further comprising: generating a key representing the group using metadata included in the set of data sets, wherein the query includes sending a request including the key.

[0108] Example 23. The method of Example 22, wherein generating the key comprises: generating the key by applying a hash operation to at least a portion of metadata included in a set of data sets.

[0109] Example 24. The method of Example 23, wherein at least a portion of the metadata included in the set of data sets indicates a domain from which the set of data sets was received.

[0110] Example 25. The method of any of Examples 11-24, wherein for a dataset in the plurality of datasets, the encrypted data represents an interaction of a user with an online resource.

[0111] Example 26. A computing system for managing a privacy budget, the computing system comprising: one or more servers; and a non-transitory computer-readable medium storing instructions thereon, the instructions, when executed by one or more processors, causing the computing system to implement a method according to any of the preceding examples.

[0112] Additional considerations

[0113] The following additional considerations apply to the foregoing discussion.

[0114] The client device (e.g., client device 102) in which the techniques of the present disclosure may be implemented may be any suitable device capable of wireless communication, such as a smartphone, a tablet computer, a laptop computer, a desktop computer, a mobile game console, a point of sale (POS) terminal, a health monitoring device, a drone, a camera, a media streaming dongle or another personal media device, a wearable device such as a smart watch, a wireless hotspot, a femtocell, or a broadband router. Further, in some cases, the client device may be embedded in an electronic system such as a head unit of a vehicle or an advanced driver assistance system (ADAS). Still further, the client device may operate as an Internet of Things (IoT) device or a mobile Internet device (MID). Depending on the type, the client device may include one or more general purpose processors, a computer readable memory, a user interface, one or more network interfaces, one or more sensors, and the like.

[0115] Certain embodiments are described in the present disclosure as including logic or multiple components or modules. A module may be a software module (e.g., code stored on a non-transitory machine-readable medium) or a hardware module. A hardware module is a tangible unit that is capable of performing certain operations and may be configured or arranged in a particular manner. A hardware module may include a dedicated circuit system or logic that is permanently configured to perform certain operations (e.g., as a dedicated processor, such as a field programmable gate array (FPGA) or an application-specific integrated circuit (ASIC)). A hardware module may also include programmable logic or circuit systems that are temporarily configured by software to perform certain operations (e.g., as contained in a general-purpose processor or other programmable processor). The decision to implement a hardware module with a dedicated and permanently configured circuit system or with a temporarily configured circuit system (e.g., configured by software) may be driven by cost and time considerations.

[0116] When implemented in software, the techniques may be provided as part of an operating system, as a library used by multiple applications, as a specific software application, etc. The software may be executed by one or more general-purpose processors or by one or more special-purpose processors.

Claims

1. A method for managing a privacy budget in one or more servers, the method comprising: receiving a plurality of data sets, each of the data sets comprising respective encrypted data and respective metadata; classifying the plurality of data sets into one or more groups of data sets based on the corresponding metadata; as well as A query is performed to determine whether there is sufficient privacy budget for a group included in the one or more groups to store results of analyzing the group, wherein the privacy budget for the group corresponds to a number of times the group can be analyzed.

2. The method of claim 1, further comprising: prior to said execution of said query, analyzing a data set included in said set to produce an output; as well as In response to determining that the group has a sufficient privacy budget, storing the output.

3. The method of claim 1, further comprising: prior to said execution of said query, analyzing a data set included in said set to produce an output; as well as In response to determining that the group does not have sufficient privacy budget, storing the output is avoided.

4. The method of claim 1, wherein: The executing of the query occurs prior to analyzing the data set included in the group; the method further comprising: In response to determining that the group has sufficient privacy budget: analyzing said data sets included in said set to produce an output, and The output is stored.

5. The method of claim 1, wherein: The executing of the query occurs prior to analyzing the data set included in the group; the method further comprising: In response to determining that the group does not have sufficient privacy budget, analyzing the data set included in the group is avoided.

6. The method of claim 3 or 4, further comprising: Decrement the privacy budget of the group.

7. A method as claimed in any one of the preceding claims, wherein: For each data set, the metadata for the data set includes a timestamp; and Classifying the plurality of data sets into the one or more groups includes classifying the plurality of data sets based on a respective plurality of time stamps corresponding to the plurality of data sets.

8. The method of claim 7, wherein classifying the plurality of data sets based on the corresponding plurality of timestamps comprises classifying the plurality of data sets into a plurality of groups such that each of the one or more groups corresponds to an hour window of a date.

9. The method of any of the preceding claims, wherein each of the one or more groups has the same initial privacy budget.

10. The method of any one of the preceding claims, wherein said performing of said query comprises: sending a request to a privacy budget service to consume the privacy budget of the group; as well as A response is received from the privacy budget service, the response indicating whether there is sufficient privacy budget to consume the privacy budget of the group.

11. The method of claim 10, wherein: The privacy budget service is a first privacy budget service, The request is a first request, and the response is a first response, and The generation of the query further comprises: sending a second request to a second privacy budget service to consume the privacy budget of the group; receiving a second response from the second privacy budget service, the second response indicating that there is sufficient privacy budget to consume the privacy budget of the group; and Whether there is sufficient privacy budget is determined based on whether both the first response and the second response indicate sufficient privacy budget.

12. A method as claimed in any one of the preceding claims, wherein: The set includes a set of data sets; The method further comprises: A key representing the set of data sets is generated using metadata included in the set, wherein the performing of the query includes sending a request including the key.

13. The method of claim 12, wherein generating the key comprises: The key is generated by applying a hash operation to at least a portion of the metadata included in the set of data sets.

14. The method of claim 13, wherein the at least a portion of the metadata included in the set of data sets indicates a domain from which the set of data sets was received.

15. A method as claimed in any preceding claim, wherein for a data set in the plurality of data sets, the encrypted data represents an interaction of a user with an online resource.

16. A computing system for managing a privacy budget, the computing system comprising: One or more servers; as well as A non-transitory computer readable medium storing thereon instructions which, when executed by the one or more servers, cause the computing system to implement the method according to any one of the preceding claims.

Images