Multi-key encrypted data deduplication

Abstract

A computer-implemented method includes sending key group information to a storage system. The key group information includes key ID information for client data keys in a key group. The client data keys enable deduplication of data chunks encrypted with any of the client data keys in the key group. The method also includes generating deduplication information. The deduplication information includes fingerprints associated with the client data chunks. The method also includes encrypting a data chunk with one of the client data keys, where a corresponding decryption key for the encrypted data chunk is not available to the storage system. The method includes sending the deduplication information to the storage system for use by the storage system in a deduplication process and sending the encrypted data chunk to the storage system.

Description

BACKGROUND

[0001] The present invention relates to data deduplication, and more specifically, the present invention relates to multi-key encrypted data deduplication in cloud storage systems and networks.

[0002] Conventional data reduction techniques, such as deduplication and / or compression, do not provide meaningful reduction when applied to encrypted data. Deduplication of multiple data sets each encrypted with a unique encryption key fails in cases where various encryption algorithms prevent conventional deduplication processes from identifying duplicate data blocks. Conventional data reduction techniques also do not provide sufficient data privacy between a client and a storage system.

[0003] For example, a known Bring Your Own Key (BYOK) encryption technique involves a multi-party trust system. Although all data reduction functionality can be provided by a storage system that has access to all data, conventional BYOK systems do not provide data privacy between the storage system and the client because the storage system has access to the client keys. A third-party key service can also have access to shared encryption keys used to encrypt the client data. Data privacy only exists between users of this form of BYOK encryption.

[0004] Conventional at-rest encryption encrypts unencrypted input data with a key known to the storage system. The storage system can decrypt all data and perform deduplication on all data in the system. However, at-rest encryption does not provide data privacy.

[0005] Conventional full client encryption encrypts data with a key unknown to the storage system. The storage system only performs deduplication on data encrypted with a public key. Full client deduplication provides relatively high data privacy but hinders deduplication efficiency. SUMMARY

[0006] Viewed from a first aspect, the present invention provides a computer-implemented method comprising sending key group information to a storage system. The key group information includes key ID information for client data keys in a key group. The client data keys enable deduplication of data blocks encrypted in any of the client data keys in the key group. The method further comprises generating deduplication information. The deduplication information includes fingerprints associated with client data blocks. The method further comprises encrypting a data block with one of the client data keys, wherein a corresponding decryption key for the encrypted data block is not available to the storage system. The method includes sending the deduplication information to the storage system for use by the storage system in a deduplication process and sending the encrypted data block to the storage system. The foregoing method provides the benefit of enabling deduplication across data encrypted with any of the data keys in the set of data keys without the storage system having access to the data keys.

[0007] Preferably, the present invention provides a computer-implemented method, wherein the encrypted data chunks are sent to the storage system in response to a request from the storage system, wherein the request from the storage system further requests key ID information identifying the client data key used to encrypt each data chunk.

[0008] Preferably, the present invention provides a computer-implemented method, wherein each client data key is a key of a type selected from the group consisting of: a client secret key and a client dedup key.

[0009] Preferably, the present invention provides a computer-implemented method, wherein the request from the storage system is for client data associated with data chunks that are not identified as being for deduplication by the storage system, the method comprising sending client data associated with data chunks encrypted in a client secret key.

[0010] Preferably, the present invention provides a computer-implemented method, the method further comprising sending client data associated with data chunks encrypted in a client dedup key in response to a request from the storage system for client data associated with data chunks identified by the storage system for deduplication.

[0011] Preferably, the present invention provides a computer-implemented method, further comprising encrypting the fingerprints with a client secret fingerprint key prior to sending the dedup information to the storage system.

[0012] Preferably, the present invention provides a computer-implemented method, further comprising: requesting client data from the storage system; and receiving the client data as data chunks, wherein one or more of the data chunks are encrypted with a client secret key and one or more other of the data chunks are encrypted with a client dedup key.

[0013] Viewed from another aspect, the present invention provides a computer- implemented method, comprising: receiving key group information at a storage system, wherein the key group information comprises key ID information for client data keys in a key group, wherein the client data keys enable deduplication of data chunks encrypted in any of the client data keys in the key group; receiving encrypted client data for storage in the storage system; receiving dedup information, wherein the dedup information is accessible by the storage system for performing operations thereon, wherein the dedup information comprises fingerprints associated with chunks of the encrypted client data; identifying data chunks for deduplication based on the dedup information; and for data chunks that are not identified for deduplication, requesting client data associated with data chunks encrypted in one of the client data keys.

[0014] Preferably, the present application provides a computer-implemented method, wherein each client data key is a key of a type selected from the group consisting of: a client secret key and a client dedup key.

[0015] Preferably, the present application provides a computer-implemented method, further comprising, in response to a request for client data from a client, identifying data chunks associated with the data request; retrieving metadata associated with the data chunks, wherein the metadata includes key ID information; retrieving encrypted client data associated with the data request based on the metadata; sending the encrypted client data to the client.

[0016] Preferably, the present application provides a computer-implemented method, wherein the key ID information identifies a client data key used to encrypt each of the data chunks associated with the data request.

[0017] Preferably, the present application provides a computer-implemented method, wherein the fingerprint is encrypted with a client secret fingerprint key, wherein the client secret fingerprint key is not available to the storage system.

[0018] Preferably, the present application provides a computer-implemented method, further comprising: identifying a data chunk stored in the storage system encrypted with one of the client data keys; and requesting client data associated with the data chunk encrypted in the client dedup key.

[0019] Viewed from another aspect, the present application provides a system comprising: a processor; and logic integrated with the processor, executable by the processor, or integrated with and executable by the processor, the logic being configured to: send key group information to a storage system, wherein the key group information includes key ID information for client data keys in a key group, wherein the client data keys enable deduplication of data chunks encrypted in any of the client data keys in the key group; generate dedup information, wherein the dedup information includes fingerprints associated with client data chunks; encrypt the data chunks with one of the client data keys, wherein a corresponding decryption key for the encrypted data chunks is not available to the storage system; send the dedup information to the storage system for use by the storage system in a deduplication process; and send the encrypted data chunks to the storage system.

[0020] Preferably, the present application provides a system, wherein the encrypted data chunks are sent to the storage system in response to a request from the storage system, wherein the request from the storage system further requests key ID information identifying the client data keys used to encrypt each of the data chunks.

[0021] Preferably, the present application provides a system wherein each client data key is a key of a type selected from the group consisting of: a client secret key and a client dedup key.

[0022] Preferably, the present application provides a system wherein a request from a storage system is for client data associated with a data block that is not identified by the storage system for deduplication, the system comprising logic configured to send client data associated with a data block encrypted in a client secret key.

[0023] Preferably, the present application provides a system further comprising logic configured to send client data associated with a data block encrypted in a client dedup key in response to a request from a storage system for client data associated with a data block identified by the storage system for deduplication.

[0024] Preferably, the present application provides a system further comprising logic configured to encrypt the fingerprint with a client secret fingerprint key prior to sending the dedup information to the storage system.

[0025] Viewed from another aspect, the present application provides a system further comprising: a processor; and logic integrated with the processor, executable by the processor, or both, the logic being configured to: receive key group information at a storage system, wherein the key group information comprises key ID information for client data keys in a key group, wherein the client data keys enable deduplication of data blocks encrypted in any of the client data keys in the key group; receive encrypted client data for storage in the storage system; receive dedup information, wherein the dedup information is accessible by the storage system for performing operations thereon, wherein the dedup information comprises fingerprints associated with blocks of the encrypted client data; identify data blocks for deduplication based on the dedup information; and for data blocks that are not identified for deduplication, request client data associated with data blocks encrypted in one of the client data keys.

[0026] Preferably, the present application provides a system wherein each client data key is a key of a type selected from the group consisting of: a client secret key and a client dedup key.

[0027] Preferably, the present application provides a system further comprising logic configured to: in response to a request for client data from a client, identify a data block associated with the data request; retrieve metadata associated with the data block, wherein the metadata comprises key ID information; retrieve encrypted client data associated with the data request based on the metadata; and send the encrypted client data to the client.

[0028] Preferably, the present application provides a system wherein the key ID information identifies a client data key used to encrypt each of the data chunks associated with the data request.

[0029] Preferably, the present application provides a system wherein the fingerprint is encrypted with a client secret fingerprint key, wherein the client secret fingerprint key is not available to the storage system.

[0030] Viewed from another aspect, the present application provides a computer program product, the computer program product comprising: one or more computer readable storage media, and program instructions collectively stored on the one or more computer readable storage media, the program instructions comprising: program instructions to send key group information to a storage system, wherein the key group information comprises key ID information for client data keys in a key group, wherein the client data keys enable deduplication of data chunks encrypted in any of the client data keys in the key group; program instructions to generate deduplication information, wherein the deduplication information comprises a fingerprint associated with a client data chunk; program instructions to encrypt the data chunk using one of the client data keys, wherein a corresponding decryption key for the encrypted data chunk is not available to the storage system; program instructions to send the deduplication information to the storage system for use by the storage system in a deduplication process; and program instructions to send the encrypted data chunk to the storage system.

[0031] Other aspects and methods of the present application will become apparent to those skilled in the art upon consideration of the following detailed description with reference to the accompanying drawings. BRIEF DESCRIPTION OF DRAWINGS

[0032] Figure 1 A cloud computing environment in accordance with an aspect of the present application is depicted.

[0033] Figure 2 An abstraction model layer in accordance with an aspect of the present application is depicted.

[0034] Figure 3 A high level architecture in accordance with an aspect of the present application.

[0035] Figure 4 A high level architecture in accordance with an aspect of the present application.

[0036] Figure 5 A high level architecture in accordance with an aspect of the present application.

[0037] Figure 6 A high level architecture in accordance with an aspect of the present application.

[0038] Figure 7is a high-level architecture in accordance with an aspect of the present application.

[0039] Figure 8 is a high-level architecture in accordance with an aspect of the present application.

[0040] Figure 9 is a high-level architecture in accordance with an aspect of the present application.

[0041] Figure 10 is a high-level architecture in accordance with an aspect of the present application.

[0042] Figure 11 is a flowchart of a method in accordance with an aspect of the present application. DETAILED DESCRIPTION

[0043] The following description is made for the purposes of illustrating the general principles of the present application and is not meant to limit the inventive concepts claimed herein. Further, described features can be combined in any combination and / or permutation of features described in this specification, in any other combination and / or permutation of features described in the art, and in any combination and / or permutation of features described in the art and features described in this specification.

[0044] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. It will be further understood that terms, such as those defined in commonly used dictionaries, should be interpreted as having a meaning that is consistent with their meaning in the context of the specification and relevant art and should not be interpreted in an overly

[0045] It must also be noted that, as used in the specification and the appended claims, the singular forms "a," "an" and "the" include plural referents unless otherwise specified. It will be further understood that the terms "includes" and / or "comprising," when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0046] The following description discloses several aspects of multi-key encrypted data deduplication.

[0047] In one general aspect, a computer-implemented method includes sending key group information to a storage system. The key group information includes key ID information for client data keys in a key group. The client data keys enable deduplication of data blocks encrypted in any of the client data keys in the key group. The method also includes generating deduplication information. The deduplication information includes fingerprints associated with client data blocks. The method also includes encrypting a data block with one of the client data keys, where a corresponding decryption key for the encrypted data block is not available to the storage system. The method includes sending the deduplication information to the storage system for use by the storage system in a deduplication process and sending the encrypted data block to the storage system.

[0048] In another general aspect, a system includes a processor; and logic integrated with the processor, executable by the processor, or integrated with and executable by the processor. The logic is configured to perform the aforementioned method.

[0049] In another general aspect, a computer program product includes one or more computer-readable storage media, and program instructions collectively stored on the one or more computer-readable storage media, the program instructions including program instructions to perform the aforementioned method.

[0050] In yet another general aspect, a computer-implemented method includes receiving, at a storage system, key group information. The key group information includes key ID information for client data keys in a key group. The client data keys enable deduplication of data blocks encrypted in any of the client data keys in the key group. The method includes receiving encrypted client data for storage in the storage system, and receiving deduplication information. The deduplication information is accessible by the storage system for performing operations thereon, and the deduplication information includes fingerprints associated with blocks of the encrypted client data. The method also includes identifying data blocks for deduplication based on the deduplication information, and for data blocks not identified for deduplication, requesting client data associated with the data blocks encrypted in one of the client data keys.

[0051] In another general aspect, a system includes a processor; and logic integrated with the processor, executable by the processor, or integrated with and executable by the processor. The logic is configured to perform the aforementioned method.

[0052] It can be appreciated that, although the present disclosure includes detailed descriptions of cloud computing, implementation of the teachings recited herein are not limited to a cloud computing environment. Rather, the aspects of the application are capable of being implemented in conjunction with any other type of computing environment now known or later developed.

[0053] Cloud computing is a model of service delivery for enabling convenient, on-demand network access to a shared pool of configurable computing resources (e.g. networks, network bandwidth, servers, processing, memory, storage, applications, virtual machines, and services) that can be rapidly provisioned and released with minimal management effort or interaction with a provider of the service. This cloud model can include at least five characteristics, at least three service models, and at least four deployment models.

[0054] Features include the following:

[0055] On-demand self-service: cloud consumers can unilaterally provision computing capabilities, such as server time and network storage, as needed automatically without requiring human interaction with the service's provider.

[0056] Wide Area Network (WAN) Access: Capabilities are available on the network and accessed through standard mechanisms that facilitate the use of heterogeneous thin or thick client platforms (e.g., mobile phones, laptops, and PDAs).

[0057] Resource pooling: The provider’s computing resources are pooled to serve multiple consumers using a multi-tenant model, where different physical and virtual resources are dynamically allocated and reallocated according to demand.

[0058] Location independence has significance because consumers typically do not control or know the exact location of the resources provided, but can specify the location at a higher level of abstraction (such as country, state, or data center).

[0059] Rapid Flexibility: In some cases, the ability to scale outwards and inwards quickly and flexibly can be provided. For consumers, the available capacity often appears unlimited and can be purchased in any quantity at any time.

[0060] Measurement services: Cloud systems automatically control and optimize resource usage by leveraging metering capabilities at a level of abstraction appropriate to the service type (e.g., storage, processing, bandwidth, and active user accounts). Resource usage can be monitored, controlled, and reported, providing transparency to both the providers and consumers of the services being utilized.

[0061] The service model is as follows:

[0062] Software as a Service (SaaS): The capability offered to consumers is the ability to use the provider's applications running on cloud infrastructure. Applications can be accessed from various client devices through thin client interfaces such as web browsers (e.g., web-based email). Consumers do not manage or control the underlying cloud infrastructure, including the network, servers, operating system, storage, or even individual application capabilities, with the possible exception of limited user-specific application configuration settings.

[0063] Platform as a Service (PaaS): This provides consumers with the ability to deploy consumer-created or acquired applications onto cloud infrastructure using programming languages ​​and tools supported by the provider. Consumers do not manage or control the underlying cloud infrastructure, including networks, servers, operating systems, or storage, but they have control over the deployed applications and the configuration of any application hosting environments.

[0064] Infrastructure as a Service (laaS): the capability provided to the consumer is to provision processing, storage, networks, and other fundamental computing resources where the consumer is able to deploy and run arbitrary software, which can include operating systems and applications. The consumer does not manage or control the underlying cloud infrastructure but has control over operating systems, storage, deployed applications, and possibly limited control of select networking components (e.g., host firewalls).

[0065] Deployment models are as follows:

[0066] Private cloud: the cloud infrastructure is operated solely for an organization. It can be managed by the organization or a third party and can exist on-premises or off-premises.

[0067] Community cloud: the cloud infrastructure is shared by several organizations and supports mission-critical enterprise resources. It can be managed by the organizations or a third party and can exist on-premises or off-premises.

[0068] Public cloud: the cloud infrastructure is made available to general public or a large industry group and is owned by an organization selling cloud services.

[0069] Hybrid cloud: the cloud infrastructure is a composition of two or more types of cloud (private, community, or public) that remain unique clouds but are bound together by standardized or proprietary technologies that enable data and application portability.

[0070] A cloud computing environment is service-oriented with a focus on statelessness, low coupling, modularity, and semantic interoperability. At the heart of cloud computing is an infrastructure of interconnected nodes, including end-points of data transfer.

[0071] Referring now to the drawing Figure 1 , illustrative cloud computing environment 50 is depicted. As shown, cloud computing environment 50 includes one or more cloud computing nodes 10 with which local computing devices used by cloud consumers, such as, for example, personal digital assistant (PDA) or cellular telephone 54A, desktop computer 54B, laptop computer 54C, and / or automobile computer system 54N can communicate. Nodes 10 can communicate with one another. They can be grouped (not shown) physically or virtually, in one or more networks, such as Private, Community, Public, or Hybrid clouds as described hereinabove, or a combination thereof. This allows cloud computing environment 50 to offer infrastructure, platforms and / or software as services with greater reliability and Figure 1The types of computing devices 54A-N shown in FIGURE 6 are intended to be illustrative only and that computing nodes 10 and cloud computing environment 50 can communicate with any type of computerized device over any type of network and / or network addressable connection (e.g., using a web browser).

[0072] Referring now to FIGURE 6, Figure 2 a set of functional abstraction layers are provided by cloud computing environment 50 Figure 1 It should be understood that Figure 2 The components, layers, and functions shown in FIGURE 6 are intended to be illustrative only and that aspects of the application are not limited thereof. As depicted, the following layers and respective functions are provided:

[0073] Hardware and software layer 60 includes hardware and software components. Examples of hardware components include: mainframes 61; RISC (Reduced Instruction Set Computer) architecture based servers 62; servers 63; blade servers 64; storage devices 65; and networks and networking components 66. In some aspects, software components include network application server software 67 and database software 68.

[0074] Virtualization layer 70 provides an abstraction layer from which the following examples of virtual entities can be provided: virtual servers 71; virtual storage 72; virtual networks 73, including virtual private networks; virtual applications and operating systems 74; and virtual clients 75.

[0075] In one example, management layer 80 can provide the functions described below. Resource provisioning 81 provides dynamic procurement of computing resources and other resources that are utilized to perform tasks within the cloud computing environment. Metering and Pricing 82 provide cost tracking as resources are utilized within the cloud computing environment, and billing or invoicing for consumption of these resources. In one example, these resources can include application software licenses. Security provides identity verification for cloud consumers and tasks, as well as protection for data and other resources. User portal 83 provides access to the cloud computing environment for consumers and system administrators. Service level management 84 provides cloud computing resource allocation and management such that required service levels are met. Service Level Agreement (SLA) planning and fulfillment 85 provide pre-arrangement for, and procurement of, cloud computing resources for which future requirements are anticipated in accordance with an SLA.

[0076] Workloads layer 90 provides examples of functionality for which the cloud computing environment can be utilized. Examples of workloads and functions which can be provided from this layer include: mapping and navigation 91; software development and lifecycle management 92; virtual classroom education delivery 93; data analysis processing 94; transaction processing 95; and multi-tenant key encryption data de-duplication 96.

[0077] Conventional data reduction techniques, such as deduplication and / or compression, do not provide meaningful reduction when applied to encrypted data. Deduplication of multiple data sets each encrypted with a unique encryption key fails in cases where the various encryption algorithms prevent a conventional deduplication process from identifying duplicate data blocks. Conventional data reduction techniques also do not provide sufficient data privacy between a client and a storage system.

[0078] A keep your own key (KYOK) method for secure deduplication enables deduplication of encrypted data without access to any other client's encryption keys. Data from a client key can be deduplicated with respect to other data in that key or data in a storage system key. It is beneficial for a client to have a set of keys and the ability to deduplicate that set of keys while providing secure access to the data. The present disclosure provides a deduplication key associated with a set of data keys to enable deduplication of data encrypted with any data key in the set of data keys without the storage system accessing the keys.

[0079] At least some aspects of the present disclosure provide additional capabilities for KYOK secure deduplication that allow a client to encrypt data using multiple keys. Various aspects improve the deduplication of KYOK by increasing the set of data that deduplication is able to operate on. The various methods described herein maintain and improve data privacy compared to conventional encryption and / or deduplication techniques. The various operations for multiple key encrypted data deduplication provide relatively better data reduction than conventional full client encryption and less client overhead than client deduplication.

[0080] At least some of the operations described herein can be used with symmetric key encryption and / or asymmetric key encryption (e.g., public key infrastructure (PKI)). Those of ordinary skill in the art will appreciate that PKI encryption can be performed according to any configuration known in the art. For example, a public key in a PKI is not a secret key, and encrypting data with a public key requires a corresponding secret private key to decrypt. Secret keys referred to in the present disclosure refer to keys that are not shared between a client and a storage system.

[0081] Clients throughout the various aspects of the present disclosure are associated with a set of processes, users, other entities, etc. that have separate data access rights. For security, each of the foregoing entities has a secret data key for encrypting its data. As used throughout the present disclosure, an entity that has its own secret key is referred to as a key user. A client can desire to allow data deduplication across some subset of key users. As used in the present disclosure, a subset of key users is referred to as a key group. For example, a client can have secret keys k0, k1, k2... K7, where keys k0, k1, and k2 form key group 1 (e.g., g1), and secret keys K3, K4, K5 form key group 2 (e.g., g2). As will be appreciated by one of ordinary skill in the art upon reading the present disclosure, access control can be provided by the client that limits access to key users that are members of a key group. As will be appreciated by one of ordinary skill in the art upon reading the present disclosure, access control can be provided by limiting access of key users to particular keys in a key group.

[0082] To enable deduplication across a set of keys within a key group, a secret dedup key is provided for each key group, which will be described in detail below. The secret dedup key is used to encrypt deduplicated blocks. In some optional approaches, a secret fingerprint key is provided for each key group, which will be described in detail below. The secret fingerprint key is used to encrypt deduplicated metadata prior to sending the deduplicated metadata and / or deduplicated blocks to a storage system. The secret fingerprint key need not be available to key users. In at least some approaches, access to the secret dedup key and the secret fingerprint key can be shared among key users of the associated key group. As will be appreciated by one of ordinary skill in the art upon reading the present disclosure, because any data encrypted in the dedup key is common data that exists in different secret data keys of the key group, such sharing is not a privacy issue. Additionally, as with KYOK dedup, the key ID associated with each data block can be used to ensure that only data with a matching key ID that includes deduplicated data is returned for read requests, as discussed in further detail below.

[0083] Figure 3 A high-level architecture is depicted in accordance with various configurations. The architecture can be implemented in accordance with the present application in any of the environments described in Figures 1-2 The architecture 300 can be implemented in various configurations in any of the environments described in Figure 3 more or less elements than those specifically described in the detailed description herein.

[0084] Architecture 300 depicts an exemplary implementation of a write operation using symmetric key encryption. Architecture 300 includes client 1 302. As shown, client 1 302 includes key set 304, which includes secret data keys: k0 306, ki 308, and k2 310. Secret data keys k0 306, ki 308, and k2 310 can be symmetric keys, asymmetric keys, or the like, or any combination thereof. As shown, secret data keys k0 306, ki 308, and k2 310 are symmetric keys. Key set 304 includes secret fingerprint key ks 312 and secret deduplication key kd 314.

[0085] Client 1 302 includes meta-generator 316. In a preferred approach, meta-generator 316 computes deduplication metadata associated with data. In various configurations, meta-generator 316 can be located on client 1 302 and / or on storage system 338, or disposed on a network between client 1 302 and storage system 338. In various configurations, a portion of the KYOK functionality (e.g., metadata generation and / or encryption) can be located on a client, on a network connecting a client to storage, in a unit attached to a storage system (e.g., where the unit does not provide the storage system with access to client keys or to unencrypted data), or the like.

[0086] Preferably, the storage system does not have access to the client keys and / or unencrypted data. Various encryption operations, which will be described in detail below, are performed after meta-generator 316 generates metadata. In configurations where meta-generator 316 is located on storage system 338, meta-generator 316 is encapsulated so that client keys and unencrypted data are not disclosed to storage system 338. Meta-generator 316 can be encapsulated in any manner known in the art, including via hardware (e.g., a smart NIC), a trusted computing environment, or the like.

[0087] In a preferred method, the meta-generator 316 performs metadata operations on the data on the client 1 302 and identifies data deduplication opportunities in the data. The meta-generator 316 identifies data deduplication opportunities using any data deduplication technique known in the art. In one aspect, the meta-generator 316 uses fixed frame deduplication. In another aspect, the meta-generator 316 uses full object deduplication, etc. In yet another aspect, the meta-generator 316 uses a sliding window variable block deduplication technique to identify data deduplication opportunities. The meta-generator 316 identifies and / or computes deduplication fingerprints (e.g., rolling minimal hash, cryptographic hash, etc.). The meta-generator 316 generates metadata (e.g., deduplication information) associated with the data deduplication opportunities, including pointers, block lengths, block fingerprints, etc. Portions of the deduplication information, such as the block fingerprints, can be encrypted with the client secret fingerprint key ks 312 discussed in detail below. In one method, the meta-generator 316 compresses the data using any data compression technique known in the art. In some methods, various compression techniques can be applied before and / or after chunking. In one configuration, pre-chunking compression can be a type of compression that improves chunking performance. In another configuration, post-chunking compression can be adjusted to minimize the resulting chunk size.

[0088] Fingerprint computation can be performed on the data chunks before and / or after compression. Pre-compression fingerprint computation allows the fingerprint to be verified after the read is decompressed. Post-compression fingerprint computation allows the fingerprint to be checked while the chunk is in a compressed state.

[0089] The client 1 302 includes a secret data key encrypter 318. As will be appreciated by one of ordinary skill in the art upon reading the present disclosure, the secret data key encrypter 318 can encrypt data with the secret data keys (e.g., ko 306, ki 308, and k2 310).

[0090] The client 1 302 includes a secret dedup key encrypter 320. As will be appreciated by one of ordinary skill in the art upon reading the present disclosure, the secret dedup key encrypter 320 can encrypt data with the secret dedup key kd 314.

[0091] The client 1 302 includes a secret fingerprint key encrypter 324. As will be appreciated by one of ordinary skill in the art upon reading the present disclosure, the secret fingerprint key encrypter 324 can encrypt fingerprints with the secret fingerprint key ks 312. In an alternative method, the secret fingerprint key and the secret dedup key can be the same key. In the above alternative method, as will be appreciated by one of ordinary skill in the art upon reading the present disclosure, the secret fingerprint key encrypter can encrypt data and / or fingerprints with the secret fingerprint key.

[0092] Client 1 302 includes a block decryptor 326. In various configurations, the block decryptor 326 can be located on client 1 302 and / or on storage system 338, or disposed on a network between client 1 302 and storage system 338. In various configurations, the block decryptor (e.g., including functionality associated therewith) can be located on a client, on a network connecting the client to storage, in a unit attached to the storage system (e.g., where the unit does not provide the storage system with access to client keys or unencrypted data), etc.

[0093] Preferably, the storage system does not have access to the client keys and / or unencrypted data. In configurations where the block decryptor 326 is located on the storage system 338, the block decryptor 326 is encapsulated such that the client keys and unencrypted data are not disclosed to the storage system 338. The block decryptor 326 can be encapsulated in any manner known in the art, including via hardware (e.g., a smart NIC), a trusted computing environment, etc.

[0094] The block decryptor 326 can block decrypt data blocks received from the storage system, where the data blocks are encrypted with a plurality of encryption keys. In a preferred approach, data blocks encrypted with a plurality of encryption keys refers to a situation where at least some data blocks are encrypted with a first key and at least some other data blocks are encrypted with a second key, where the first key and the second key are different keys. Unless specifically disclosed herein, data blocks encrypted with a plurality of encryption keys should not be interpreted as data blocks that are serially and / or simultaneously subject to a plurality of client secret encryption keys.

[0095] The block decryptor 326 can identify which encryption key a data block is encrypted with and decrypt the data block with a corresponding decryptor (e.g., secret data key decryptor 328, secret fingerprint key decryptor 330, secret dedup key decryptor 332, etc.). As will be appreciated by one of ordinary skill in the art upon reading the present disclosure, each decryptor can utilize a corresponding key to decrypt data.

[0096] Architecture 300 optionally includes a key ID manager 336. As will be appreciated by one of ordinary skill in the art upon reading the present disclosure, the key ID manager 336 stores and / or manages keys and any associated key IDs in any manner known in the art.

[0097] Architecture 300 includes a storage system 338. Storage system 338 can be any type of storage system known in the art. One of ordinary skill in the art can appreciate that storage system 338 can have more or less components than listed here. Storage system 338 preferably performs various deduplication operations described herein.

[0098] The storage system 338 includes a secret storage encryption key 340. In a preferred approach, the secret storage encryption key 340 never leaves the storage system 338. In this preferred approach, the secret storage encryption key 340 is not shared with the client 1 302, the client 2 342, any other client, any other storage system, etc. The secret storage encryption key 340 can be a symmetric key, an asymmetric key, etc. As shown, the secret storage encryption key 340 is a symmetric key. As will be appreciated by one of ordinary skill in the art upon reading the present disclosure, the secret storage encryption key 340 can optionally be used to encrypt client data, data blocks, block fingerprints, etc. received from a client.

[0099] The storage system 338 includes a deduplication manager 344. The deduplication manager 344 locates and / or identifies duplicate data based on deduplication information provided by the client 1 302. For chunked deduplication, the deduplication information is used with block metadata to identify duplicate data blocks. As will be appreciated by one of ordinary skill in the art upon reading the present disclosure, the deduplication manager 344 performs any associated deduplication operations. Any known deduplication technique can be used to deduplicate data blocks.

[0100] The storage system 338 includes a block store 346. The block store 346 can include any storage component known in the art. The block store 346 can store data blocks, deduplicated data, non-deduplicated data, encrypted data, non-encrypted data, deduplication information, etc. in any manner known in the art. In one approach, the block store 346 stores data blocks, deduplicated data, non-deduplicated data, encrypted data, non-encrypted data, deduplication information, etc. in a table.

[0101] The storage system 338 includes block metadata in a storage 348. The storage 348 can include any storage component known in the art. The storage 348 can store block metadata, data blocks, deduplicated data, non-deduplicated data, encrypted data, non-encrypted data, deduplication information, etc. in any manner known in the art. The block metadata can include any metadata associated with non-deduplicated and / or deduplicated data blocks, including pointers, block lengths, block fingerprints, etc. The block metadata can be updated in response to receiving new data from the deduplication manager 344.

[0102] The storage system 338 includes a key ID list 350. The key ID list 350 includes a list of key IDs associated with a set of secret data keys (e.g., k0 306, k1 308, and k2 310) used by the client 1 302 to encrypt data.

[0103] The storage system 338 includes a secret storage key encrypter 352. As will be appreciated by one of ordinary skill in the art upon reading the present disclosure, the secret storage key encrypter 352 can optionally utilize the secret storage encryption key 340 to encrypt data.

[0104] The storage system 338 includes a secret storage key decrypter 354. As will be appreciated by one of ordinary skill in the art upon reading the present disclosure, the secret storage key decrypter 354 can decrypt data encrypted with the secret storage encryption key 340.

[0105] The client 1 302 secret data keys (e.g., k0 306, k1 308, and k2 310) are not shared with the storage system 338, and the secret storage encryption key 340 is not shared with any client. The clients and / or the storage system can choose to use a key manager (see optional key ID manager 336) and / or any additional means to store and / or manage their keys.

[0106] The architecture 300 includes a client 2 342. As shown, the client 2 342 uses static encryption storage and sends unencrypted data (e.g., non-private data) to the storage system 338. Because the client 2 342 does not have any data privacy considerations in at least some approaches, the client 2 342 can send unencrypted data. Unencrypted data can refer to data that is received by the storage system in plaintext. In various approaches, any communication encryption known in the art can be implemented between various components. Any communication encryption between at least some of the various components can be implemented according to methods known in the art.

[0107] Before writing data, the client 1 302 can issue a request 358 to the storage system 338. The client 1 302 can send client key group information from the key ID list 350 associated with the request 358. The client key group information from the key ID list 350 informs the storage system 338 of the set of keys that the client will use, and uniquely identifies each key that the client will use. The client key group information from the key ID list 350 can include a client identifier, a group identifier, a tag indicating whether the group allows deduplication against plaintext (e.g., unencrypted blocks), a list of key IDs in the group, and any associated key type identifiers (e.g., data, dedup, etc.) for each key ID, etc. The client key group information from the key ID list 350 preferably does not include the keys associated with the key IDs.

[0108] In a preferred approach, the key ID is not easily predictable and / or determinable from the key itself. The key ID can be generated using a random value and / or a value encrypted with a separate key. Any other technique for generating a key ID can be used. A key ID that is not easily predictable and / or determinable from the key itself prevents an unauthorized entity from gaining access to the storage. For example, malware can gain access to a process that is authorized to use a given client secret data key ID. It is beneficial to make it relatively difficult for the process to use that information to infer other key ID values, thereby limiting the ability to leak and / or alter data with other keys. The 360 key ID can be obtained from the key ID manager 336. The key ID manager 336 can be used to communicate 362 the key ID to the storage system 338. The key ID manager 336 can be implemented as software, hardware, a separate entity, a process on the client 1 302, or the like, or any combination thereof.

[0109] As part of the request 358, the storage system 338 can verify the identity and access permissions associated with the client 1 302 and the request 358. In one approach, the client 1 302 establishes a session with the storage system 338 via the request 358 to the storage system 338.

[0110] In one approach, the client 1 302 encrypts the write data prior to sending the write data to the metabuilder 316. The client 1 302 sends the write data and a key ID representing the secret data key used to encrypt the write data (e.g., k0 306, k1 308, k2 310) to the metabuilder 316.

[0111] In a preferred approach, the metabuilder 316 computes deduplication information (e.g., chunk metadata) associated with the write data. The deduplication information can include operational details of the deduplication approach that will be understood by those of ordinary skill in the art upon reading the present disclosure. In one configuration, for chunk deduplication, the deduplication information can include a chunk start position, a chunk length, a chunk fingerprint, similarity information, and the like. In one approach, the chunk fingerprint is a cryptographic hash that can be computed in any manner known in the art.

[0112] In one approach, the write data is compressed by the metabuilder 316. Any form of compression known in the art can be used. In some aspects, the write data can be sent in response to a request from the storage system 338 (to be discussed in detail below). In response to the request, the compressed data can be sent to the secret data key encryptor 318 and encrypted with the appropriate secret data key from the key set 304. The secret data key encrypted data and the key ID associated with the secret data key from the key set 304 can be sent to the storage system 338 (e.g., the deduplication manager 344 on the storage system 338).

[0113] In at least some methods, the write data is sent to the secret fingerprint key encrypter 324 and the corresponding fingerprint is encrypted with the secret fingerprint key ks 312. The write data and key ID associated with the secret fingerprint key ks 312 can be sent to the storage system 338 (e.g., to the dedup manager 344 on the storage system 338).

[0114] In yet another method, the write data can be sent in response to a request from the storage system 338 (to be discussed in detail below). In response to the request, the write data is sent to the secret dedup key encrypter 320 and encrypted with the secret dedup key kd 314. The write data and key ID associated with the secret dedup key kd 314 can be sent to the storage system 338 (e.g., to the dedup manager 344).

[0115] In any of the foregoing methods, the data is preferably opaque to the storage system 338 (e.g., the data is encrypted with the secret data key, the secret fingerprint key ks 312, or the secret dedup key kd 314, where these keys are not available to the storage system 338). In various methods, the dedup information and the write data are tagged and / or otherwise indicated as being associated.

[0116] The dedup manager 344 locates and / or identifies duplicate data based on the dedup information. For chunk dedup, the dedup information is used with the chunk metadata to identify duplicate chunks of data. For example, the dedup manager 344 compares the pointers, fingerprints, chunk lengths, etc. from the decrypted dedup information to any pointers, fingerprints, chunk lengths, etc. stored in the chunk metadata. The dedup manager 344 performs any associated dedup operations, as will be appreciated by one of ordinary skill in the art upon reading the present disclosure.

[0117] In response to performing the associated dedup operations, updates can be performed to the chunk metadata. The updates reflect changes due to using any deduplicated chunks, adding information about any remaining chunks, etc. The chunk metadata includes information about the key IDs used to encrypt a given chunk of data. Any remaining chunks of data associated with new fingerprints that were not deduplicated (e.g., that did not pre-exist in the chunk metadata) are written to the chunk store 346. Any new, non-deduplicated chunks of data are written to the chunk store 346.

[0118] Subsequent data access requests include a key ID associated with the client secret data key used to encrypt the data. The key ID can be part of a particular message. The key ID can be part of an established session, process, etc.

[0119] The following detailed description of the various aspects discloses the behavior of the aforementioned configuration when using a sliding window variable block size for deduplication. Prior to the operations described in detail below, the client may request key group information 358 from key ID list 350. Note that a client operation sending unencrypted data may send key group information to storage system 338. For plaintext data, default information can be assumed. Note that group and key ID information can be transmitted as part of the session, rather than being sent for each input / output (IO). It is beneficial for the storage system to encrypt all stored data 368 using a secret storage key encryptor 352, even for data encrypted with a private client key. The aforementioned encryption is assumed in the following description.

[0120] Figure 4 A high-level architecture based on various configurations is described. According to the present invention, it is possible to... Figures 1-3 Architecture 400 can be implemented in various configurations in any environment described in 5-11. Of course, as those skilled in the art will understand upon reading this description, architecture 400 may include more than [previous configurations]. Figure 4 The number of components described in the text may be more or less than the number of components.

[0121] Architecture 400 describes an exemplary implementation of a read operation encrypted using a symmetric key. Architecture 400 includes a client 1 302. As shown, client 1 302 includes a key set 304, which includes secret data keys: k0 306, k1 308, and k2 310. The secret data keys k0 306, k1 308, and k2 310 can be symmetric keys, asymmetric keys, etc., or any combination thereof. As shown, the secret data keys k0 306, k1 308, and k2 310 are symmetric keys. Key set 304 includes a secret fingerprint key ks 312 and a secret deduplication key kd 314.

[0122] Client 1 302 includes a metadata generator 316. In various configurations, the metadata generator 316 may reside on client 1 302 and / or storage system 338, or be disposed on a network between client 1 302 and storage system 338. In various configurations, a portion of the KYOK functionality (e.g., metadata generation and / or encryption) may reside on the client, on a network connecting the client to the storage device, or in a unit attached to the storage system (e.g., where the unit does not provide the storage system with access to the client's key or to unencrypted data), etc.

[0123] Preferably, the storage system does not have access to the client keys and / or unencrypted data. Various encryption operations, which will be described in detail below, are performed after the metadata is generated by the metadata generator 316. In configurations where the metadata generator 316 is located on the storage system 338, the metadata generator 316 is encapsulated such that the client keys and unencrypted data are not exposed to the storage system 338. The metadata generator 316 can be encapsulated in any manner known in the art, including via hardware (e.g., a smart NIC), a trusted computing environment, etc.

[0124] In preferred methods, the metadata generator 316 computes deduplication metadata associated with the data. In various configurations, the metadata generator 316 can be located on the client 1 302 and / or on the storage system 338.

[0125] In preferred methods, the metadata generator 316 performs metadata operations on the data on the client 1 302 and identifies data deduplication opportunities in the data. The metadata generator 316 identifies data deduplication opportunities using any data deduplication technique known in the art. In one aspect, the metadata generator 316 uses a sliding window variable block deduplication technique to identify data deduplication opportunities. The metadata generator 316 identifies and / or computes deduplication fingerprints (e.g., rolling minimum hash, cryptographic hash, etc.). The metadata generator 316 generates metadata (e.g., deduplication information) associated with the data deduplication opportunities, including pointers, block lengths, block fingerprints, etc. Portions of the deduplication information, such as the block fingerprints, can be encrypted with the client secret fingerprint key ks 312, as discussed in detail below.

[0126] In one method, the metadata generator 316 compresses the data using any data compression technique known in the art. In some methods, various compression techniques can be applied before and / or after chunking. In one configuration, pre-chunking compression can be a compression that improves chunking performance. In another configuration, post-chunking compression can be adjusted to minimize the resulting chunk size.

[0127] Fingerprint computation can be performed on the data chunks before and / or after compression. Pre-compression fingerprint computation allows the fingerprint to be verified after the read is decompressed. Post-compression fingerprint computation allows the fingerprint to be checked while the chunk is in a compressed state.

[0128] The client 1 302 includes a secret data key encryptor 318. As will be appreciated by one of ordinary skill in the art upon reading the present disclosure, the secret data key encryptor 318 can encrypt data with secret data keys (e.g., k0 306, ki 308, and k2 310).

[0129] Client 1 302 includes a secret dedup key encrypter 320. As will be appreciated by one of ordinary skill in the art upon reading the present disclosure, secret dedup key encrypter 320 can encrypt data with a secret dedup key kd 314.

[0130] Client 1 302 includes a secret fingerprint key encrypter 324. As will be appreciated by one of ordinary skill in the art upon reading the present disclosure, secret fingerprint key encrypter 324 encrypts fingerprints with a secret fingerprint key ks 312.

[0131] Client 1 302 includes a block decrypter 326. In various configurations, block decrypter 326 can be located on client 1 302 and / or on storage system 338, or disposed on a network between client 1 302 and storage system 338. In various configurations, the block decrypter (e.g., including functionality associated therewith) can be located on a client, located on a network connecting the client to storage, located in a unit attached to the storage system (e.g., where the unit does not provide the storage system with access to client keys or unencrypted data), etc.

[0132] Preferably, the storage system does not have access to the client keys and / or unencrypted data. In configurations where block decrypter 326 is located on storage system 338, block decrypter 326 is encapsulated such that the client keys and unencrypted data are not disclosed to storage system 338. Block decrypter 326 can be encapsulated in any manner known in the art, including via hardware (e.g., a smart NIC), a trusted computing environment, etc.

[0133] Block decrypter 326 can block decrypt data blocks received from the storage system, where the data blocks are encrypted with multiple encryption keys. In a preferred approach, the data blocks encrypted with multiple encryption keys are where at least some data blocks are encrypted with a first key and at least some other data blocks are encrypted with a second key, where the first key and the second key are different keys. Unless specifically disclosed herein, data blocks encrypted with multiple encryption keys should not be interpreted as data blocks that are serially and / or simultaneously subject to multiple client secret encryption keys.

[0134] Block decrypter 326 can identify which encryption key a data block is encrypted with and decrypt the data block with a corresponding decrypter (e.g., secret data key decrypter 328, secret fingerprint key decrypter 330, secret dedup key decrypter 332, etc.). As will be appreciated by one of ordinary skill in the art upon reading the present disclosure, each decrypter can decrypt data with a corresponding key.

[0135] Architecture 400 optionally includes a key ID manager 336. As will be appreciated by one of ordinary skill in the art upon reading the present disclosure, key ID manager 336 stores and / or manages keys and any associated key IDs in any manner known in the art.

[0136] Architecture 400 includes a storage system 338. Storage system 338 can be any type of storage system known in the art. One of ordinary skill in the art will appreciate that storage system 338 can have more or less components than listed here. Storage system 338 preferably performs the various deduplication operations described herein.

[0137] Storage system 338 includes a secret storage encryption key 340. In a preferred approach, secret storage encryption key 340 never leaves storage system 338. In this preferred approach, secret storage encryption key 340 is not shared with client 1 302, client 2 342, any other client, any other storage system, etc. Secret storage encryption key 340 can be a symmetric key, an asymmetric key, etc. As shown, secret storage encryption key 340 is a symmetric key.

[0138] Storage system 338 includes a deduplication manager 344. Deduplication manager 344 locates and / or identifies duplicate data based on deduplication information provided by client 1 302. For chunk deduplication, the deduplication information is used with chunk metadata to identify duplicate chunks of data. As will be appreciated by one of ordinary skill in the art upon reading the present disclosure, deduplication manager 344 performs any associated deduplication operations. Any known deduplication technique can be used to deduplicate chunks of data.

[0139] Storage system 338 includes a chunk storage 346. Chunk storage 346 can include any storage component known in the art. Chunk storage 346 can store chunks of data, deduplicated data, non-deduplicated data, encrypted data, non-encrypted data, deduplication information, etc. in any manner known in the art. In one approach, chunk storage 346 stores chunks of data, deduplicated data, non-deduplicated data, encrypted data, non-encrypted data, deduplication information, etc. in a table.

[0140] Storage system 338 includes chunk metadata stored in storage 348. Storage 348 can include any storage component known in the art. Storage 348 can store chunk metadata, chunks of data, deduplicated data, non-deduplicated data, encrypted data, non-encrypted data, deduplication information, etc. in any manner known in the art. Chunk metadata can include any metadata associated with non-deduplicated and / or deduplicated chunks of data, including pointers, chunk lengths, chunk fingerprints, etc. Chunk metadata is updated in response to receiving new data from deduplication manager 344.

[0141] The storage system 338 includes a key ID list 350. The key ID list 350 includes a list of key IDs associated with the set of secret data keys (e.g., k0 306, ki 308, and k2 310) used by the client 1 302 to encrypt data.

[0142] The storage system 338 includes a secret storage key encrypter 352. As will be appreciated by one of ordinary skill in the art upon reading the present disclosure, the secret storage key encrypter 352 can optionally encrypt data with the secret storage encryption key 340.

[0143] The storage system 338 includes a secret storage key decrypter 354. As will be appreciated by one of ordinary skill in the art upon reading the present disclosure, the secret storage key decrypter 354 can decrypt data encrypted with the secret storage encryption key 340.

[0144] The client 1 302 secret data keys (e.g., k0 306, ki 308, and k2 310) are not shared with the storage system 338, and the secret storage encryption key 340 is not shared with any client. The clients and / or the storage system can choose to store and / or manage their keys using a key manager (see optional key ID manager 336) and / or any additional means.

[0145] The architecture 400 includes a client 2 342. As shown, the client 2 342 uses static encryption storage and sends unencrypted data (e.g., non-private data) to the storage system 338. Because the client 2 342 does not have any data privacy considerations in at least some approaches, the client 2 342 can send unencrypted data. Unencrypted data can refer to data that is received by the storage system in plaintext. In various approaches, any communication encryption known in the art can be implemented between various components. Any communication encryption between at least some of the various components can be implemented according to methods known in the art.

[0146] Before reading the data, client 1 302 can issue a request 358 to storage system 338. In preferred methods, request 358 includes a request for keyset information from key ID list 350 associated with request 358. The client keyset information from key ID list 350 informs storage system 338 of the set of keys that the client will use, and uniquely identifies each key that the client will use. The client keyset information from key ID list 350 can include a client identifier, a group identifier, a tag indicating whether the group allows deduplication against plaintext (e.g., unencrypted blocks), a list of key IDs in the group, and any associated key type identifiers (e.g., data, dedup, etc.) for each key ID, etc. The client keyset information from key ID list 350 preferably does not include the keys associated with the key IDs.

[0147] Client 1 302 sends a read request 402 to storage system 338 that includes the key ID values associated with reading the data. In some methods, authorization for read request 402 can require a non-zero reference count for each block that includes read request 402.

[0148] Dedup manager 344 uses block metadata associated with block storage 346 (e.g., metadata storage) to retrieve information associated with the blocks associated with read request 402. The block metadata includes key IDs for the keys used to encrypt each block, reference counts, locations, any other block information, etc., or any combination thereof. At operation 404, dedup manager 344 requests the appropriate set of blocks from block storage 346 on storage system 338. Each block and associated metadata (e.g., including the key ID for each block) is sent 406 to secret storage key decryptor 354. Each block and associated metadata (e.g., including the key ID for each block) is sent 408 to block decryptor 326. Block decryptor 326 checks the key ID for each block. Block decryptor 326 decrypts each block with the appropriate decryptor for the key associated with the key ID (e.g., secret data key decryptor 328, secret dedup key decryptor 332, a decryptor for plaintext key IDs, etc.). As will be appreciated by one of ordinary skill in the art upon reading the present disclosure, each decryptor can utilize a corresponding key to decrypt the data.

[0149] Block decryptor 326 performs any necessary decompression and merges the blocks to produce output data 414 for client 1 302. Client 1 302 decryptor (e.g., secret fingerprint key decryptor 330, secret data key decryptor 328, secret dedup key decryptor 332, etc.) can be provided by client 1 302 separate from block decryptor 326 as a means of keeping client keys secret in some approaches. As will be appreciated by one of ordinary skill in the art, the decryptors (e.g., secret fingerprint key decryptor 330, secret data key decryptor 328, secret dedup key decryptor 332, etc.) can be separate components (as shown) and / or at least two of the decryptors can be a single unit with associated decryption keys provided as appropriate for each block.

[0150] In various operations, dedup manager 344 uses the block metadata to identify the data blocks associated with read request 402. The identified data blocks are read from block storage 346. Dedup manager 344 compares the pointers, fingerprints, block lengths, etc. from read request 402 to any pointers, fingerprints, block lengths, etc. stored in the block metadata. In preferred approaches, the block metadata includes an identifier for the key (e.g., key ID) used to encrypt each data block. As will be appreciated by one of ordinary skill in the art upon reading the present disclosure, encrypted data blocks are decrypted with the appropriate decryptor. For example, data blocks encrypted with secret storage encryption key 340 can be sent to secret storage key decryptor 354. Secret storage key decryptor 354 can decrypt data encrypted with secret storage encryption key 340.

[0151] Block decryptor 326 can block decrypt data blocks received from the storage system, where the data blocks are encrypted with multiple encryption keys. In preferred approaches, data blocks encrypted with multiple encryption keys is the case where at least some data blocks are encrypted with a first key and at least some other data blocks are encrypted with a second key, where the first key and the second key are different keys. Unless specifically disclosed herein, data blocks encrypted with multiple encryption keys should not be interpreted as data blocks that are serially and / or simultaneously subjected to multiple client secret encryption keys. Block decryptor 326 can identify which encryption key a data block is encrypted with and decrypt the data block using the corresponding decryptor (e.g., secret fingerprint key decryptor 330, secret data key decryptor 328, secret dedup key decryptor 332, etc.). As will be appreciated by one of ordinary skill in the art upon reading the present disclosure, each decryptor can decrypt data blocks encrypted with the corresponding key.

[0152] In various approaches, block decryptor 326 performs any decompression operations known in the art. Block decryptor 326 can merge the data blocks to produce output data 414 for client 1 302.

[0153] As will be appreciated by one of ordinary skill in the art upon reading the present disclosure, the decryptor can be provided separately from the chunk decryptor 326 by the client 1 302 to keep various secret keys secret.

[0154] The encrypted data chunks with client keys sent 408 from the storage system 338 can include data chunk metadata that includes chunk fingerprints to verify that each chunk is delivered correctly and decrypted with the appropriate key. The data chunk metadata can provide end-to-end checking.

[0155] In preferred configurations of the architecture 400, various compression and / or deduplication techniques can be performed while the keys never leave the client 1 302. The secret keys are not available to the storage system 338, the client 2 342, any other client, any third-party service, any other storage system, etc. Part of the deduplication process is preferably performed on the client side (e.g., the chunking process in the architecture 400), which significantly reduces or eliminates the external visibility of the keys.

[0156] In preferred methods, the communication between the client and the storage system can be encrypted using any encryption mechanism known in the art. For example, the encryption mechanism can be standard transport encryption (e.g., Transport Layer Security (TLS)).

[0157] In preferred methods, in contrast to conventional deduplication operations, additional chunk metadata is associated with each data chunk to indicate the key with which the chunk is encrypted. Different clients can have the same data chunk (e.g., a duplicate with the same fingerprint). However, if the duplicate data chunk from different clients is encrypted with different keys, conventional deduplication of the data chunk of a first client to the data chunk of a second client requires sharing the key. In sharp contrast, various configurations of the present disclosure provide a chunk identifier that includes information that identifies the key used to encrypt the data chunk, an index, a tag, a fingerprint, a key ID, etc. The information about the key ID can be provided by the client 1 302 to the storage system 338 prior to any input / output (I / O) operations.

[0158] In the context of data privacy, the key ID is not the key itself. For example, the provided key ID can be a cryptographic hash, such as a SHA256 of a secret data key of the key group 304. In another example, the key ID can be a unique index number for a secret data key of the key group 304. The storage system 338 can create an internal reference key ID. For example, the storage system 338 can create an index table of key IDs (e.g., the key ID list 350).

[0159] Block metadata can include a reference counter in various methods to indicate how many different pieces of data point to the data block. Metadata indicating the source key ID and the reference count for each key ID can be added to each data block. Attempts to read a block can be intercepted without a non-zero reference count for the requested key ID. The interception provides access control for deduplicated data blocks. For example, client 1 302 using the key ID associated with secret data key kl 308 can have data including deduplicated block A. Block A can be the only piece of data that is deduplicated for block A encrypted with secret data key kl 308. Client 1 302 can delete the data including deduplicated block A. The reference count for the client 1 302 key ID associated with secret data key kl 308 is decremented to 0. Any subsequent attempts to read block A with the key ID associated with client 1 302 secret data key kl 308 can be blocked, and any requests with a non-zero associated reference count can be allowed.

[0160] If data encrypted with a secret data key from key group 304 is not initially deduplicated, the data can be deduplicated in response to a data block encrypted with a secret storage encryption key 340 with a matching fingerprint written and / or an unencrypted data block written. In various configurations described herein, the process for deduplicating an initially non-deduplicated data block can be referred to as backporting. The backporting operation includes a deduplication operation that stores a new data block (e.g., encrypted with a key available to storage system 338). Specifically, the original block is written in the secret data key, and the backporting stores the new block in the secret deduplication key of the same key group. If the flag to allow plaintext deduplication is on, the new block is written only with keys known to the storage, and the new block is written in plaintext (e.g., sent to storage system 338), and the storage system encrypts the at-rest data with its own secret storage encryption key 340.

[0161] Backporting includes deleting any pre-existing blocks with matching fingerprints but with different key IDs. Metadata for the stored and / or deleted data blocks can be converted to point to the new data block (e.g., as if the new data block existed first). The order in which data blocks are written with different keys does not affect the final deduplication efficiency. Any subsequent at-rest and / or unencrypted writes of a data block copy use the backporting techniques described herein (where many private copies of the data block exist) to reduce storage capacity on the system.

[0162] The various configurations described herein can be implemented using asymmetric encryption keys (e.g., PKI). For relatively large data sets, symmetric key encryption can be preferable to asymmetric key encryption. Asymmetric key encryption includes a client having a public key shared with the storage system and a private key not shared with the storage system. The storage system can have a public key shared with the client(s) and a private key not shared with the client(s). The public key in PKI is not secret, but encrypting data in the public key requires the corresponding secret private key to decrypt the data.

[0163] Figure 5 A high-level architecture is depicted in accordance with various configurations. The architecture can be implemented in accordance with the present application in any of the environments described in FIGS. 1-11. Figures 1-4 The architecture 500 can be implemented in various configurations in any of the environments described in FIGS. 1-11. Of course, more or less elements than those specifically described herein can be included in the architecture 500, as would be understood by one of ordinary skill in the art upon reading the present descriptions. Figure 5 The elements specifically described in FIGS. 1-11 can be included in the architecture 500, as would be understood by one of ordinary skill in the art upon reading the present descriptions.

[0164] The architecture 500 describes an exemplary implementation of writing data in plaintext from the client c2 502 to the storage 504. The architecture 500 includes client c2 key group gO information 506. The client c2 key group gO information 506 includes a dct (deduplication of plaintext) flag. The dct flag is set to 1 to indicate that deduplication of plaintext is allowed. The only key in the key group is clr (plaintext), which indicates that no encryption is used in this configuration. The standard key ID of kid.clr is used to indicate that the data is not encrypted. For example, the key ID value 0 can be reserved as kid.clr so that all clients have the same plaintext key ID. The key is not sent to the storage 504 as part of the group information sent to the storage 504 (the send operation is not shown).

[0165] The storage 504 includes a cell generator 508. The cell generator 508 is executed in a manner similar to the cell generator 316 described in detail above with reference to FIG. 3. Figures 3-4 In other approaches, the cell generator 508 is located at the client c2 502.

[0166] Client c2 502 writes data object M2 510 with data m2. Operation 512 includes sending data object M2 510 with data m2 to meta generator 508 on storage 504. Meta generator 508 identifies 2 data blocks 514 as shown (e.g., data block d3 516 and data block d4 518). Data block d3 516 is associated with fingerprint S3 and data d3. Data block d4 518 is associated with fingerprint S4 and data d4. The fingerprints can include the block length of each respective data block in various methods as described in detail above. Metadata 520 includes key group identifier g0 and key ID kid.clr indicating that the data is in the clear. The data blocks are stored in operation 522. Operation 524 stores data block d3 516 with fingerprint S3 and associated data d3 in kid.clr (e.g., stored in the clear). Operation 526 stores data block d4 518 with fingerprint S4 and associated data d4 in kid.clr (e.g., stored in the clear).

[0167] Table 528 includes deduplicated key ID list metadata associated with each block, assuming no blocks are deduplicated at this time. The first row of table 528 shows that data block d3 516 with fingerprint S3 is owned by client c2 502. The first row of table 528 shows that data block d3 516 with fingerprint S3 is encrypted with kid.clr (e.g., the data block is in the clear), and is only accessible by using the kid.clr key ID. Table 528 shows that data block d3 516 with fingerprint S3 has a reference count of 1, and that data block d3 516 with fingerprint S3 is stored at location li. The second row of table 528 shows corresponding information about data block d4 518 with fingerprint S4 stored at location 12.

[0168] Figure 6 A high-level architecture is depicted in accordance with various configurations. In accordance with the present application, the architecture 600 can be implemented in various configurations in any of the environments described in Figures 1-5 The architecture 600 can include more or less elements than specifically described in the detailed description of FIGS. 1-11, in various configurations. Of course, as will be appreciated by one of ordinary skill in the art upon reading the present descriptions, many of the elements described in the architecture 600 can be implemented in software and / or hardware, in various configurations. Figure 6 The architecture 600 can include more or less elements than specifically described in the detailed description of FIGS. 1-11, in various configurations. Of course, as will be appreciated by one of ordinary skill in the art upon reading the present descriptions, many of the elements described in the architecture 600 can be implemented in software and / or hardware, in various configurations.

[0169] Architecture 600 depicts an exemplary implementation of further data being added to storage 504 by client cO 602 with key kO. Client cO key group gO information 604 includes a dct flag (dedup to cleartext) set to 1 to indicate that in addition to data that is allowed to be written in cleartext, deduplication is allowed for data written with key IDs that belong to group gO. Specifically, the dct = 1 flag allows deduplication for cleartext. Client cO key group gO information 604 shows that key group gO includes data key kO and data key k1 with corresponding key IDs kid.kO and kid.k1. Client cO key group gO information 604 includes a deduplication key kd with key ID kid.kd. Client cO key group gO information 604 includes a cleartext data key clr with an associated key ID kid.clr, as described above. The dct is equal to 1 to indicate that cleartext deduplication is allowed, and the fingerprints are not encrypted with a fingerprint key.

[0170] In the case of data being encrypted, a meta generator 606 resides in client cO 602. Meta generator 606 is similar to meta generator 316 described in detail above with reference to FIG. 3. Figures 3-4 Meta generator 316.

[0171] Client cO 602 writes data object M0 608 with data mO. Operation 610 includes sending data object M0 608 with data mO to meta generator 606. Meta generator 606 identifies 3 data blocks 612 as shown (e.g., data block d3 614, data block d2 616, and data block dO 618). Data block d3 614 is associated with fingerprint S3 and data d3. Data block d2 616 is associated with fingerprint S2 and data d2. Data block dO 618 is associated with fingerprint S0 and data dO. The fingerprints can include the block length of each respective data block in various methods as described in detail above. Metadata 620 includes key group identifier gO and key ID kid.kO indicating that the data is encrypted with a secret key associated with key ID kid.kO. Operation 622 includes sending fingerprints S0, S2, and S3 to storage 504. Key group identifier gO allows storage 504 to determine that the deduplication key ID is kid.kd from previously communicated group information (e.g., client cO key group gO information 604, also see request 358 with reference to FIG. 3). Key group identifier gO allows storage 504 to determine that client cO 602 of group gO is allowed to deduplicate for cleartext data. Figures 3-4

[0172] Storage 504 checks current block metadata table 528 (see Figure 5 ​To determine if S0 and S2 are new fingerprints, store 504 checks the block metadata table 528 to determine if S3 already exists in kid.clr. Because client c0 602 group g0 allows plaintext deduplication, and kid.clr is available to all clients (e.g., kid.clr indicates plaintext), block S3 (e.g., data block d3 614) can be deduplicated.

[0173] Storage 504 may request client c0 602 to send block S0 (e.g., data block d0 618) and block S2 (e.g., data block d2 616) from kid.k0. Storage 504 does not request block S3. Client c0 602 encrypts data block d0 with key k0 626 and sends the encrypted block along with metadata (e.g., including fingerprint S0 and key ID kid.k0) to storage 504 628. Similarly, after encrypting data block d2 630, block S2 is sent to storage 504 632. Storage 504 stores the block and updates the block metadata table 528. Row 1 of table 528 shows the update of S3 to include the information that the block was also written by client c0 602 with kid.k0 and has a reference count of 1 in the key ID, but retains the encryption key ID as kid.clr. S3 can be accessed using either kid.clr or kid.k0. Line 3 of Table 528 includes new metadata indicating that block S0 was written by client c0602, encrypted with a secret key associated with key ID kid.k0, accessible by kid.k0, having a reference count of 1 in that key ID, and stored in location l3. Line 4 of Table 528 shows similar information for block S2 stored in location l4.

[0174] Figure 7 A high-level architecture based on various configurations is described. According to the present invention, it is possible to... Figures 1-6 Architecture 700 can be implemented in various configurations in any environment described in 8-11. Of course, as those skilled in the art will understand upon reading this description, architecture 700 may include more than... Figure 7 The more or fewer elements specifically described in the text.

[0175] Architecture 700 describes an exemplary implementation where client c0 602 adds further data to storage 504 using key k1. Key k1 is part of key group g0 (see example...). Figure 6 The client c0 key group g0 information shown is 604).

[0176] Client cO 602 writes data object Ml 702 with data ml. Operation 704 includes sending data object Ml 702 with data ml to meta generator 606. Meta generator 606 identifies 2 data chunks 706 as shown (e.g., data chunk dl 708 and data chunk d2 710). Data chunk dl 708 is associated with fingerprint S I and data dl. Data chunk d2 710 is associated with fingerprint S2 and data d2. Metadata 712 includes key group identifier go and key ID kid.kl indicating that the data is encrypted with a secret key associated with key ID kid.kl. Operation 714 includes sending fingerprints S I and S2 to storage 504. Key group identifier go allows storage 504 to determine that the deduplication key ID is kid.kd from the previously transmitted group information (e.g., client cO key group go information 604, also see request 358 with reference to FIG. 3). Figure 3

[0177] Storage 504 checks current chunk metadata table 528 and determines that S I is a new fingerprint. Storage 504 checks current chunk metadata table 528 and determines that S2 exists in kid.kO. Chunk S2 (e.g., data chunk d2 710) can be deduplicated using group go deduplication kid.kd. Storage 504 can request 716 client cO 602 to send S I in kid.kl chunk (e.g., data chunk dl 708) and chunk S2 in kid.kd. Client cO 602 encrypts 718 data chunk dl with key kl and sends 720 the encrypted chunk with metadata (e.g., including fingerprint S I and key ID kid.kl ) to storage 504. Client cO 602 encrypts 722 data chunk d2 with key kd and sends 724 the encrypted chunk with metadata (e.g., including fingerprint S2 and key ID kid.kd) to storage 504. This latter operation backports chunk S2 from key ID kid.kO to key ID kid.kd, allowing the chunk to be decrypted by key users associated with key kO and key users associated with key kl.

[0178] Storage 504 stores the chunks and updates chunk metadata table 528. Row 4 of table 528 shows the update for S2 to include information that the chunk was also written by client cO 602 with kid.kl and has a reference count of 1 in that key ID and changes the encryption key ID to kid.kd. This location is shown as 15, such as a copy for a write-once system. In a direct overwrite system, this location remains at 14. Row 5 is new metadata indicating that chunk S I was written by client cO 602 encrypted with a secret key associated with key ID kid.kl, accessed with kid.kl, and has a reference count of 1 in that key and is stored in location 16. ​

[0179] Figure 8 A high-level architecture is depicted according to various configurations. According to the present application, the architecture 800 can be implemented in various configurations in any of the environments described in 1-9 and 11. Of course, more or less elements than those specifically described in 1-9 and 11 can be included in the architecture 800, as would be understood by one of skill in the art upon reading the present descriptions. Figures 1-7 The architecture 800 depicts an exemplary implementation by client c0602 adding further data to storage 504 with key k2 without allowing deduplication against plaintext. In this architecture, it is assumed that operations follow the operations shown in 1-9. Figure 8

[0180] If deduplication with plaintext is not allowed, data privacy is increased. The architecture 800 depicts an exemplary implementation by client c0602 adding further data to storage 504 with key k2 without allowing deduplication against plaintext. In this architecture, it is assumed that operations follow the operations shown in 1-9. Figure 5

[0181] Client cO key group gO information 802 includes a dct flag (dedup against plaintext) set to 0 to indicate that data written with key IDs belonging to group gO can be deduplicated, but not against data written in plaintext. Client cO key group gO information 802 includes data keys kO and k2 with corresponding key IDs kid.kO and kid.k2. Client cO key group gO information 802 includes a dedup key ke with key ID kid.ke. Client cO key group gO information 802 includes a fingerprint key ks with key ID kid.ks, where deduplication against plaintext is not allowed.

[0182] In encrypted data, the meta generator 606 resides in client cO 602. The meta generator 606 is similar to the meta generator 316 described in detail above with reference to 1-9. Figures 3-4

[0183] ​​​Client c0 602 writes data object M0 804 with data m0. Operation 806 includes sending data object M0 804 with data m0 to meta-generator 606. Meta-generator 606 identifies 3 data chunks 808 (e.g., data chunk d3 810, data chunk d2 812, and data chunk d0 814) as shown. Data chunk d3 810 is associated with fingerprint S3 and data d3. Data chunk d2 812 is associated with fingerprint S2 and data d2. Data chunk d0 814 is associated with fingerprint S0 and data d0. At 816, the fingerprints are encrypted with secret fingerprint key ks. S0 is encrypted into S0', S2 into S2', and S3 into S3'. Metadata 818 includes key group identifier g1 and key ID kid.k2 (e.g., indicating that the data was encrypted with the secret key associated with key ID kid.k2) and encrypted fingerprints S0', S2', and S3' are sent 820 to storage 504. Key group identifier g1 allows storage 504 to determine that the dedup key ID is kid.ke from the previously communicated group information (e.g., client c0 key group g1 information 802, also see request 358 referenced in Figure 3

[0184] Storage 504 checks Figure 5 current chunk metadata table 528 of Figure 5 S0', S2', and S3' are new fingerprints. The fingerprint of chunk S3 (e.g., data chunk d3 810) is encrypted into S3', and this value is passed to storage 504. Thus, even if the data content d3 matches chunk S3 stored in table 528 and written by client c2 (see

[0185] Without allowing plaintext dedup, storage 504 does not compare the fingerprints of plaintext chunks. The dedup operation of storage 504 limits the fingerprint comparison to those key IDs belonging to the key group of the operation. If fingerprints from different groups match, then dedup is still not performed (e.g., typically, for cryptographic hash fingerprints, the probability of such a collision is so small as to be infeasible to detect). This reduces the table space that needs to be searched for fingerprint matches. For example, even if the plaintext fingerprints of data chunk 516 Figure 5 ) and data chunk 810 match, the metadata fingerprints S3' and S3 do not.

[0186] ​The storage 504 stores the blocks and updates the block metadata table 528. Row 3 is new metadata indicating that the block S0' was written by the client c0 602, encrypted with a key associated with kid.k2, is accessible with kid.k2, has a reference count of 1 in the key ID kid.k2, and is stored in location 13. Row 4 shows similar information for the block S2'. Row 5 shows similar information for the block S3'.

[0187] The storage 504 stores the blocks and updates the block metadata table 528. Row 3 is new metadata indicating that the block S0' was written by the client c0 602, encrypted with a key associated with kid.k2, is accessible with kid.k2, has a reference count of 1 in the key ID kid.k2, and is stored in location 13. Row 4 shows similar information for the block S2'. Row 5 shows similar information for the block S3'.

[0188] Figure 9 A high-level architecture is depicted in accordance with various configurations. In accordance with the present application, the architecture can be implemented in various configurations in any of the environments described in Figures 1-8 The architecture 900 can be implemented in various configurations in any of the environments described in 1-11. Of course, as will be appreciated by one of ordinary skill in the art upon reading the present descriptions, more or less elements than those specifically described herein can be included in the architecture 900. Figure 9 The elements specifically described in 1-11 can be included in the architecture 900 in various configurations.

[0189] The architecture 900 depicts an exemplary implementation of the client c0 602 adding other data in the key k3. The key k3 is part of the key group g1 (see Figure 8 The client key group information 802 in 1-11). In this architecture, it is assumed that the operations follow those shown in Figure 8

[0190] ​Client c0 602 writes a data object Ml 902 with data ml. Operation 904 includes sending the data object Ml 902 with data ml to the meta-generator 606. The meta-generator 606 identifies 2 data chunks 906 as shown (e.g., data chunk dl 908 and data chunk d2 910). Data chunk dl 908 is associated with fingerprint S1 and data dl. Data chunk d2 910 is associated with fingerprint S2 and data d2. At 912, the fingerprints are encrypted with the secret fingerprint key ks. S1 is encrypted into S1'and S2 is encrypted into S2'. The metadata 914 includes the key group identifier gl and the key ID kid.k3 (e.g., indicating that the data was encrypted with the secret key associated with key ID kid.k3) and the encrypted fingerprints S1'and S2' are sent 916 to the storage 504. The key group identifier gl allows the storage 504 to determine that the deduplication key ID is kid.ke from the previously communicated group information (e.g., client c0 key group gl information 802, see also request 358 referenced in Figure 3

[0191] The storage 504 checks the chunk metadata table 528 of Figure 8 and determines that S1'is a new fingerprint and S2' already exists in kid.k2. The chunk S2' can be deduplicated using the group gl deduplication key ke. The storage 504 requests 918 the client c0 602 to send the chunk S1'in kid.k3 and the chunk S2' in kid.ke. For example, the client c0 602 encrypts 920 the data chunk dl with key k3 and sends 922 the encrypted chunk with metadata including the encrypted fingerprint S1'and the key ID kid.k3 to the storage 504. Similarly, the client c0 602 encrypts 924 the data chunk d2 with key ke and sends 926 the encrypted chunk with metadata including the encrypted fingerprint S2' and the key ID kid.ke to the storage 504. This latter operation backports the chunk S2' from key ID kid.k2 to key ID kid.ke, thus allowing the chunk to be decrypted by both key users associated with key kl and key users associated with key k2.

[0192] The storage 504 stores the chunks and updates the chunk metadata table 528. Row 4 shows the update for S2' includes information that the chunk was also written by client c0 602 in kid.k2, the reference count in key ID kid.k2 is 1, the encryption key ID is changed to kid.ke, and is now stored in location l6. Row 6 is the new metadata indicating that the chunk S1'is written by client c0 602, encrypted with the key associated with kid.k3, is accessible with kid.k3, the reference count in key ID kid.k3 is 1, and is stored in location l7.

[0193] ​As described in the preceding architecture, it is beneficial to maintain a reference count for each key ID in each block. Therefore, a non-zero reference count in a given key ID is required to read a data block. If a block has a positive reference count, this count is decremented whenever a block reference is removed, until the counter reaches 0. Once the reference count reaches 0, the key ID can be removed from the block's list of access key IDs. This removal occurs simultaneously with non-zero reference counts in the other key IDs.

[0194] Figure 10 A high-level architecture based on various configurations is described. According to the present invention, it is possible to... Figures 1-9 Architecture 1000 can be implemented in various configurations in any environment described in section 11. Of course, as those skilled in the art will understand upon reading this description, architecture 1000 may include more than... Figure 10 The number of components described in the text may be more or less than the number of components.

[0195] Architecture 1000 illustrates the use of data object M1 (see...) Figure 9 An exemplary read operation is described below. Data object M1 is written to key ID kid.k3. Operation 1002 includes client c0 602 issuing a read request to memory 504 for M1 tagged with group g1 and kid.k3. Memory 504 determines that blocks S1' and S2' include object M1 in a manner known to those skilled in the art (e.g., from a list of object blocks). Memory 504 reads S1' and S2' in operation 1004. Block S1' and associated metadata 1006 and block S2' and associated metadata 1008 are sent to client c0 602, respectively, for deduplication. Block S1' is decrypted using data key k3 determined from key ID kid.k3 in the metadata. Block S2' is decrypted using deduplication key ke determined from key ID kid.k3 in the metadata. Fingerprint 1018 is decrypted using the fingerprint key ks determined from the key group g1 strategy (the fingerprint path to decryption 1018 is not shown). The decrypted fingerprints 1020S1 and S2 can be compared with the fingerprints of decrypted data blocks d1 1024 and d2 1030 to check for data corruption. The decrypted blocks 1024 and 1030 are sent to block merger 1028 and 1026 respectively. The merged block 1022 is a data object M1 with data m1. Data object M1 is returned as 1034 to complete the read operation.

[0196] Now see Figure 11 It shows a flowchart of method 1100 according to one aspect. In various aspects, it is possible to use the present invention... Figures 1-10Method 1100 may be performed in any environment depicted herein. Of course, as those skilled in the art will understand upon reading this specification, method 1100 may include more than Figure 11 The specific operations described in the text may include more or fewer operations.

[0197] Each step of method 1100 can be performed by any suitable component of the operating environment. For example, in various aspects, method 1100 can be performed partially or entirely by a computer or some other device having one or more processors therein. One or more steps of method 1100 can be performed in any device using a processor (e.g., processing circuitry, chips, and / or modules implemented in hardware and / or software and preferably having at least one hardware component). Illustrative processors include, but are not limited to, central processing units (CPUs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), combinations thereof, or any other suitable computing devices known in the art.

[0198] like Figure 11 As shown, method 1100 includes operation 1102. Operation 1102 includes sending key group information to a storage system. The key group information can be sent to the storage system in any manner known in the art. The key group information preferably includes key ID information of client data keys in the key group. The client data keys belonging to the key group can be client secret keys or client deduplication keys. The key group information preferably does not include the actual client secret keys or the actual client deduplication keys. In a preferred method, the key group includes client secret keys used for encrypting data. The client secret keys are associated with the key group. In a preferred method, the key ID is not easily predicted and / or determined from the key itself. Random values ​​and / or values ​​encrypted with a separate key can be used to generate the key ID. Any other techniques for generating key IDs can be used.

[0199] In various aspects, the client data key enables deduplication of data blocks encrypted with any client data key in the key set. For example, client data encrypted with a client secret key belonging to the key set can be deduplicated, compared to other data encrypted with that client secret key or data encrypted with any other client secret key in the key set. The client secret key provides the ability to perform deduplication across the set of keys in the key set, and the storage system cannot access the client secret key. In a preferred approach, the key set includes a secret deduplication key (e.g., a client deduplication key) to encrypt deduplicated blocks, thereby enabling deduplication across the client secret key set.

[0200] Operation 1104 includes generating deduplication information. The deduplication information includes a fingerprint associated with a client data block. Preferably, the deduplication information includes deduplication opportunities. In some methods, a meta-generator identifies and / or computes deduplication fingerprints (e.g., rolling minimum hash, cryptographic hash, etc.). In other methods, other components known in the art can identify and / or compute deduplication fingerprints to identify deduplication opportunities. In some methods, metadata associated with data deduplication opportunities (e.g., deduplication information) is generated, including pointers, block lengths, block fingerprints, etc. In a preferred method, the deduplication information (e.g., at least including block fingerprints) is generated before the data and / or data blocks to be sent to the storage system are encrypted, as described below.

[0201] The deduplication information may include operational details of the deduplication method that would be understood by one of ordinary skill in the art upon reading this disclosure. In one configuration, for block deduplication, the deduplication information may include the block start position, block length, block signature, similarity information, etc. In one method, the block fingerprint is a cryptographic hash that can be computed in any manner known in the art.

[0202] In some alternative methods, portions of the data deduplication information, such as block fingerprints, can be encrypted using a client-secret fingerprint key. The client-secret fingerprint key is preferably not used by the storage system. As will be understood by those skilled in the art upon reading this disclosure, a client-secret fingerprint key encryptor can use the client-secret fingerprint key to encrypt fingerprints. In a preferred aspect, the fingerprint is encrypted using the client-secret fingerprint key before sending the deduplication information to the storage system.

[0203] Operation 1106 includes encrypting a data block using one of the client data keys from the client data keys. In a preferred method, the data block is encrypted before being sent to the storage system. In various methods, the data block is encrypted using one of the client secret keys from the key set. In a preferred aspect, the corresponding decryption key for encrypting the data block is not available to the storage system.

[0204] Operation 1108 includes sending the deduplication information to the storage system for use by the storage system in the deduplication process. The deduplication information can be sent to the storage system in any manner known in the art. In a preferred method, the storage system uses the deduplication information to identify data blocks for deduplication based on the deduplication information. For data blocks that are not identified for deduplication, the storage system can request client data associated with these data blocks. As will be appreciated by one of ordinary skill in the art in view of the present disclosure, the client data associated with these data blocks is encrypted in one of the client secret keys. The client data associated with the data blocks preferably includes corresponding key ID information that identifies the client secret key used to encrypt the data blocks. In various methods, a request for client data associated with data blocks that are not identified by the storage system for deduplication can be received from the storage system. The client data associated with data blocks encrypted in a client secret key can be sent to the storage system in response to the request in any manner known in the art.

[0205] Operation 1110 includes sending the encrypted data blocks to the storage system. In some methods, the encrypted data blocks are sent to the storage system in response to a request from the storage system. In other methods, the encrypted data blocks can be sent to the storage system in response to a request from the client or from another source known in the art. The request can include a request for data blocks and / or associated data according to any of the aspects described in detail above. The request preferably also requests key ID information that identifies the client data key used to encrypt the data blocks. The encrypted data blocks can be sent to the storage system in any manner known in the art.

[0206] In some methods, a request for client data associated with data blocks encrypted in a client secret key that are stored in the storage system can be received from the storage system. The request preferably includes that the client data associated with the data blocks can be sent to the storage system, the client data encrypted in a client deduplication key. The client data can be sent to the storage system in response to the request in any manner known in the art in accordance with the present disclosure.

[0207] The storage system performs deduplication in accordance with the various aspects described herein. The storage system can deduplicate data blocks stored in the storage system with respect to blocks of client data, where deduplication information is used to select data blocks for deduplication. In one approach, a deduplication manager locates and / or identifies duplicate data based on the deduplication information. For chunked deduplication, the deduplication information is used with block metadata to identify duplicate data blocks. For example, the deduplication manager compares the pointers, fingerprints, block lengths, etc. from the decrypted deduplication information to any pointers, fingerprints, block lengths, etc. stored in a block metadata database on the storage system. The deduplication manager performs any relevant deduplication operations as will be appreciated by one of ordinary skill in the art upon reading the present disclosure. In a preferred approach, the storage system stores metadata for data blocks, where the metadata includes a key identifier that uniquely identifies a key used to encrypt the data block.

[0208] In various approaches, encrypted data stored in the storage system is retrieved in response to a request for client data from a client connected to the storage system. As will be appreciated by one of ordinary skill in the art, the request can include multiple requests for client data. Further, the request can include a request (or multiple requests) for client data in more than one client data key. The storage system preferably identifies the data blocks associated with the data request. The approach for retrieving encrypted data from the storage system includes retrieving metadata associated with the data blocks. The metadata preferably includes key ID information. The storage system retrieves encrypted client data associated with the data request based on the metadata in accordance with any of the approaches described herein. The encrypted client data can be sent to the client in any manner known in the art as will be appreciated by one of ordinary skill in the art upon reading the present disclosure. In some approaches, the encrypted client data can be received as data blocks. One or more of the data blocks can be encrypted with a client secret key and one or more different data blocks can be encrypted with a client deduplication key.

[0209] In a preferred approach, the client secret key never leaves the client side. In this preferred approach, the storage system never sees the encrypted data in plaintext at any point in the data transfer. In this preferred approach, the client data key is not shared with the storage system, any other client, any other storage system, etc.

[0210] The present application can be a system, a method, and / or a computer program product at any possible technical detail level of integration. The computer program product can include a computer readable storage medium (or media) having computer readable program instructions thereon for causing a processor to carry out aspects of the present application.

[0211] The computer readable storage medium can be a tangible device that can retain and store instructions for use by an instruction execution device. The computer readable storage medium can be, for example, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of the computer readable storage medium includes the following: a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanically encoded device such as punch-cards or raised structures in a groove having instructions recorded thereon, and any suitable combination of the foregoing. A computer readable storage medium, as used herein, is not to be construed as being transitory signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission media (e.g., light pulses passing through a fiber-optic cable), or electrical signals transmitted via a wire cable.

[0212] Computer readable program instructions described herein can be downloaded to respective computing / processing devices from a computer readable storage medium or to an external computer or external storage device via a network, for example, the Internet, a local area network, a wide area network and / or a wireless network. The network can comprise copper transmission cables, optical transmission fibers, wireless transmission, routers, firewalls, switches, gateway computers and / or edge servers. A network adapter card or network interface in each computing / processing device receives computer readable program instructions from the network and forwards the computer readable program instructions into the computing / processing device for storage in a computer readable storage medium within the respective computing / processing device.

[0213] Computer readable program instructions for carrying out operations of the present application can be assembly instructions, instruction-set architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state-setting data, configuration data for an integrated circuit, or source code or object code written in any combination of one or more programming languages, including an object oriented programming language such as Smalltalk, C++ or the like, and procedural programming languages, such as the "C" programming language or similar programming languages. The computer readable program instructions can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example, programmable logic circuitry, field-programmable gate arrays (FPGA), or programmable logic arrays (PLA) can execute the computer readable program instructions by utilizing state information of the computer readable program instructions to personalize the electronic circuitry, in order to perform aspects of the present application.

[0214] The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer implemented process such that the instructions which execute on the computer, other programmable apparatus, or other device implement the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0215] These computer readable program instructions can be provided to a processor of a computer, or other programmable data processing apparatus, to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks. These computer readable program instructions can also be stored in a computer readable storage medium that can direct a computer, a programmable data processing apparatus, and / or other devices to function in a particular manner, such that the computer readable storage medium having instructions stored therein comprises an article of manufacture including

[0216] The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer implemented process such that the instructions which execute on the computer, other programmable apparatus, or other device implement the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0217] The computer program product of the present application can be a computer program product which implements the methods of the present application on a data carrier, such as a diskette, a memory stick, a CD-ROM, a USB memory, a hard disk, etc. The computer program product can also be a data signal embodied in a carrier wave, such as a digital message, a downloaded computer file, such as a finite memory, a physical transmission medium, such as a wired network or a wireless network. As such, the computer program product can comprise a computer-readable medium, having stored thereon, a computer program comprising computer program code means adapted to perform the methods of the present application when the computer program product is executed on a computer with a processor and a memory.

[0218] Furthermore, a system according to various embodiments can comprise a processor and logic integrated with and / or integrable with the processor, the logic configured to perform one or more of the process steps described herein. The processor can be of any configuration as described herein, e.g. a discrete processor or a processing circuit comprising many components such as processing hardware, memory, I / O interface, etc. By integrated with it is meant that the processor has logic embedded with it as hardware logic, such as an application-specific integrated circuit (ASIC), FPGA, etc. By executable by the processor it is meant that the logic is hardware logic; software logic, e.g. part of firmware, part of an operating system, part of an application, etc.; or some combination of hardware and software logic, which is accessible to the processor and configured to cause the processor to perform some function when executed by the processor. The software logic can be stored on local and / or remote memory of any memory type, as is well known in the art. Any processor known in the art can be used, such as a software processor module and / or a hardware processor, such as an ASIC, FPGA, central processing unit (CPU), integrated circuit (IC), graphics processing unit (GPU), etc.

[0219] Clearly, various features of the foregoing system and / or method can be combined in any manner, resulting in multiple combinations from the descriptions presented above.

[0220] It is also to be understood that embodiments of the present application can be provided in the form of services deployed on behalf of customers, to provide services on demand.

[0221] The description of the various embodiments of the present application have been presented for purposes of illustration, but is not intended to be exhaustive or limited to the embodiments disclosed. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope of the described embodiments. The terminology used herein was chosen to best explain the principles of the embodiments, the practical application, or technical improvement over technologies found in the marketplace, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A computer-implemented method comprising: sending key group information to a storage system, wherein the key group information includes key ID information that identifies a client secret key and a client dedup key in the key group; generating dedup information, wherein the dedup information includes a fingerprint associated with a client data block; sending the dedup information to the storage system for use by the storage system in a deduplication process; in response to a request from the storage system and a determination that a fingerprint included in the sent dedup information is a new fingerprint, sending a data block encrypted with the client secret key to the storage system to store the data block encrypted with the client secret key and key ID information of the client secret key in the storage system; and in response to a request from the storage system and a determination that a fingerprint included in the sent dedup information is an existing fingerprint, sending a data block encrypted with the client dedup key to the storage system to update a data block stored in the storage system that is associated with the existing fingerprint to be encrypted with the client dedup key and to update key ID information of the client secret key for the data block to key ID information of the client dedup key.

2. The computer-implemented method of claim 1, further comprising: encrypting the fingerprint with a client secret fingerprint key prior to sending the dedup information to the storage system.

3. The computer-implemented method of claim 1, further comprising: requesting client data from the storage system; and receiving the client data as the data block, wherein one or more of the data blocks are encrypted with the client secret key and one or more other of the data blocks are encrypted with the client dedup key.

4. A computer-implemented method comprising: receiving key group information at a storage system, wherein the key group information includes key ID information that identifies a client secret key and a client dedup key in the key group; receiving dedup information, wherein the dedup information is accessible by the storage system for use in performing operations thereon, wherein the dedup information includes a fingerprint associated with an encrypted client data block; in a case where the fingerprint included in the dedup information is determined to be a new fingerprint, requesting the client for a data block encrypted with the client secret key to store the data block encrypted with the client secret key received from the client and key ID information of the client secret key; and in a case where the fingerprint included in the dedup information is determined to be an existing fingerprint, requesting the client for a data block encrypted with the client dedup key to update a data block associated with the existing fingerprint to be encrypted with the client dedup key and to update key ID information of the client secret key for the data block to key ID information of the client dedup key. ​ 5. The computer-implemented method of claim 4, further comprising: in response to a request for client data from the client, identifying the data chunk associated with the request; retrieving metadata associated with the data chunk, wherein the metadata includes the key ID information; retrieving the encrypted client data associated with the request based on the metadata; and sending the encrypted client data to the client.

6. The computer-implemented method of claim 5, wherein, encrypting the fingerprint with a client secret fingerprint key, wherein the client secret fingerprint key is not available to the storage system.

7. A system for multi-key encrypted data deduplication, comprising: a processor; and logic integrated with the processor, executable by the processor, or both, the logic being configured to: send key group information to a storage system, wherein the key group information includes key ID information identifying a client secret key and a client dedup key in the key group; generate dedup information, wherein the dedup information includes a fingerprint associated with a client data chunk; send the dedup information to the storage system for use by the storage system in a deduplication process; and in response to a request from the storage system and a determination that the fingerprint included in the sent dedup information is a new fingerprint, send a data chunk encrypted with the client secret key to the storage system to store the data chunk encrypted with the client secret key and key ID information for the client secret key in the storage system; and in response to a request from the storage system and a determination that the fingerprint included in the sent dedup information is an existing fingerprint, send a data chunk encrypted with the client dedup key to the storage system to update a data chunk stored in the storage system associated with the existing fingerprint to a data chunk encrypted with the client dedup key and update key ID information for the client secret key of the data chunk to key ID information for the client dedup key.

8. The system of claim 7, further comprising logic configured to encrypt the fingerprint with a client secret fingerprint key prior to sending the dedup information to the storage system.

9. A system for multi-key encrypted data deduplication, comprising: a processor; and logic integrated with the processor, executable by the processor, or both, the logic being configured to: receive key group information at a storage system, wherein the key group information includes key ID information identifying a client secret key and a client dedup key in the key group; receive dedup information, wherein the dedup information is accessible by the storage system for use in performing operations thereon, wherein the dedup information includes a fingerprint associated with an encrypted client data chunk; identify a data chunk for deduplication based on the dedup information; in response to determining that the fingerprint included in the deduplication information is a new fingerprint, requesting from the client a data block encrypted with the client secret key to store in the storage system the data block encrypted with the client secret key and the key ID information of the client secret key received from the client; in response to determining that the fingerprint included in the deduplication information is an existing fingerprint, requesting from the client a data block encrypted with the client deduplication key to update a data block associated with the existing fingerprint to the data block encrypted with the client deduplication key and to update the key ID information of the client secret key of the data block to the key ID information of the client deduplication key.

10. The system of claim 9, further comprising logic configured to, in response to a request for client data from the client, identify the data block associated with the request; obtaining metadata associated with the data block, wherein the metadata includes the key ID information; retrieve the encrypted client data associated with the request based on the metadata; and send the encrypted client data to the client.

11. The system of claim 9, wherein, encrypting the fingerprint with a client secret fingerprint key, wherein the client secret fingerprint key is not available to the storage system.

12. A computer program product, the computer program product comprising: one or more computer readable storage media, and program instructions collectively stored on the one or more computer readable storage media, which program instructions, when executed by a processor, perform the method of any of claims 1-6.

Images