Opaque Encryption for Data Deduplication

Abstract

A computer-implemented method includes receiving deduplication information at a storage system. The deduplication information is accessible to the storage system for performing operations thereon. The deduplication information includes signatures associated with portions of client data. The method also includes receiving client data encrypted with a client key. The client key is not available to the storage system. The method includes deduplicating data chunks stored in the storage system against chunks of client data, where the client data chunks are selected from the client data for deduplication using the deduplication information.

Description

Technical Field

[0001] The present invention relates to data deduplication, and more particularly, to opaque encryption for data deduplication in cloud storage systems and networks. Background Art

[0002] Traditional data reduction techniques such as deduplication and / or compression do not provide meaningful reduction when applied to encrypted data. In cases where different encryption algorithms prevent the conventional deduplication process from identifying duplicate data chunks, the deduplication breakdown of multiple data sets encrypted with respective unique encryption keys. Traditional data reduction techniques also do not provide sufficient data privacy between the client and the storage system.

[0003] For example, a known Bring Your Own Key (BYOK) encryption technique involves a multi-party trust system. Although all data reduction functions can be provided by a storage system that has access to all data, traditional BYOK systems do not have data privacy between the storage system and the client because the storage system can access the client key. A third-party key service can also access the shared encryption key used to encrypt the client data. For this form of BYOK encryption, data privacy only exists between users.

[0004] Traditional static encryption uses a key known to the storage system to encrypt unencrypted input data. The storage system can decrypt all data and perform deduplication on all data in the system. However, static encryption does not provide data privacy.

[0005] Traditional full client-side encryption encrypts data with a key unknown to the storage system. The storage system only performs deduplication on data encrypted with a common key. Full client-side deduplication provides relatively high data privacy but hinders deduplication efficiency.

[0006] Therefore, there is a need in the art to solve the above problems. Summary of the Invention

[0007] From a first aspect, the present invention provides a computer-implemented method, comprising: receiving, at a storage system, deduplication information, wherein the deduplication information is accessible to the storage system for performing operations thereon, and wherein the deduplication information includes a signature associated with a portion of client data; receiving client data encrypted with a client secret key, wherein the client secret key is not available to the storage system; and performing deduplication of data chunks stored in the storage system against client data chunks, wherein the client data chunks are selected from the client data for deduplication using the deduplication information.

[0008] On the other hand, the present invention provides a computer program product for opaque encryption. The computer program product includes a computer-readable storage medium having program instructions included therein. The program instructions are executable by a computer to cause the computer to: receive deduplication information at a storage system, where the deduplication information is accessible to the storage system for performing operations thereon, and where the deduplication information includes a signature associated with a portion of client data; receive client data encrypted with a client key, where the client key is not available to the storage system; and perform deduplication on data chunks stored in the storage system against chunks of the client data, where the client data chunks are selected from the client data for deduplication using the deduplication information.

[0009] On the other hand, the present invention provides a system including: a processor; and logic integrated with, executable by, or integrated and executable with the processor, the logic being configured to: receive deduplication information at a storage system, where the deduplication information is accessible to the storage system for performing operations thereon, and where the deduplication information includes a signature associated with a portion of client data; receive client data encrypted with a client key, where the client key is not available to the storage system; and perform deduplication on data chunks stored in the storage system against chunks of the client data, where the client data chunks are selected from the client data for deduplication using the deduplication information.

[0010] On the other hand, the present invention provides a computer-implemented method including: generating deduplication information, where the deduplication information includes a signature associated with a portion of client data; encrypting the deduplication information with a shared key; encrypting the client data with a client key; sending the deduplication information and the client data to a storage system for deduplication; requesting the client data from the storage system; and receiving the client data as data chunks, where one or more of the data chunks are encrypted with the client key and one or more of the data chunks are encrypted with the shared key.

[0011] On the other hand, the present invention provides a system including: a processor; and logic integrated with, executable by, or integrated and executable with the processor, the logic being configured to: generate deduplication information, where the deduplication information includes a signature associated with a portion of client data; encrypt the deduplication information with a shared key; encrypt the client data with a client key; send the deduplication information and the client data to a storage system for deduplication; request the client data from the storage system; and receive the client data as data chunks, where one or more of the data chunks are encrypted with the client key and one or more of the data chunks are encrypted with the shared key.

[0012] On the other hand, the present invention provides a computer program product for data deduplication in a cloud storage system and network, the computer program product including a computer-readable storage medium readable by a processing circuit and storing instructions for execution by the processing circuit to perform a method for performing the steps of the present invention.

[0013] On the other hand, the present invention provides a computer program stored on a computer-readable medium and loadable into the internal memory of a digital computer, the computer program including software code portions for performing the steps of the present invention when the program runs on the computer.

[0014] A computer-implemented method according to the method includes receiving deduplication information at a storage system. The deduplication information is accessible to the storage system for performing operations thereon. The deduplication information includes signatures associated with portions of client data. The method further includes receiving client data encrypted with a client key. The client key is not available to the storage system. The method includes deduplicating data chunks stored in the storage system against chunks of the client data, wherein the client data chunks are selected from the client data for deduplication using the deduplication information. The method provides improved data privacy, enhanced data reduction, and reduced client overhead as compared to conventional techniques.

[0015] The method may optionally include: receiving data encrypted with a second client key and deduplicating the data chunks, wherein one or more of the data chunks are encrypted with a first client key and one or more of the data chunks are encrypted with a second client key. The foregoing method enables deduplication of data chunks encrypted with different keys and provides data privacy between users and between users and the storage system.

[0016] According to one aspect, a computer program product for opaque encryption includes a computer-readable storage medium having program instructions included therein. The program instructions are executable by a computer to cause the computer to perform the foregoing method.

[0017] According to one aspect, a system includes a processor and logic integrated with, executable by, or integrated with and executable by the processor. The logic is configured to perform the above method.

[0018] A computer-implemented method according to the method includes generating deduplication information. The deduplication information includes signatures associated with portions of client data. The method includes encrypting the deduplication information with a shared key and encrypting the client data with a client key. The method further includes sending the deduplication information and the client data to a storage system for deduplication. The method includes requesting the client data from the storage system and receiving the client data as data chunks, where one or more of the data chunks are encrypted with the client key and one or more of the data chunks are encrypted with the shared key. The method provides improved data privacy, enhanced data reduction, and reduced client overhead as compared to conventional techniques.

[0019] The method optionally includes requesting the shared key from the storage system. The method enables the encryption of the deduplication information with an encryption key accessible to the storage system while maintaining data privacy between the storage system and the client.

[0020] According to one scenario, a system includes a processor and logic integrated with, executable by, or integrated with and executable by the processor. The logic is configured to perform the above method.

[0021] Other aspects and configurations of the present invention will become apparent from the following detailed description, which, when taken in conjunction with the drawings, illustrate the principles of the present invention by way of example. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The present invention will now be described by way of example only with reference to the preferred embodiments as shown in the following drawings:

[0023] Figure 1 A cloud computing node according to an embodiment of the present invention is shown.

[0024] Figure 2 A cloud computing environment according to a configuration of the present invention is shown.

[0025] Figure 3 An abstract model layer according to a configuration of the present invention is described.

[0026] Figure 4 is a high-level architecture according to a configuration of the present invention.

[0027] Figure 5 is a high-level architecture according to a configuration of the present invention.

[0028] Figure 6 is a high-level architecture according to a configuration of the present invention.

[0029] Figure 7 is a high-level architecture according to a configuration of the present invention.

[0030] Figure 8 is a flow chart of a method according to a configuration of the present invention.

[0031] Figure 9 is a flow chart of a method according to a configuration of the present invention. DETAILED DESCRIPTION

[0032] The following description is made for the purpose of illustrating the general principles of the invention and is not meant to limit the inventive concepts claimed herein. Further, specific features described herein may be used in combination with other described features in each of the various possible combinations and permutations.

[0033] Unless otherwise specifically defined herein, all terms are to be given their broadest possible interpretations, including the meanings implied from the specification and as understood by those skilled in the art and / or as defined in dictionaries, treatises, etc.

[0034] It must also be noted that, as used in the specification and the appended claims, the singular forms "a," "an," and "the" include plural referents unless otherwise specified. It should also be understood that when the terms "comprises" and / or "comprising" are used in this specification, they specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or combinations thereof.

[0035] The following description discloses several configurations of data deduplication with opaque encryption.

[0036] In one general configuration, a computer-implemented method includes receiving deduplication information at a storage system. The deduplication information is accessible to the storage system for performing operations thereon. The deduplication information includes a signature associated with a portion of client data. The method also includes receiving client data encrypted with a client key. The client key is not available to the storage system. The method includes deduplicating data chunks stored in the storage system against chunks of the client data, wherein the chunks of the client data are selected from the client data for deduplication using the deduplication information.

[0037] In another general configuration, a computer program product for opaque encryption includes a computer-readable storage medium having program instructions embodied therein. The program instructions are executable by a computer to cause the computer to perform the aforementioned method.

[0038] In yet another general configuration, a system includes a processor and logic integrated with the processor, executable by the processor, or integrated with the processor and executable by the processor. The logic is configured to perform the above method.

[0039] In one general configuration, a computer-implemented method includes generating deduplication information. The deduplication information includes signatures associated with portions of client data. The method includes encrypting the deduplication information with a shared key and encrypting the client data with a client key. The method also includes sending the deduplication information and the client data to a storage system for deduplication. The method includes requesting client data from the storage system and receiving the client data as data chunks, where one or more of the data chunks are encrypted with the client key and one or more of the data chunks are encrypted with the shared key.

[0040] In another general configuration, a system includes a processor and logic integrated with, executable by, or integrated with and executable by the processor. The logic is configured to perform the method described above.

[0041] It should be understood that although this disclosure includes a detailed description of cloud computing, the implementation of the teachings recited herein is not limited to a cloud computing environment. Instead, the configurations of the present invention are capable of being implemented in conjunction with any other type of computing environment now known or later developed.

[0042] Cloud computing is a model for service delivery that enables convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, network bandwidth, servers, processing, memory, storage, applications, virtual machines, and services) that can be rapidly provisioned and released with minimal management effort or interaction with the provider of the service. The cloud model can include at least five characteristics, at least three service models, and at least four deployment models.

[0043] The characteristics are as follows:

[0044] On-demand self-service: Cloud consumers can unilaterally and automatically provision computing capabilities, such as server time and network storage, as needed without human interaction with the provider of the service.

[0045] Broad network access: Capabilities are available over a network and accessed through standard mechanisms that facilitate the use of heterogeneous thin client platforms or thick client platforms (e.g., mobile phones, laptop computers, and PDAs).

[0046] Resource pooling: The provider's computing resources are pooled to serve multiple consumers using a multi-tenant model, where different physical and virtual resources are dynamically assigned and reassigned as needed. There is a sense of location independence because consumers generally do not have control over or knowledge of the exact location of the resources provided, but may be able to specify a location at a higher level of abstraction (e.g., country, state, or data center).

[0047] Rapid elasticity: Capable of providing capacity quickly and elastically, automatically scaling down rapidly and releasing quickly in some cases to scale up quickly. To the consumer, the available capacity often appears unlimited and can be purchased in any quantity at any time.

[0048] Measured service: The cloud system automatically controls and optimizes resource use by leveraging metering capabilities at some abstract level appropriate to the service type (e.g., storage, processing, bandwidth, and active user accounts). Resource use can be monitored, controlled, and reported, providing transparency for both the provider and consumer of the utilized service.

[0049] The service models are as follows:

[0050] Software as a Service (SaaS): The capabilities provided to the consumer are to use the provider's applications running on the cloud infrastructure. The applications can be accessed from different client devices through a thin client interface such as a web browser (e.g., web-based email). The consumer does not manage or control the underlying cloud infrastructure, including the network, servers, operating systems, storage, or even the individual application capabilities, with the possible exception of limited user-specific application configuration settings.

[0051] Platform as a Service (PaaS): The capabilities provided to the consumer are to deploy applications created by the consumer or acquired using programming languages and tools supported by the provider onto the cloud infrastructure. The consumer does not manage or control the underlying cloud infrastructure, including the network, servers, operating systems, or storage, but has control over the deployed applications and possibly the application hosting environment configuration.

[0052] Infrastructure as a Service (IaaS): The capabilities provided to the consumer are to provide processing, storage, networks, and other basic computing resources where the consumer can deploy and run arbitrary software, which may include operating systems and applications. The consumer does not manage or control the underlying cloud infrastructure but has control over the operating systems, storage, deployed applications, and possibly limited control over the selected networking components (e.g., host firewall).

[0053] The deployment models are as follows:

[0054] Private cloud: The cloud infrastructure is for the exclusive use of an organization. It can be managed by the organization or a third party and can exist on-premises or off-premises.

[0055] Community cloud: The cloud infrastructure is shared by several organizations and supports a specific community that shares concerns (e.g., missions, security requirements, policies, and compliance considerations). It can be managed by the organization or a third party and can exist on-premises or off-premises.

[0056] Public cloud: Makes cloud infrastructure available to the general public or a large industry group and is owned by an organization that sells cloud services.

[0057] Hybrid cloud: Cloud infrastructure is a combination of two or more clouds (private, community, or public) that remain unique entities but are bound together by standardized or proprietary technologies that enable data and application portability (e.g., cloud bursting for load balancing between clouds).

[0058] The cloud computing environment is service-oriented, focusing on statelessness, low coupling, modularity, and semantic interoperability. At the core of cloud computing is an infrastructure that includes a network of interconnected nodes.

[0059] Now refer to Figure 1 , which shows a schematic diagram of an example of a cloud computing node. The cloud computing node 10 is merely an example of a suitable cloud computing node and is not intended to impose any limitation on the scope of use or functionality of the embodiments of the invention described herein. In any case, the cloud computing node 10 can be implemented and / or execute any of the functions set forth above.

[0060] In the cloud computing node 10, there is a computer system / server 12, which can operate with many other general-purpose or special-purpose computing system environments or configurations. Examples of well-known computing systems, environments, and / or configurations that can be applicable to the computer system / server 12 include, but are not limited to, personal computer systems, server computer systems, thin clients, fat clients, handheld or laptop devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputer systems, mainframe computer systems, and distributed cloud computing environments including any of the above systems or devices, etc.

[0061] The computer system / server 12 can be described in the general context of computer system-executable instructions (such as program modules) executed by a computer system. Generally speaking, program modules can include routines, programs, objects, components, logic, data structures, etc. that perform specific tasks or implement specific abstract data types. The computer system / server 12 can be practiced in a distributed cloud computing environment where tasks are executed by remote processing devices linked through a communication network. In a distributed cloud computing environment, program modules can be located in local and remote computer system storage media including memory storage devices.

[0062] As Figure 1 shown, the computer system / server 12 in the cloud computing node 10 is shown in the form of a general-purpose computing device. The components of the computer system / server 12 can include, but are not limited to, one or more processors or processing units 16, a system memory 28, and a bus 18 that couples the different system components including the system memory 28 to the processor 16.

[0063] Bus 18 represents any one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an Accelerated Graphics Port, and a processor or local bus using any one of a variety of bus architectures. By way of example and not limitation, such architectures include Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MCA) bus, Enhanced ISA (EISA) bus, Video Electronics Standards Association (VESA) local bus, and Peripheral Component Interconnect (PCI) bus.

[0064] Computer system / server 12 typically includes a variety of computer system readable media. Such media can be any available media accessible by computer system / server 12, and it includes volatile and nonvolatile media, removable and non-removable media.

[0065] System memory 28 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 30 and / or cache memory 32. Computer system / server 12 may further include other removable / non-removable, volatile / nonvolatile computer system storage media. By way of example only, a storage system 34 may be provided for reading from and writing to a non-removable, nonvolatile magnetic medium (not shown, typically referred to as a "hard disk drive"). Although not shown, a disk drive for reading from or writing to a removable nonvolatile disk (such as a "floppy disk"), and an optical disk drive for reading from or writing to a removable nonvolatile optical disk (such as a CD-ROM, DVD-ROM, or other optical medium) may be provided. In such cases, each may be connected to bus 18 through one or more data media interfaces. As will be further depicted and described below, memory 28 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of embodiments of the present invention.

[0066] A program / utility 40 having a set (at least one) of program modules 42, as well as an operating system, one or more application programs, other program modules, and program data, may be stored in memory 28 by way of example and not limitation. Each or some combination of the operating system, one or more application programs, other program modules, and program data may include an implementation of a network environment. Program modules 42 typically execute the functions and / or methods of embodiments of the present invention as described herein.

[0067] The computer system / server 12 can also communicate with one or more external devices 14 (such as a keyboard, a pointing device, a display 24, etc.); and / or any device that enables the computer system / server 12 to communicate with one or more other computing devices (such as, a network card, a modem, etc.). Such communication can be carried out via the input / output (I / O) interface 22. In addition, the computer system / server 12 can communicate with one or more networks through the network adapter 20, such as a local area network (LAN), a general wide area network (WAN), and / or a public network (such as the Internet). As shown in the figure, the network adapter 20 communicates with other components of the computer system / server 12 through the bus 18. It should be understood that although not shown, other hardware and / or software components can be used in combination with the computer system / server 12. Examples include, but are not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data archival storage systems, etc.

[0068] Now refer to Figure 2 , an illustrative cloud computing environment 50 is described. As shown in the figure, the cloud computing environment 50 includes one or more cloud computing nodes 10 with which local computing devices used by cloud consumers can communicate, such as a personal digital assistant (PDA) or a cellular phone 54A, a desktop computer 54B, a laptop computer 54C, and / or an in-vehicle computer system 54N. The nodes 10 can communicate with each other. They can be physically or virtually grouped (not shown) in one or more networks, such as a private cloud, a community cloud, a public cloud, or a hybrid cloud, as described above, or a combination thereof. This allows the cloud computing environment 50 to provide infrastructure, platform, and / or software as services for which cloud consumers do not need to maintain resources on local computing devices. It should be understood that Figure 1 the types of the computing devices 54A-N shown in are only intended to be illustrative, and the computing nodes 10 and the cloud computing environment 50 can communicate with any type of computerized device through any type of network and / or network addressable connection (such as, using a web browser).

[0069] Now refer to Figure 3 , a set of functional abstraction layers provided by the cloud computing environment 50 ( Figure 2 ) is shown. It should be understood in advance that Figure 3 the components, layers, and functions shown in are only intended to be illustrative, and the configuration of the present invention is not limited thereto. As shown in the figure, the following layers and corresponding functions are provided:

[0070] The hardware and software layer 60 includes hardware and software components. Examples of hardware components include: mainframes 61; servers 62 based on RISC (Reduced Instruction Set Computer) architecture; servers 63; blade servers 64; storage devices 65; and network and networking components 66. In some configurations, the software components include web application server software 67 and database software 68.

[0071] The virtualization layer 70 provides an abstraction layer from which the following examples of virtual entities can be provided: virtual servers 71; virtual memories 72; virtual networks 73, including virtual private networks; virtual applications and operating systems 74; and virtual clients 75.

[0072] In one example, the management layer 80 can provide the functions described below. Resource provisioning 81 provides for the dynamic procurement of computing resources and other resources for tasks executed within the cloud computing environment. Metering and pricing 82 provides cost tracking when resources are utilized within the cloud computing environment and bills or invoices for the consumption of these resources. In one example, these resources can include application software licenses. Security provides authentication for cloud consumers and tasks, as well as protection for data and other resources. The user portal 83 provides access to the cloud computing environment for consumers and system administrators. Service level management 84 provides cloud computing resource allocation and management such that the required service levels are met. Service level agreement (SLA) planning and fulfillment 85 provides for the pre-arrangement and procurement of cloud computing resources in anticipation of future demands according to the SLA.

[0073] The workload layer 90 provides examples of functions that can utilize the cloud computing environment. Examples of workloads and functions that can be provided from this layer include: maps and navigation 91; software development and lifecycle management 92; virtual classroom education delivery 93; data analysis processing 94; transaction processing 95; and data deduplication with opaque encryption 96.

[0074] Traditional data reduction techniques such as deduplication and / or compression do not provide meaningful reduction when applied to encrypted data. In cases where different encryption algorithms prevent the conventional deduplication process from identifying duplicate data chunks, the deduplication breakdown of multiple data sets encrypted with respective unique encryption keys. Traditional data reduction techniques also do not provide sufficient data privacy between the client and the storage system.

[0075] For example, a known Bring Your Own Key (BYOK) encryption technique involves a multi-party trust system. The client requests an encryption key from a third-party key service, and the key is returned to the client. The storage system can view the same encryption key. The client encrypts the data using the shared encryption key and sends the data to the storage system. The storage system can use the shared encryption key to decrypt the data and use the storage system key to re-encrypt the data for storage. During a client read operation, the storage system decrypts the data using the storage system key, re-encrypts the data using the shared encryption key, and returns the data to the client. Although all data reduction functions can be provided by a storage system that has access to all data (e.g., plaintext, encrypted with the storage system key, encrypted with the shared encryption key, etc.), traditional BYOK systems do not provide data privacy between the storage system and the client because the storage system can access the client key (e.g., the shared encryption key). The third-party key service can also access the shared encryption key used to encrypt the client's data. For this exemplary BYOK encryption technique, data privacy only exists between users.

[0076] In another BYOK technique, no third-party service is involved in the exchange of data, and the encryption key is shared directly with the storage system. This technique does not provide data privacy between the user and the storage system. Another exemplary conventional BYOK technique uses a "trusted decryptor" function on the storage system but still lacks in terms of data privacy.

[0077] Traditional static encryption uses a key known to the storage system to encrypt unencrypted input data. The storage system can decrypt all data and perform deduplication on all data in the system. Static encryption does not provide data privacy.

[0078] Traditional full client-side encryption encrypts data using a key unknown to the storage system. The storage system only performs deduplication on data encrypted with a common key. Full client-side deduplication provides relatively high data privacy but hampers deduplication efficiency.

[0079] In sharp contrast, the various schemes described herein enable the client to transfer encrypted data to the storage system for data compression and / or deduplication without the storage system having access to the client's secret encryption and / or decryption keys. At least some of the schemes disclosed herein encrypt the data in a manner that is opaque to the storage system and perform deduplication of the data against other data in the storage system. Compared to conventional encryption techniques including BYOK techniques, opaque encryption provides more data privacy. The different operations for opaque encryption provide relatively better data reduction than conventional full client-side encryption and less client overhead than client-side deduplication.

[0080] At least some of the configurations presented herein perform compression on the client system and generate deduplication metadata, and perform deduplication in the storage system. In one scenario, the client data is encrypted with the client secret encryption key and then sent to the storage system. Compared with the known BYOK technology, in a preferred scenario, the client data encrypted with the client secret encryption key is written to the storage system, and the storage system cannot decrypt the client data because the storage system can never access the client secret decryption key. At least some of the operations described herein can be used in conjunction with symmetric key encryption and / or asymmetric encryption (e.g., public key infrastructure (PKI)). Those of ordinary skill in the art should understand that PKI encryption can be performed according to any configuration known in the art. For example, the public key in PKI is not a secret key, and encrypting data with the public key requires the corresponding secret private key for decryption.

[0081] Figure 4 is an advanced architecture according to various configurations. In various configurations, architecture 400 can be implemented according to the present invention in any environment including Figures 1-3 and those shown in FIGS. 5-9. Of course, as those skilled in the art will understand upon reading this specification, architecture 400 may include more or fewer elements than Figure 4 specifically described herein.

[0082] Architecture 400 depicts an exemplary implementation of a write operation using symmetric key encryption. Architecture 400 includes a client 1 side 402. As shown, the client 1 side 402 sends the encrypted data to a storage system 420 (which will be discussed in detail below).

[0083] The client 1 side 402 includes a client secret encryption key 404. In a preferred scenario, the client secret encryption key 404 never leaves the client 1 side 402. In this preferred scenario, the storage system 420 never sees the encrypted data in plaintext at any time during data transfer. In a preferred scenario, the client secret encryption key 404 is not shared with the storage system 420, the client 2 side 436 (which will be discussed in detail below), any other client, any other storage system, etc. The client secret encryption key 404 can be a symmetric key, an asymmetric key, etc. As shown, the client secret encryption key 404 is a symmetric key.

[0084] The client 1 side 402 includes a meta-generator 406. In a preferred scenario, the meta-generator 406 calculates deduplication metadata associated with the data. In various configurations, the meta-generator 406 can be located on the client 1 side 402 and / or on the storage system 420.

[0085] In a preferred embodiment, the metadata generator 406 performs metadata operations on the data on the client 1 side 402 and identifies data deduplication opportunities in the data. The metadata generator 406 uses any data deduplication technique known in the art to identify data deduplication opportunities. In one aspect, the metadata generator 406 uses a sliding window variable block group deduplication technique to identify data deduplication opportunities. The metadata generator 406 identifies and / or calculates deduplication signatures (e.g., rolling minimal hash, cryptographic hash, etc.). The metadata generator 406 generates metadata (e.g., deduplication information) associated with the data deduplication opportunities, including pointers, block group lengths, block group signatures, etc. The deduplication information can be encrypted using an ephemeral key provided by the storage system as discussed in detail below.

[0086] In one embodiment, the metadata generator 406 uses any data compression technique known in the art to compress the data. In some embodiments, different compression techniques can be applied before and / or after chunking. In one configuration, pre-chunking compression can be a type of compression that improves the performance of chunking. In another configuration, post-chunking compression can be adjusted to minimize the resulting chunk size.

[0087] Signature calculations can be performed on the data chunks before and / or after compression. Pre-compression signature calculation allows verification of the signature after decompression during read. Post-compression signature calculation allows checking of the signature while the chunk is in a compressed state.

[0088] The client 1 side 402 includes a client key encryptor 408. As will be understood by those of ordinary skill in the art after reading this disclosure, the client key encryptor 408 can encrypt the data using the client secret encryption key 404.

[0089] The client 1 side 402 includes an ephemeral key encryptor 410. As will be understood by those of ordinary skill in the art when reading this disclosure, the ephemeral key encryptor 410 can encrypt the data using the storage ephemeral key 440 (to be discussed in detail below).

[0090] Client 1 side 402 includes a dechunker 412. The dechunker 412 can dechunk data chunks received from the storage system, where the data chunks are encrypted with multiple encryption keys. In a preferred embodiment, the data chunks encrypted with multiple encryption keys refer to the data chunks encrypted with multiple chunk encryption keys. For example, the written data 444 (to be discussed in detail below) can include a set of data chunks where a subset of the data chunks and / or individual data chunks are encrypted with different (e.g., unique) keys. Specifically, in an exemplary configuration, at least some of the data chunks are encrypted with a first key and at least some other data chunks are encrypted with a second key, where the first key and the second key are different keys. Unless explicitly disclosed herein, data chunks encrypted with multiple encryption keys should not be construed as data chunks that have undergone multiple encryptions. For example, data chunks that have undergone multiple encryptions refer to data chunks encrypted with a first key and then encrypted a second time with a different key, and so on.

[0091] The dechunker 412 can identify which encryption key a particular data chunk was encrypted with and forward the data chunk to the corresponding decryptor (e.g., client key decryptor 414, ephemeral key decryptor 416, etc.). As will be understood by those of ordinary skill in the art after reading this disclosure, each decryptor can use the corresponding key to decrypt the data.

[0092] The architecture 400 includes a storage system 420. The storage system 420 can be any type of storage system known in the art. Those skilled in the art will understand that the storage system 420 can have more or fewer components than those listed herein. The storage system 420 preferably performs the various deduplication operations described herein.

[0093] The storage system 420 includes a stored secret encryption key 422. In a preferred embodiment, the stored secret encryption key 422 never leaves the storage system 420. In this preferred embodiment, the stored secret encryption key 422 is not shared with client 1 side 402, client 2 side 436, any other client, any other storage system, etc. The stored secret encryption key 422 can be a symmetric key, an asymmetric key, etc. As shown, the stored secret encryption key 422 is a symmetric key.

[0094] The storage system 420 includes a deduplication manager 424. The deduplication manager 424 locates and / or identifies duplicate data based on the deduplication information provided by client 1 side 402. For chunking deduplication, the deduplication information is used together with the chunk metadata to identify duplicate data chunks. As will be understood by those of ordinary skill in the art when reading this disclosure, the deduplication manager 424 performs any associated deduplication operations. Any known deduplication technique can be used to deduplicate the data chunks.

[0095] The storage system 420 includes a chunk memory 426. The chunk memory 426 may include any storage components known in the art. The chunk memory 426 may store data chunks, deduplicated data, non-deduplicated data, encrypted data, unencrypted data, deduplication information, etc. in any manner known in the art. In one scenario, the chunk memory 426 stores data chunks, deduplicated data, non-deduplicated data, encrypted data, unencrypted data, deduplication information, etc. in a table.

[0096] The storage system 420 includes chunk metadata 428. The chunk metadata 428 may include any metadata associated with non-deduplicated and / or deduplicated data chunks, including pointers, chunk lengths, chunk signatures, etc. The chunk metadata 428 may be updated in response to receiving new data from the deduplication manager 424.

[0097] The storage system 420 includes a storage key encryptor 430. As those skilled in the art will understand upon reading this disclosure, the storage key encryptor 430 may encrypt data with the storage secret encryption key 422.

[0098] The storage system 420 includes a storage key decryptor 432. As those skilled in the art will understand after reading this disclosure, the storage key decryptor 432 may decrypt data encrypted with the storage secret encryption key 422.

[0099] The storage system 420 includes a storage temporary key encryptor 434. As those of ordinary skill in the art will understand upon reading this disclosure, the storage temporary key encryptor 434 may encrypt data using the storage temporary key 440 (which will be discussed in detail below).

[0100] The architecture 400 includes a client 2 side 436. As shown, the client 2 side 436 uses static encryption storage and sends unencrypted data to the storage system 420. The client 2 side 436 may send unencrypted data because the client 2 side 436 does not have any data privacy issues in at least some scenarios. Unencrypted data may refer to data received by the storage system in plain text. In various scenarios, any communication encryption known in the art may be implemented between various components.

[0101] The client 1 side 402 may issue a request 438 to the storage system 420 for a temporary key from the storage system 420. In one scenario, the client 1 side 402 establishes a session with the storage system 420 by sending the request 438 for the temporary key to the storage system 420. This temporary key may be used to encrypt data so that the storage system 420 can decrypt data sent from the client 1 side 402. The storage temporary key 440 may be returned to the request 438 in response to 442.

[0102] In one scenario, the client 1 side 402 can encrypt the write data 444 before sending it to the metadata generator 406. The client 1 side 402 sends the write data 444 and a key identifier (ID) indicating the client key (e.g., the client secret encryption key 404) used to encrypt the write data 444 to the metadata generator 406.

[0103] In a preferred scenario, the metadata generator 406 calculates the deduplication information (e.g., metadata) 446 associated with the write data 444. As those of ordinary skill in the art will understand when reading this disclosure, the deduplication information 446 can include the operational details of the deduplication method. In one configuration, for chunking dedupe, the deduplication information 446 can include the chunk start position, chunk length, chunk signature, similarity information, etc. In one scenario, the chunk signature is a cryptographic hash that can be calculated in any manner known in the art. The deduplication information 446 can be encrypted by the ephemeral key encryptor 410 with the stored ephemeral key 440 and sent to the storage system 420 (e.g., the deduplication manager 424 in the storage system 420).

[0104] In one scenario, the write data 444 is compressed by the metadata generator 406. Any compression form known in the art can be used. The compressed data 448 can be sent to the client key encryptor 408 and encrypted with the client secret encryption key 404. The client key-encrypted data 450 and the key ID associated with the client secret encryption key 404 can be sent to the storage system 420 (e.g., the deduplication manager 424). In another scenario, the write data 444 is sent to the client key encryptor 408 and encrypted with the client secret encryption key 404. The write data 444 and the key ID associated with the client secret encryption key 404 can be sent to the storage system 420 (e.g., the deduplication manager 424). The data 450 is opaque to the storage system 420 (e.g., the client secret encryption key 404 is not available to the storage system 420). The write data 444 is opaque to the storage system 420 (e.g., the client secret encryption key 404 is not available to the storage system 420).

[0105] In various scenarios, the deduplication information 446 and the write data 444 are tagged and / or otherwise indicated as being associated. In one aspect, the deduplication information 446 is calculated on the unencrypted data. The unencrypted data can include the chunk signature. The deduplication information 446 can be calculated on the encrypted data. The deduplication information 446 calculated on the encrypted data can include the signature of the encrypted chunk.

[0106] In a preferred embodiment, the deduplication manager 424 receives encrypted write data 444 (e.g., data encrypted with a key unavailable to the storage system 420) and encrypted deduplication information 446 (e.g., metadata encrypted with a key available to the storage system 420).

[0107] In one configuration, the deduplication manager 424 includes a temporary key decryptor. As will be understood by those of ordinary skill in the art upon reading this disclosure, the temporary key decryptor can decrypt data encrypted with the storage temporary encryption key 440.

[0108] The deduplication manager 424 locates and / or identifies duplicate data based on the deduplication information 446. For chunk-based deduplication, the deduplication information 446 is used with the chunk metadata 428 to identify duplicate data chunks. For example, the deduplication manager 424 compares pointers, signatures, chunk lengths, etc. from the decrypted deduplication information 446 with any pointers, signatures, chunk lengths, etc. stored in the chunk metadata 428. As will be understood by those of ordinary skill in the art upon reading this disclosure, the deduplication manager 424 performs any associated deduplication operations.

[0109] In response to performing the associated deduplication operations, an update 452 can be performed on the chunk metadata 428. The update 452 reflects changes due to using any deduplicated chunks, adding information related to any remaining chunks, etc. The chunk metadata 428 includes information about the key ID for encrypting a given data chunk. Any remaining data chunks associated with a new signature (e.g., not pre-existing in the chunk metadata 428) are written 454 to the chunk memory 426. Any new, non-deduplicated data chunks are written 454 to the chunk memory 426.

[0110] In a preferred configuration, the deduplication manager 424 performs deduplication for the client (e.g., client 1 side 402) on data chunks that the client can decrypt. Data chunks that the client 1 side 402 can decrypt include data chunks encrypted with a key belonging to the client 1 side 402 (e.g., the client secret encryption key 404), a key belonging to the storage system 420 (e.g., the storage secret encryption key 422), unencrypted data, etc. The client 1 side 402 may not be able to decrypt data encrypted with a unique client secret encryption key that the client 1 side 402 cannot access.

[0111] The storage secret encryption key 422 can be used at the storage key encryptor 430 to encrypt any remaining non-deduplicated data chunks 456 associated with the metadata update 452. The non-deduplicated data chunks 456 associated with the metadata update 452 can refer to data chunks pre-existing within the chunk metadata 428, and the metadata update 452 includes additional pointers to the corresponding storage locations.

[0112] In a preferred embodiment, the storage system 420 can access static encrypted data, unencrypted data, shared key encrypted data, etc., because as those of ordinary skill in the art will understand after reading this disclosure, the storage system 420 can decrypt any such data.

[0113] As shown, the client 2 side 436 sends unencrypted data 458 to the storage system 420 using static encryption. The unencrypted data 458 can be encrypted and stored in the chunk memory 426. The unencrypted data 458 can be received unencrypted at the client 2 side 436 in response to a read request. In a preferred embodiment, the deduplication manager 424 performs deduplication in a manner substantially similar to the deduplication performed for the client 1 side 402. The unencrypted data 458 can be deduplicated against data chunks encrypted with a storage key (e.g., the storage secret encryption key 422) and / or unencrypted data chunks.

[0114] In some configurations, the deduplication manager 424 can include a metadata generator (not shown) that performs functions substantially similar to those of the metadata generator 406. Any remaining undeduplicated data chunks 456 can be encrypted with the storage secret encryption key 422 at the storage key encryptor 430. The encrypted, undeduplicated data chunks 460 can be written to the chunk memory 426. The storage system 420 can decrypt any data transmitted according to the processes described herein. All data chunks 460 can be used for deduplication against any data chunks encrypted with any key.

[0115] Figure 5 is an advanced architecture according to various configurations. In various configurations, the architecture 500 can be implemented according to the present invention in any environment including Figures 1-4 and those depicted in FIGS. 6-9. Of course, as those skilled in the art will understand upon reading this specification, the architecture 500 can include more or fewer elements than Figure 5 specifically described herein.

[0116] The architecture 500 depicts an exemplary embodiment of a read operation using symmetric key encryption. In a preferred configuration, the read operation includes receiving data chunks encrypted with at least two keys (e.g., a client secret encryption key and a storage temporary key). For example, the read operation includes receiving one or more data chunks encrypted with a first key and one or more data chunks encrypted with a second key. The architecture 500 includes the client 1 side 402.

[0117] Client 1 side 402 includes a client secret encryption key 404. In a preferred embodiment, the client secret encryption key 404 never leaves the client 1 side 402. In this preferred embodiment, the storage system 420 never sees the plaintext client-side encrypted data at any time during data transfer. In this preferred embodiment, the client secret encryption key 404 is not shared with the storage system 420, client 2 side 436 (which will be discussed in detail below), any other client, any other storage system, etc. The client secret encryption key 404 can be a symmetric key, an asymmetric key, etc. As shown, the client secret encryption key 404 is a symmetric key.

[0118] Client 1 side 402 includes a metadata generator 406. In a preferred embodiment, the metadata generator 406 calculates deduplication metadata associated with the data. In various configurations, the metadata generator 406 can be located on the client 1 side 402 and / or on the storage system 420.

[0119] In a preferred embodiment, the metadata generator 406 performs metadata operations on the data on the client 1 side 402 and identifies deduplication opportunities in the data. The metadata generator 406 uses any deduplication technique known in the art to identify deduplication opportunities. In one aspect, the metadata generator 406 uses a sliding window variable block group deduplication technique to identify deduplication opportunities. The metadata generator 406 identifies and / or calculates a deduplication signature (e.g., rolling minhash, cryptographic hash, etc.). The metadata generator 406 generates metadata associated with the deduplication opportunity (e.g., deduplication information), including pointers, chunk lengths, chunk signatures, etc. The deduplication information can be encrypted with a temporary key provided by the storage system, as will be discussed in detail below.

[0120] In one embodiment, the metadata generator 406 uses any data compression technique known in the art to compress the data. In some embodiments, different compression techniques can be applied before and / or after chunking. In one configuration, pre-chunking compression can be a type of compression that improves the performance of chunking. In another configuration, post-chunking compression can be adjusted to minimize the resulting chunk size.

[0121] Signature calculations can be performed on the data chunks before and / or after compression. Pre-compression signature calculation allows verification of the signature after decompression during read. Post-compression signature calculation allows checking of the signature while the chunk is in the compressed state.

[0122] Client 1 side 402 includes a client key encryptor 408. As will be understood by those of ordinary skill in the art after reading this disclosure, the client key encryptor 408 can encrypt the data using the client secret encryption key 404.

[0123] Client 1 side 402 includes a temporary key encryptor 410. As will be understood by those of ordinary skill in the art upon reading this disclosure, the temporary key encryptor 410 may encrypt data with a temporary key 504 (which will be discussed in detail below).

[0124] Client 1 side 402 includes a dechunker 412. The dechunker 412 may dechunk data chunks received from the storage system, where the data chunks are encrypted with multiple encryption keys. Data chunks encrypted with multiple encryption keys preferably refer to a set of data chunks in which one or more data chunks are encrypted with a first key and one or more data chunks are encrypted with a second key. The dechunker 412 may identify which encryption key a certain data chunk is encrypted with and forward the data chunk to the corresponding decryptor (such as the client key decryptor 414, the temporary key decryptor 416, etc.). As will be understood by those of ordinary skill in the art after reading this disclosure, each decryptor may decrypt the data chunk encrypted with the corresponding key.

[0125] The architecture 500 includes a storage system 420. The storage system 420 may be any type of storage system known in the art. Those skilled in the art will understand that the storage system 420 may have more or fewer components than those listed herein. The storage system 420 preferably performs the various deduplication operations described herein.

[0126] The storage system 420 includes a stored secret encryption key 422. In a preferred embodiment, the stored secret encryption key 422 never leaves the storage system 420. In this preferred embodiment, the stored secret encryption key 422 is not shared with the client 1 side 402, the client 2 side 436, any other client, any other storage system, etc. The stored secret encryption key 422 may be a symmetric key, an asymmetric key, etc. As shown, the stored secret encryption key 422 is a symmetric key.

[0127] The storage system 420 includes a deduplication manager 424. The deduplication manager 424 locates and / or identifies duplicate data based on the deduplication information provided by the client 1 side 402. For chunk-based deduplication, the deduplication information is used together with the chunk metadata to identify duplicate data chunks. As will be understood by those of ordinary skill in the art upon reading this disclosure, the deduplication manager 424 performs any associated deduplication operations. Any deduplication technique known in the art may be used.

[0128] The storage system 420 includes a chunk memory 426. The chunk memory 426 can include any storage components known in the art. The chunk memory 426 can store data chunks, deduplicated data, non-deduplicated data, encrypted data, unencrypted data, deduplication information, etc. in any manner known in the art. In one scenario, the chunk memory 426 stores data chunks, deduplicated data, non-deduplicated data, encrypted data, unencrypted data, deduplication information, etc. in a table.

[0129] The storage system 420 includes chunk metadata 428. The chunk metadata 428 can include any metadata associated with non-deduplicated and / or deduplicated data chunks, including pointers, chunk lengths, chunk signatures, etc.

[0130] The storage system 420 includes a storage key encryptor 430. As those skilled in the art will understand upon reading this disclosure, the storage key encryptor 430 can encrypt data using the storage secret encryption key 422.

[0131] The storage system 420 includes a storage key decryptor 432. As those skilled in the art will understand after reading this disclosure, the storage key decryptor 432 can decrypt data encrypted with the storage secret encryption key 422.

[0132] The storage system 420 includes a storage temporary key encryptor 434. As those of ordinary skill in the art will understand upon reading this disclosure, the storage temporary key encryptor 434 can encrypt data using the storage temporary encryption key 504.

[0133] The architecture 500 includes a client 2 side 436. As shown, the client 2 side 436 sends unencrypted data 458 to the storage system 420 using static encryption. The unencrypted data 458 can be encrypted and stored in the chunk memory 426. The unencrypted data 458 can be received unencrypted at the client 2 side 436 in response to a read request. In a preferred scenario, the deduplication manager 424 performs deduplication in a manner substantially similar to the deduplication performed for the client 1 side 402. The unencrypted data 458 can be deduplicated against data chunks encrypted with a storage system key (such as the storage secret encryption key 422) and / or unencrypted data chunks.

[0134] Client 1 side 402 may issue a request 502 for a temporary key from storage system 420 to storage system 420. In one scenario, client 1 side 402 establishes a session with storage system 420 by making a request 502 for a temporary key to storage system 420. This temporary key may be used to encrypt the returned data chunks. In response to 506, the temporary key 504 may be returned to request 502. The temporary key 504 may be applied to deduplication and at least some of the data chunks during read. Client 1 side 402 sends a read request 508 to storage system 420.

[0135] In a preferred scenario, the deduplication manager 424 uses the chunk metadata 428 to identify the data chunks associated with read request 508. The identified data chunks are read 510 from the chunk memory 426. The deduplication manager 424 compares the pointers, signatures, chunk lengths, etc. from read request 508 with any pointers, signatures, chunk lengths, etc. stored in the chunk metadata 428. In a preferred scenario, the chunk metadata 428 includes an identifier for the key used to encrypt each data chunk. The data chunks encrypted with the client key may be sent 512 to the dechunker 412. An indicator that the data chunks are encrypted with the client key may also be sent 512.

[0136] The data chunks 514 encrypted with the storage secret encryption key 422 may be sent to the storage key decryptor 432. The storage key decryptor 432 may decrypt the data encrypted with the storage secret encryption key 422. The output 516 of the storage key decryptor 432 may be sent to the storage temporary key encryptor 434. As those of ordinary skill in the art will understand upon reading this disclosure, the storage temporary key encryptor 434 may use the storage temporary encryption key 504 to encrypt the data.

[0137] In one configuration, the output 518 of the storage temporary key encryptor 434 can be sent to the de-chunker 412. An indicator that the output 518 is encrypted with the temporary key 504 can be sent to the de-chunker 412. The de-chunker 412 can de-chunk data chunks received from the storage system, where the data chunks are encrypted with multiple encryption keys. The data chunks encrypted with multiple encryption keys preferably refer to a set of data chunks where one or more data chunks are encrypted with a first key and one or more data chunks are encrypted with a second key. The de-chunker 412 can identify which encryption key a certain data chunk is encrypted with and forward the data chunk to the corresponding decryptor (e.g., the client key decryptor 414, the temporary key decryptor 416, etc.). Specifically, the de-chunker 412 sends the client key encrypted data chunk 520 to the client key decryptor 414 and / or the de-chunker 412 sends the temporary key encrypted data chunk 522 to the temporary key decryptor 416. As those of ordinary skill in the art will understand upon reading this disclosure, each decryptor can decrypt the data chunk encrypted with the corresponding key. In various scenarios, the de-chunker 412 can perform any decompression operations known in the prior art. The de-chunker 412 can merge the data chunks to generate data 524 for the client 1 side 402.

[0138] As those of ordinary skill in the art will understand after reading this disclosure, the client key decryptor 414 can be provided by the client 1 side 402 separate from the de-chunker 412 to keep the client secret encryption key 404 confidential.

[0139] The client key encrypted data chunk and the output 518 of the storage temporary key encryptor 434 sent from the storage system 420 can include data chunk metadata, which includes a chunk signature for verifying that each chunk is correctly transmitted and decrypted with the appropriate key. The data chunk metadata can provide an end-to-end check. In some scenarios, when using the client key, the metadata generator 406 can calculate the signature of each data chunk on the unencrypted data. The signature of each data chunk can be calculated by a meta-generator (not shown) in the deduplication manager 424 for any statically encrypted and / or unencrypted data chunks.

[0140] In a preferred configuration of the architecture 500, various compression and / or deduplication techniques can be performed without the client secret encryption key 404 ever leaving the client 1 side 402. The client secret encryption key 404 is not available to the storage system 420, the client 2 side 436, any other client, any third-party service, any other storage system, etc. A part of the deduplication process is preferably performed on the client side (e.g., the chunking process in the architecture 500), which significantly reduces or eliminates the external visibility of the client secret encryption key 404.

[0141] As described above, the ephemeral key 504 can be applied to deduplication and at least some data chunks during reading to help protect the privacy of the data. The ephemeral key 504 can be independent transport encryption (e.g., Transport Layer Security (TLS)). The deduplication metadata and chunk data can be provided in plaintext to the transport layer. In this scenario, the data encrypted with the client key can be re-encrypted with the ephemeral key 504.

[0142] In a preferred scenario, compared to a conventional deduplication operation, additional chunk metadata is associated with each data chunk to indicate the key used to encrypt the chunk. Different clients may have the same data chunks (e.g., copies with the same signature). However, if duplicate data chunks from different clients are encrypted with different keys, conventional deduplication of client 1 data chunks against client 2 data chunks requires sharing a key. In stark contrast, various configurations of the present disclosure provide chunk identifiers that include information such as the key, index, tag, signature, key ID, etc. used to encrypt the data chunk. If a client uses more than one key, information about the key ID can be provided by the client 1 side 402 to the storage system 420 prior to any input / output (I / O) operation (e.g., including key exchange, which includes a request 502 to store the ephemeral key 440 and a response 506 that includes storing the ephemeral key 440). If a client uses a single key for all data encryption, the key ID uniquely identifies the client.

[0143] In the context of data privacy, the key ID is not the key itself. For example, the provided key ID can be a cryptographic hash, such as a SHA 256 of the client secret encryption key 404. In another example, the key ID can be the unique index number of the client secret encryption key 404. The storage system 420 can create an internal reference key ID. For example, the storage system 420 can create a key index table for the ID of the client secret encryption key 404.

[0144] In response to encrypted data from at least two clients with a common signature and different key IDs, the deduplication manager 424 does not deduplicate the encrypted data because both the signature and the key ID must match to perform deduplication. As those of ordinary skill in the art will understand upon reading the present disclosure, data chunks encrypted with a storage key and / or any unencrypted data chunks can be deduplicated against data chunks encrypted with any client key.

[0145] The chunk metadata 428 in the deduplicated storage (such as the chunk memory 426) may include a reference counter indicating how many data fragments point to the data chunk in various scenarios. Metadata indicating the source key ID and the reference counter for each key ID may be added to each data chunk. An attempt to read a chunk without a non-zero reference count for the requested key ID may be intercepted. This interception provides access control for the deduplicated data chunks. For example, the client 1 side 402 using the key ID associated with the client secret encryption key 404 may have data including the deduplicated chunk A. Chunk A may be the only piece of data encrypted with the client secret encryption key 404 copied to block group A. The client 1 side 402 may delete the data including the deduplicated chunk A. The reference counter for the key ID of the client 1 side 402 associated with the secret encryption key 404 is decremented to 0. Any subsequent attempt to read chunk A with the key ID associated with the client 1 side 402 secret encryption key 404 may be blocked, and any request with a non-zero associated reference count may be allowed.

[0146] If the data encrypted with the client secret encryption key 404 is not initially deduplicated, the data may be deduplicated in response to writing a data chunk encrypted with the storage secret encryption key 422 with a matching signature and / or writing an unencrypted data chunk. In various configurations described herein, the process for deleting an initially non-deduplicated data chunk may be referred to as backporting. Backporting may include a deduplication operation for storing a new data chunk (such as encrypted with a key available to the storage system 420). Backporting includes deleting any pre-existing chunks with a matching signature but a different key ID. The metadata of the stored and / or deleted data chunks may be converted to point to the new data chunk (as if the new data chunk existed first). The order in which data chunks are written with different keys does not affect the final deduplication efficiency. Any subsequent static and / or unencrypted writes of copies of the data chunks reduce the storage capacity on a system with many private copies of the data chunks using the backup techniques described herein.

[0147] Various configurations described herein may be implemented using asymmetric encryption keys (such as PKI). For relatively large data sets, symmetric key encryption may be preferred over asymmetric key encryption. Asymmetric key encryption includes a client having a public key shared with the storage system and a private key not shared with the storage system. The storage system may have a public key shared with the client and a private key not shared with the client. The public key in PKI is not secret, but encrypting data with the public key requires the corresponding secret private key to decrypt the data.

[0148] Figure 6 is an advanced architecture according to various configurations. In various configurations, the architecture 600 may be in accordance with the present invention in including Figures 1-5and Figures 7-9 implemented in any environment depicted in Figure 6 . Of course, as those skilled in the art will understand upon reading this specification, architecture 600 may include more or fewer elements than those specifically described in Figure 6 . Figure 6 the elements specifically described therein.

[0149] Architecture 600 depicts an exemplary implementation of a write operation using asymmetric key encryption. Architecture 600 includes a client 1 side 602.

[0150] The client 1 side 602 includes a client private encryption key 604 and an associated client public key 606. The client 1 side 602 encrypts data with the client public key 606 before sending the data to the storage system 608. Contrary to the case of symmetric key encryption, the storage system 608 may return all data encrypted with the client public key 606 to the client 1 side 602.

[0151] Architecture 600 includes a storage system 608. The storage system 608 includes a storage private key 610 and an associated storage public key 612.

[0152] The storage system 608 may return all data encrypted with the client public key 606 to the client 1 side 602. The storage system 608 includes a dechunking unit 614. The dechunking unit 614 can operate without knowledge of the client private encryption key 604.

[0153] In a preferred embodiment, before writing data, the client 1 side 602 and the storage system 608 exchange public keys. The client public key 606 may be sent 616 to the storage system 608. The storage public key 612 may be sent 618 to the client 1 side 602. The client 1 side 602 sends the write data 620 to a metadata generator 622.

[0154] In a preferred embodiment, the metadata generator 622 calculates deduplication information (e.g., metadata) 624 associated with the write data 620. The deduplication information 624 may include operational details of the deduplication method as those of ordinary skill in the art will understand upon reading this disclosure. In one configuration, for chunk-based deduplication, the deduplication information 624 may include chunk start position, chunk length, chunk signature, similarity information, etc. In one embodiment, the chunk signature is a cryptographic hash that can be calculated in any manner known in the art. The deduplication information 624 may be encrypted by a storage public key encryptor 626 using the storage public key 612. The deduplication information 624 may be sent to the storage system 608.

[0155] In one scenario, the write data 620 is compressed by a metadata generator 622. Any compression form known in the art may be used. The compressed data 628 may be sent to a client public key encryptor 630 and encrypted with a client public key 606. The client public key encrypted data 632 may be sent to a storage system 608 (e.g., a deduplication manager 634). The client public key encrypted data 632 and the deduplication information 624 may be sent to the storage system 608 (e.g., the deduplication manager 634). The data 632 is opaque to the storage system 608 (e.g., the client private encryption key 604 is not available to the storage system 608). The deduplication information 624 may be encrypted by a storage public key encryptor 626 using a storage public key 612. In another scenario, the write data 620 may be encrypted by a storage public key encryptor 626 using a storage public key 612. The encrypted write data 620 and the deduplication information 624 may be sent to the storage system 608 (e.g., the deduplication manager 634). In various scenarios, the deduplication information 624 and the write data 620 are tagged and / or otherwise indicated as being associated.

[0156] In one configuration, the deduplication manager 634 includes a storage private key decryptor. As will be understood by one of ordinary skill in the art after reading this disclosure, the storage private key decryptor may decrypt data encrypted with a storage private key 610.

[0157] The deduplication manager 634 locates and / or identifies duplicate data based on the deduplication information 624. For chunk-based deduplication, the deduplication information 624 is used with chunk metadata 428 to identify duplicate data chunks. For example, the deduplication manager 634 compares pointers, signatures, chunk lengths, etc. from the decrypted deduplication information 624 with any pointers, signatures, chunk lengths, etc. stored in the chunk metadata 428. As will be understood by one of ordinary skill in the art upon reading this disclosure, the deduplication manager 634 performs any associated deduplication operations. Any deduplication technique known in the art may be used.

[0158] In response to performing the associated deduplication operations, an update 452 may be performed on the chunk metadata 428. The update 452 reflects changes due to using any deduplicated chunks, adding information related to any remaining chunks, etc. Any remaining data chunks that are not deduplicated are written 454 to a chunk memory 426. The chunk metadata 428 includes information about the key ID that encrypted a given data chunk.

[0159] In a preferred configuration, the deduplication manager 634 performs deduplication for the client against data chunks that the client (e.g., client 1 side 602) can decrypt. The data chunks that client 1 side 602 can decrypt include data chunks encrypted with a key belonging to client 1 side 602 (e.g., client private encryption key 604), a key belonging to the storage system 608 (e.g., storage private key 610), as well as unencrypted data, etc.

[0160] Any remaining undeduplicated data chunks 456 can be encrypted with the storage private key 610 at the storage private key encryptor 636.

[0161] In a preferred scenario, the storage system 608 can access statically encrypted data, unencrypted data, shared key encrypted data, etc., because as those of ordinary skill in the art will understand upon reading this disclosure, the storage system 608 can decrypt any such data.

[0162] The storage system 608 includes a storage private key decryptor 638. As those of ordinary skill in the art will understand after reading this disclosure, the storage private key decryptor 638 can decrypt data encrypted with the storage private key 610.

[0163] The storage system 608 includes a client public key encryptor 640. As those of ordinary skill in the art will understand after reading this disclosure, the client public key encryptor 640 can encrypt data with the client public key 606.

[0164] In one scenario, the storage system 608 can return data chunks through the dechunking unit 614 for decryption by the client private key decryptor 642 in any manner known in the art.

[0165] As shown, client 2 side 436 sends unencrypted data 458 to the storage system 608 using static encryption. The unencrypted data 458 can be encrypted and stored in the chunk memory 426. The unencrypted data 458 can be received unencrypted at client 2 side 436 in response to a read request. In a preferred scenario, the deduplication manager 634 performs deduplication in a manner substantially similar to the deduplication performed for client 1 side 602. The unencrypted data 458 can be deduplicated against data chunks encrypted with a storage system key (e.g., storage private key 610) and / or unencrypted data chunks. Any remaining undeduplicated data chunks 456 can be encrypted with the storage private key 610 at the storage private key encryptor 636. The encrypted, undeduplicated data chunks 460 can be written to the chunk memory 426. The storage system 608 can decrypt any data transmitted according to the foregoing operations. All data chunks 460 can be used for deduplication against any chunks encrypted herein.

[0166] Figure 7is an advanced architecture according to various configurations. In various configurations, the architecture 700 can be implemented according to the present invention in any environment depicted in Figures 1-6 and Figures 8-9. Of course, as those skilled in the art will understand after reading this description, the architecture 700 may include more or fewer elements than those Figure 7 specifically described herein.

[0167] The architecture 700 depicts an exemplary implementation of a read operation using asymmetric key encryption. The architecture 700 includes a client 1 side 602. The client 1 side 602 includes a client private encryption key 604 and an associated client public key 606. The architecture 700 includes a storage system 608. The storage system 608 includes a storage private key 610 and an associated storage public key 612.

[0168] In a preferred embodiment, before writing data, the client 1 side 602 and the storage system 608 exchange public keys. The client public key 606 can be sent 616 to the storage system 608. The storage public key 612 can be sent 618 to the client 1 side 602. The client 1 side 602 sends the write data 620 to the metadata generator 622.

[0169] The client 1 side 602 sends a read request 702 to the storage system 608 (e.g., the deduplication manager 634). The deduplication manager 634 uses the chunk metadata 428 to locate and / or identify the data chunks associated with the read request 702. Any identified data chunks can be read from the chunk memory 426. The chunk metadata 428 includes an identifier of the key used to encrypt the data chunks. The data chunks encrypted with the client private encryption key 604 can be sent to the dechunking unit 614. An indicator that the data chunks are encrypted with the client private encryption key 604 can be sent to the dechunking unit 614.

[0170] Data chunks recognized as being encrypted with the stored private key 610 can be sent 704 to the stored private key decryptor 638. As will be understood by those of ordinary skill in the art after reading this disclosure, the stored private key decryptor 638 can decrypt data encrypted with the stored private key 610. The output 706 of the stored private key decryptor 638 can be sent to the client public key encryptor 640. As will be understood by those of ordinary skill in the art after reading this disclosure, the client public key encryptor 640 can encrypt the data with the client public key 606. The output 708 of the client public key encryptor 640 can be sent to the de-chunker 614. Contrary to the symmetric key implementation, all data chunks sent to the de-chunker 614 are encrypted with the client public key 606. The de-chunker 614 can perform any decompression on the data chunks. In a preferred embodiment, the de-chunker 614 combines all data chunks into a combined data set 710 and sends the combined data set 710 to the client private key decryptor 642. As will be understood by those of ordinary skill in the art after reading this disclosure, the client private key decryptor 642 can decrypt the combined data set 710. The output 712 of the client private key decryptor 642 includes any data chunks associated with the initial read request 702.

[0171] Now referring Figure 8 , a flowchart of a method 800 according to one configuration is shown. In various configurations, the method 800 may be performed according to the present invention in any environment including Figures 1-7 and Figure 9 depicted herein. Of course, as will be understood by those skilled in the art after reading this description, the method 800 may include more or fewer operations than those specifically described in Figure 8 herein.

[0172] Each step of the method 800 can be performed by any suitable component of the operating environment. For example, in various configurations, the method 800 may be performed in part or in whole by a computer or some other device having one or more processors. A processor (e.g., a processing circuit, chip, and / or module implemented in hardware and / or software and preferably having at least one hardware component) can be used in any device to perform one or more steps of the method 800. Illustrative processors include, but are not limited to, a central processing unit (CPU), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), etc., combinations thereof, or any other suitable computing device known in the art.

[0173] As Figure 8As shown, method 800 includes operation 802, which includes receiving deduplication information at a storage system, where the deduplication information includes a signature associated with a portion of client data. In a preferred scenario, the deduplication information is encrypted using a shared temporary key. In one scenario, the storage system receives the deduplicated information encrypted with the shared temporary key at a deduplication manager. In one configuration, the deduplication manager includes a temporary key decryptor. As those of ordinary skill in the art will understand upon reading this disclosure, the temporary key decryptor can decrypt data encrypted with a storage temporary encryption key. In a preferred configuration, the deduplication information is accessible to the storage system (e.g., received in plain text by the storage system, received in a manner that the storage system can operate on the information, etc.).

[0174] As those of ordinary skill in the art will understand upon reading this disclosure, the deduplication information can include operational details of the deduplication method. In one configuration, for chunk-based deduplication, the deduplication information can include a chunk start position, a chunk length, a chunk signature, similarity information, etc. In one scenario, the chunk signature is a cryptographic hash that can be calculated in any manner known in the art. Preferably, the deduplication information is encrypted using a storage temporary key (e.g., a shared temporary key).

[0175] Operation 804 includes receiving client data encrypted with a client key, where the client key is not available to the storage system. The client data is opaque to the storage system (e.g., the client key is not available to the storage system). A key that is not available to the storage system is a key that encrypts data that the storage system cannot decrypt.

[0176] Operation 806 includes deduplicating data chunks stored in the storage system against client data chunks, where the client data chunks are selected from the client data for deduplication using the deduplication information. In one scenario, the deduplication manager locates and / or identifies duplicate data based on the deduplication information. For chunk-based deduplication, the deduplication information can be used together with chunk metadata to identify duplicate data chunks. For example, the deduplication manager compares pointers, signatures, chunk lengths, etc. from the decrypted deduplication information with any pointers, signatures, chunk lengths, etc. stored in a chunk metadata repository on the storage system. The deduplication manager performs any associated deduplication operations as those of ordinary skill in the art will understand upon reading this disclosure. In a preferred scenario, the storage system stores metadata for data chunks, where the metadata includes a key identifier that uniquely identifies the key used to encrypt the data chunk.

[0177] In response to performing an associated deduplication operation, an update can be performed on the chunk metadata repository. The update reflects changes due to using any deduplicated chunks, adding information related to any remaining chunks, etc. The chunk metadata repository includes information about the key IDs for encrypting a given data chunk. Any remaining data chunks that are not deduplicated (e.g., not pre-existing in the chunk metadata repository) and are associated with a new signature are written to the storage system. Any new, non-deduplicated data chunks are written to the storage system.

[0178] In a preferred configuration, the deduplication manager performs deduplication for the client against data chunks that the client can decrypt. Data chunks that can be decrypted on the client side include data chunks encrypted with a key belonging to the client (e.g., encrypted with the client key), data chunks encrypted with a key belonging to the storage system (e.g., encrypted with the storage secret encryption key), unencrypted data, etc. The client may not be able to decrypt data encrypted with a unique client secret encryption key that the client cannot access.

[0179] Any remaining non-deduplicated data chunks associated with the metadata update can be encrypted at the storage key encryptor using the storage secret encryption key. The non-deduplicated data chunks associated with the metadata update can refer to data chunks pre-existing within the chunk metadata repository, and the metadata update includes additional pointers to the corresponding storage locations. In various scenarios, any non-deduplicated client data chunks encrypted with the client key are stored in the storage system.

[0180] In a preferred scenario, the storage system can access statically encrypted data, unencrypted data, shared key encrypted data, etc., because as those of ordinary skill in the art will understand after reading this disclosure, the storage system can decrypt any such data.

[0181] In one scenario, the storage system includes data chunks encrypted with a shared temporary key. The shared temporary key is available to the storage system. The storage system can perform a deduplication operation on the data chunks encrypted with the shared temporary key against data chunks encrypted with the client key and against data chunks encrypted with the shared temporary key.

[0182] In one scenario, the storage system can receive data encrypted with a second client key. The storage system can perform deduplication of the data chunks encrypted with the second client key against data chunks encrypted with multiple keys according to the above-described backporting operation.

[0183] Now refer to Figure 9 , a flowchart of a method 900 according to one configuration is shown. In various configurations, method 900 can be performed according to the present invention in any environment including Figures 1-8 depicted herein. Of course, as those skilled in the art will understand when reading this specification, method 900 can include more thanFigure 9 more or fewer operations than those specifically described therein.

[0184] Each step of method 900 may be performed by any suitable component of the operating environment. For example, in various configurations, method 900 may be performed in part or in whole by a computer or some other device having one or more processors therein. A processor (e.g., a processing circuit, chip, and / or module implemented in hardware and / or software and preferably having at least one hardware component) may be used in any device to perform one or more steps of method 900. Illustrative processors include, but are not limited to, a central processing unit (CPU), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), etc., combinations thereof, or any other suitable computing device known in the art.

[0185] As Figure 9 shown, method 900 includes operation 902, which includes generating deduplication information, where the deduplication information includes a signature associated with a portion of the client data. In a preferred embodiment, a metadata generator calculates deduplication metadata associated with the data. In various configurations, the metadata generator may be located on the client side and / or on the storage system.

[0186] In a preferred embodiment, the metadata generator performs metadata operations on the data on the client side and identifies data deduplication opportunities in the data. The metadata generator uses any data deduplication technique known in the art to identify data deduplication opportunities. In one aspect, the metadata generator uses a sliding window variable block group deduplication technique to identify data deduplication opportunities. The metadata generator identifies and / or calculates deduplication signatures (e.g., rolling minimal hash, cryptographic hash, etc.). The metadata generator generates metadata (e.g., deduplication information) associated with the data deduplication opportunities, including pointers, block lengths, block signatures, etc. The deduplication information may be encrypted using a shared temporary key provided by the storage system in response to a request for the shared temporary key from the client side.

[0187] As will be understood by those of ordinary skill in the art upon reading this disclosure, the deduplication information may include operational details of the deduplication method. In one configuration, for block-based deduplication, the deduplication information may include block start positions, block lengths, block signatures, similarity information, etc. In one embodiment, the block signature is a cryptographic hash that may be calculated in any manner known in the art.

[0188] Operation 904 includes encrypting the deduplication information using the shared temporary key. The client side may include a temporary key encryptor. As will be understood by those of ordinary skill in the art upon reading this disclosure, the temporary key encryptor may use the stored temporary key to encrypt the data.

[0189] Operation 906 includes encrypting client data with a client key. The client side may include a client key encryptor. As will be understood by those of ordinary skill in the art after reading this disclosure, the client key encryptor may encrypt data with the client key. In a preferred scenario, the client key never leaves the client. In this preferred scenario, the storage system can never see the encrypted data in plaintext at any time during data transfer. In this preferred scenario, the client key is not shared with the storage system, any other client, any other storage system, etc. The client key may be a symmetric key, an asymmetric key, etc. In one aspect, the client key is a symmetric key.

[0190] In some scenarios, different compression techniques may be applied to the client data before and / or after chunking. In one configuration, pre-chunking compression may be a type of compression that improves the performance of chunking. In another configuration, post-chunking compression may be adjusted to minimize the resulting chunk size.

[0191] Signature calculations may be performed on data chunks before and / or after compression. Pre-compression signature calculation allows verification of the signature after decompression during read. Post-compression signature calculation allows checking of the signature while the chunk is in a compressed state.

[0192] In various scenarios, the deduplication information and the client data are tagged and / or otherwise indicated as being associated. In a preferred scenario, the deduplication information is calculated on the unencrypted data. The unencrypted data may include chunk signatures. The deduplication information may be calculated on the encrypted data. The deduplication information calculated on the encrypted data may include the signature of the encrypted chunk.

[0193] Operation 908 includes sending the deduplication information and the client data to the storage system for deduplication. The deduplication information and the client data may be sent to the storage system for deduplication in any manner known in the art.

[0194] Operation 910 includes requesting client data from the storage system. The client data may be requested in any manner known in the art.

[0195] In one scenario, the request for client data from the storage system includes verifying the request with a signed access control. In some scenarios, an asymmetric key encrypted signature may be used as a secure access control for symmetric key encrypted data transfer. The client may sign all I / O requests with a client private encryption key (e.g., a client private PKI key). For example, for symmetric key encryption of client data, the client private PKI key may be used for authentication. The storage system may use the public key to verify the signature. Similarly, the storage system may sign the I / O return with a storage private encryption key, which may be verified by the client using the public key.

[0196] Operation 912 includes receiving client data as data chunks, where at least some of the data chunks are encrypted with a client key and at least some of the data chunks are encrypted with a shared temporary key. The client data as data chunks can be received in any manner known in the art.

[0197] In one scenario, the client side includes a de-chunker. The de-chunker can de-chunk the data chunks received from the storage system, where the data chunks are encrypted with multiple encryption keys. The data chunks encrypted with multiple encryption keys preferably refer to a set of data chunks where one or more data chunks are encrypted with a first key and one or more data chunks are encrypted with a second key. The de-chunker can identify which encryption key a certain data chunk is encrypted with and forward the data chunk to the corresponding decryptor (such as a client private key decryptor, a shared temporary key decryptor, etc.). As will be understood by those of ordinary skill in the art after reading this disclosure, each decryptor can decrypt the data chunk encrypted with the corresponding key.

[0198] In various configurations disclosed herein, compression and / or deduplication is performed without the client secret encryption key ever leaving the client. In a preferred configuration, the storage system never has access to the client secret encryption key, and a third party cannot view the client secret encryption key.

[0199] In other methods, a hybrid of symmetric encryption and asymmetric encryption can be used. For example, the storage system can use symmetric key encryption for storage to store chunks encrypted with a storage key, and can use an asymmetric key for client data.

[0200] In one scenario, the client and / or the storage system can have more than one key. The various configurations disclosed herein can be applied to block, object, file storage systems, etc. Those of ordinary skill in the art should understand when reading this disclosure that unencrypted data can refer to data encrypted with a key available to the storage system (e.g., the storage system can view the data in plaintext).

[0201] In some scenarios, the metadata (such as deduplication information) created by a metadata generator can be more general than that used for deduplication. For example, keyword indexes for text data can be created with the deduplication information. The keyword indexes can be used by the storage system to perform search operations on encrypted data, including returning any found information to the client to be decrypted.

[0202] In another example, the deduplication information can be used to create an event detector for video information. The event detector can create pointers and / or information related to the detected events. The event detector can provide additional classification of the detected events. The storage system can perform a search operation against the metadata to identify portions of data that meet certain criteria for reducing the I / O workload. Various other classification, tagging localization, and other techniques can benefit from these operations with or without deduplication.

[0203] The present invention can be a system, method, and / or computer program product at any possible technical detail integration level. The computer program product can include a computer-readable storage medium having computer-readable program instructions thereon for causing a processor to execute aspects of the present invention.

[0204] A computer-readable storage medium can be a tangible device that can retain and store instructions for use by an instruction execution device. The computer-readable storage medium can be, for example, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of the computer-readable storage medium includes the following: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disc (DVD), a memory stick, a floppy disk, a mechanical encoding device such as a punch card or a raised structure in a groove having instructions recorded thereon, and any suitable combination of the foregoing. As used herein, a computer-readable storage medium should not be construed as being a transitory signal per se, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., an optical pulse passing through an optical fiber cable), or an electrical signal transmitted through a wire.

[0205] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to a corresponding computing / processing device via a network (e.g., the Internet, a local area network, a wide area network, and / or a wireless network), or to an external computer or external storage device. The network can include copper transmission cables, optical transmission fibers, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in a computer-readable storage medium within the corresponding computing / processing device.

[0206] The computer-readable program instructions for carrying out operations of the present invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state-setting data, configuration data for integrated circuits, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages such as Smalltalk, C++, etc. and procedural programming languages such as the "C" programming language or similar programming languages. The computer-readable program instructions may be executed entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer, or entirely on the remote computer or server. In the latter case, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider). In some embodiments, an electronic circuit, including, for example, a programmable logic circuit, a field-programmable gate array (FPGA), or a programmable logic array (PLA), can execute the computer-readable program instructions by utilizing the state information of the computer-readable program instructions to personalize the electronic circuit so as to perform aspects of the present invention.

[0207] The present invention has been described with reference to the flowcharts and / or block diagrams of methods, apparatuses (systems), and computer program products according to embodiments of the present invention. It should be understood that each block of the flowcharts and / or block diagrams, and combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer-readable program instructions.

[0208] These computer-readable program instructions can be provided to a processor of a computer or other programmable data processing apparatus to produce a machine, such that the instructions executed via the processor of the computer or other programmable data processing apparatus create a means for implementing the functions / acts specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium, which instructions cause a computer, a programmable data processing apparatus, and / or other devices to work in a particular manner, so that the computer-readable storage medium storing the instructions includes a manufacture comprising instructions implementing aspects of the functions / acts specified in one or more blocks of the flowchart and / or block diagram.

[0209] The computer-readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device, such that a series of operation steps are performed on the computer, other programmable apparatus, or other device to produce a computer-implemented process, so that the instructions executed on the computer, other programmable apparatus, or other device implement the functions / acts specified in one or more blocks of the flowchart and / or block diagram.

[0210] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of the possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagram may represent a module, segment, or portion of instructions, which includes one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions noted in the blocks may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be accomplished as one step, executed concurrently, substantially concurrently, in partial or complete temporal overlap, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It should also be noted that each block of the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented by special purpose hardware-based systems that perform the specified functions or acts or combinations of special purpose hardware and computer instructions.

[0211] In addition, a system according to various embodiments may include a processor and logic integrated with and / or executable by the processor, the logic being configured to perform one or more of the processing steps described herein. Integrated with the processor means that the processor has logic embedded therein as hardware logic, such as an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), etc. Executable by the processor means that the logic is hardware logic; software logic such as firmware, a part of an operating system, a part of an application; or some combination of hardware and software logic that can be accessed by the processor and configured to cause the processor to perform some functions when executed by the processor. The software logic may be stored in local and / or remote memory of any memory type known in the art. Any processor known in the art may be used, such as a software processor module and / or a hardware processor such as an ASIC, an FPGA, a central processing unit (CPU), an integrated circuit (IC), a graphics processing unit (GPU).

[0212] It will be clear that the different features of the foregoing systems and / or methods may be combined in any manner, thereby creating multiple combinations from the description presented above.

[0213] It should also be understood that embodiments of the present invention may be provided in the form of services on behalf of customer deployments to provide services on demand.

[0214] The description of the various embodiments of the present invention has been presented for purposes of illustration, but is not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope of the described embodiments. The terms used herein have been chosen to best explain the principles of the embodiments, the practical application, or the technical improvement of the technology found in the marketplace, or to enable those of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A computer-implemented method, comprising: Receiving deduplication information at a storage system, wherein the storage system has access to the deduplication information to perform operations, and wherein the deduplication information includes signatures associated with portions of client data; Receiving client data encrypted with a client key, wherein the client key is not available to the storage system; and Deduplicating data chunks stored in the storage system against chunks of client data, wherein the client data chunks are selected from the client data for deduplication using the deduplication information.

2. The computer-implemented method according to claim 1, wherein, The deduplication information is encrypted with a shared key, wherein the storage system includes data chunks encrypted with the shared key, wherein the shared key is available to the storage system, and wherein deduplication is performed on data chunks encrypted with a key selected from the group consisting of the client key and the shared key.

3. The computer-implemented method according to claim 2, wherein, The non-deduplicated client data chunks encrypted with the client key are stored in the storage system.

4. The computer-implemented method according to any one of claims 1-3, wherein, The storage system stores metadata for the data chunks, wherein the metadata includes a key identifier that uniquely identifies the key used to encrypt the data chunks.

5. The computer-implemented method according to any one of claims 1-3, comprising: Receiving data encrypted with a second client key; And Deduplicating the data chunks, wherein one or more of the data chunks are encrypted with a first client key and one or more of the data chunks are encrypted with a second client key.

6. A computer-implemented method, comprising: Generating deduplication information, wherein the deduplication information includes signatures associated with portions of client data; Encrypting the deduplication information with a shared key; Encrypting the client data with a client key; Sending the deduplication information and the client data to a storage system for deduplication; Requesting the client data from the storage system; and Receiving the client data as data chunks, wherein one or more of the data chunks are encrypted with the client key and one or more of the data chunks are encrypted with the shared key.

7. The computer-implemented method according to claim 6, wherein Requesting the client data from the storage system includes verifying the request with signed access control.

8. The computer-implemented method according to any one of claims 6 or 7, comprising requesting the shared key from the storage system.

9. The computer-implemented method according to any one of claims 6 to 7, wherein, The de-chunker receives the data as data chunks, wherein the de-chunker determines which key each data chunk is encrypted with.

10. The computer-implemented method according to any one of claims 6 to 7, comprising compressing the client data; and Encrypting the client data with a client key.

11. A computer program product for data deduplication in a cloud storage system and network, the computer program product comprising: A computer-readable storage medium that can be read by a processing circuit and stores instructions for execution by the processing circuit to perform the method according to any one of claims 1 to 10.

12. A computer storage medium stored on a computer-readable medium and loadable into the internal memory of a digital computer, comprising a software code portion for performing the method of any one of claims 1 to 10 when the program runs on the computer.

13. A computer system, comprising: A processor; and logic integrated with, executable by, or integrated with and executable by a processor, the logic being configured to perform the steps of the method according to any one of claims 1 to 10.

Images