System and method for encryption of distributed file system
By distributing metadata functions and using non-volatile memory in distributed file systems, combining distributed fault-tolerant address space and forward error correction solutions, the problem of insufficient scalability and data protection capabilities of traditional systems is solved, and high Improved performance and data recovery capabilities.
Patent Information
- Application Number
- CN202411264730.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2019-02-13
- Filing Date
- 2019-06-04
- Publication Date
- 2024-12-06
AI Technical Summary
Traditional distributed file systems have limitations in scalability and data protection, especially the number of metadata servers is limited, which cannot be effectively scaled, and data is easily lost in the event of failure.
Adopting a distributed file system, high-performance parallel submission is achieved by distributing metadata functions across multiple servers, using nonvolatile memory (such as solid-state drives) and virtual file systems, and through distributed fault-tolerant address space and front Protect data to error correction schemes, support expansion to thousands of nodes.
It realizes a high-performance and highly scalable distributed file system, which can quickly recover data in the event of failure, and improves data protection capabilities, avoiding data loss caused by a single point of failure.
Smart Images

Figure CN119089476A_ABST
Abstract
Description
[0001] This application is a divisional application of the PCT international application with application number 201980036527.0, application date June 4, 2019, and invention name “Encryption for Distributed File Systems” entering the national phase in China. The application number of the PCT international application is PCT / IB2019 / 000686, and the national phase entry date is November 30, 2020.
[0002] Priority claim
[0003] This application claims priority to U.S. Provisional Patent Application No. 62 / 682,198, filed on June 8, 2018, entitled “Encryption for Distributed File Systems,” and U.S. Patent Application No. 16 / 274,541, filed on February 13, 2019, entitled “Encryption for Distributed File Systems.” Technical Field
[0004] The present application relates to systems and methods for encryption of distributed file systems. Background Art
[0005] By comparing conventional methods of data storage with some aspects of the methods and systems of the present application as set forth in the remainder of this disclosure with reference to the accompanying drawings, the limitations and disadvantages of conventional methods will become apparent to those skilled in the art.
[0006] Incorporation by Reference
[0007] US Patent Application No. 15 / 243,519, entitled “Distributed Erasure Coded Virtual File System,” is incorporated herein by reference in its entirety. Summary of the invention
[0008] There is provided a method and system for encryption in a distributed file system substantially as set forth more fully in the claims and as shown and / or described in conjunction with at least one of the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS
[0009] Figure 1 Various exemplary configurations of distributed file systems according to various aspects of the present disclosure are shown.
[0010] Figure 2 Exemplary configurations of distributed file system nodes according to various aspects of the present disclosure are shown.
[0011] Figure 3 Another form of a distributed file system according to an exemplary implementation of the present disclosure is shown.
[0012] Figure 4An example of a computing device in an encrypted file system according to one or more embodiments of the present disclosure is shown.
[0013] Figure 5 An example of a cluster of computing devices is shown in accordance with one or more embodiments of the present disclosure.
[0014] Figure 6 is a flow chart illustrating an exemplary method for reading from a client in an encrypted file system.
[0015] Figure 7 is a flow chart illustrating an exemplary method for writing to a client in an encrypted file system.
[0016] Figure 8 An exemplary implementation in which two distributed fault-tolerant address spaces reside on multiple solid-state storage disks is shown.
[0017] Fig. 9 A forward error correction scheme that can be used to protect data stored in a non-volatile memory of a virtual file system according to an exemplary implementation of the present disclosure is shown. DETAILED DESCRIPTION
[0018] Traditionally, file systems use centralized control over metadata structures (e.g., directories, files, attributes, file contents, etc.). If a local file system is accessible from a single server and that server fails, the file system's data may be lost without further protection. To increase protection, some file systems (e.g., provided by NetApp) have used one or more pairs of controllers in an active-passive manner to replicate metadata on two or more computers. Other solutions have used multiple metadata servers in a clustered manner (e.g., provided by IBM GPFS, Dell EMC Isilon, Lustre, etc.). However, because the number of metadata servers in a traditional cluster system is limited to a small number, such a system cannot be expanded.
[0019] The system of the present invention can be applied to small clusters and can also be expanded to tens of thousands of nodes. An exemplary embodiment of a non-volatile memory (NVM) (e.g., flash memory in the form of a solid-state drive (SSD)) is discussed. The NVM can be divided into 4kB blocks and 128MB chunks. Extents can be stored in volatile memory, such as RAM for fast access, and can also be backed up by NVM storage devices. Extents can store pointers for blocks, for example, 256 pointers are stored for 1MB of data stored in a block. In other embodiments, larger or smaller memory partitions can also be used. The metadata functions in the present disclosure can be effectively distributed on many servers. For example, in the case of a "hot spot" for a specific part of the file system namespace where a large load is directed, the load can be distributed across multiple nodes.
[0020] Figure 1 Various exemplary configurations of distributed file systems according to various aspects of the present disclosure are shown. Figure 1 Shown is a local area network (LAN) 102 that includes one or more nodes 120 (indexed by integers from 1 to J, where j ≥ 1), and optionally includes (indicated by dashed lines): one or more dedicated storage nodes 106 (indexed by integers from 1 to M, where M ≥ 1), one or more compute nodes 104 (indexed by integers from 1 to N, where N ≥ 1), and / or an edge router that connects the LAN 102 to a remote network 118. The remote network 118 optionally includes one or more storage services 114 (indexed by integers from 1 to K, where K ≥ 1) and / or one or more dedicated storage nodes 115 (indexed by integers from 1 to L, where L ≥ 1).
[0021] 120 per node j (j is an integer, where 1≤j≤J) is a network computing device (e.g., a server, a personal computer, etc.) that includes a computer for directly accessing the device 104. n operating system and / or on the device 104 n A circuit for running a process (e.g., a client process) in one or more virtual machines running in.
[0022] Compute node 104 is a network device that can run a virtual front end without a virtual back end. Compute node 104 can run a virtual front end by incorporating single root input / output virtualization (SR-IOV) into a network interface card (NIC) and occupying a complete processor core. Alternatively, compute node 104 can run a virtual front end by routing networking via the Linux kernel network stack and using kernel process scheduling, so a complete core is not required. This will be very useful in cases where the user does not want to allocate a complete core for the file system or the network hardware is incompatible with the file system requirements.
[0023] Figure 2 An exemplary configuration of a node according to various aspects of the present disclosure is shown. The node includes a front end 202 and a driver 208, a memory controller 204, a back end 206, and an SSD agent 214. The front end 202 can be a virtual front end; the memory controller 204 can be a virtual memory controller; the back end 206 can be a virtual back end; and the driver 208 can be a virtual driver. As used in the present disclosure, a virtual file system (VFS) process is a process that implements one or more of the front end 202, the memory controller 204, the back end 206, and the SSD agent 214. Therefore, in an exemplary implementation, the resources (e.g., processing and memory resources) of the node can be shared between the client process and the VFS process. The VFS process can be configured to require a relatively small amount of resources to minimize the impact on the performance of the client application. The front end 202, the memory controller 204 and / or the back end 206 and / or the SSD agent 214 can run on a processor of the host 201 or a processor of the network adapter 218. For a multi-core processor, different VFS processes can run on different cores and can run different subsets of services. From the perspective of the client process 212, the interface with the virtual file system is independent of the specific physical machine on which the VFS process runs. The client process only requires the presence of the driver 208 and the front end 202 to provide services for them.
[0024] A node can be implemented as a single tenant server running directly on an operating system (e.g., a bare-metal server) or as a virtual machine (VM) and / or container (e.g., Linux container (LXC)) within a bare-metal server. The VFS can run within an LXC container as a VM environment. Therefore, inside the VM, the only thing that can run is the LXC container that includes the VFS. In a classic bare-metal environment, there are user space applications and the VFS runs in an LXC container. If the server is running other containerized applications, the VFS can run within an LXC container outside the management scope of the container deployment environment (e.g., Docker).
[0025] The node may be served by an operating system and / or a virtual machine monitor (VMM) (e.g., a hypervisor). The VMM may be used to create and run nodes on a host 201. Multiple cores may reside within a single LXC container running a VFS, and the VFS may run on a single host 201 using a single Linux kernel. Thus, a single host 201 may include multiple front ends 202, multiple memory controllers 204, multiple back ends 206, and / or one or more drivers 208. The driver 208 may run in kernel space outside the scope of the LXC container.
[0026] SR-IOV PCIe virtual functions can be used to run the network stack 210 in user space 222. SR-IOV allows PCIExpress isolation so that a single physical PCI Express can be shared on a virtual environment, and different virtual functions can be provided to different virtual components on a single physical server machine. The I / O stack 210 enables the VFS node to bypass the standard TCP / IP stack 220 and communicate directly with the network adapter 218. Portable Operating System Interface (POSIX) VFS functions can be provided to the VFS driver 208 through a lock-free queue. SR-IOV or full PCIe physical function addresses can also be used to run a non-volatile memory specification (NVMe) driver 214 in user space 222, thereby completely bypassing the Linux IO stack. NVMe can be used to access non-volatile storage media 216 connected via a PCI Express (PCIe) bus. The non-volatile storage medium 220 can be, for example, a flash memory in the form of a solid-state drive (SSD) or a storage-class memory (SCM) in the form of an SSD or a memory module (DIMM). Other examples may include storage-class memory technologies such as 3D-XPoint.
[0027] By coupling the physical SSD 216 with the SSD agent 214 and the network 210, the SSD can be implemented as a network device. Alternatively, the SSD can be implemented as a network-attached NVMe SSD 222 or 224 by using a network protocol such as NVMe-oF (NVMe over Fabrics). NVMe-oF can allow the use of redundant network links to access NVMe devices, thereby providing a higher level of fault tolerance. Network adapters 226, 228, 230, and 232 may include hardware acceleration for connecting to NVMe SSDs 222 and 224 to convert them into network NVMe-oF devices without using a server. NVMe SSDs 222 and 224 may each include two physical ports, and all data may be accessed through any one of these ports.
[0028] Each client process / application 212 can run directly on the operating system, or can run in a virtual machine and / or container served by the operating system and / or hypervisor. The client process 212 can read data from a storage device and / or write data to a storage device in the process of performing its main function. However, the main function of the client process 212 is not storage-related (i.e., the process only focuses on its data being reliably stored and able to be retrieved when needed, without focusing on when and how the data is stored and where). Exemplary applications that generate such processes include: email servers, web servers, office productivity applications, customer relationship management (CRM), animated video presentation, genomics computing, chip design, software construction, and enterprise resource planning (ERP).
[0029] The client application 212 can make a system call to the kernel 224 that communicates with the VFS driver 208. The VFS driver 208 places the corresponding request on the queue of the VFS front end 202. If there are several VFS front ends, the driver can load balance access to different front ends, ensuring that a single file / directory is always accessed via the same front end. This can be achieved by sharing the front end based on the ID of the file or directory. The VFS front end 202 provides an interface for routing file system requests to the appropriate VFS back end based on the bucket responsible for the operation. The appropriate VFS back end can be on the same host or it can be on another host.
[0030] The VFS backend 206 hosts several buckets, each of which services the file system requests it receives and performs tasks to otherwise manage the virtual file system (e.g., load balancing, logging, maintaining metadata, caching, moving data between tiers, removing stale data, correcting corrupted data, etc.).
[0031] The VFS SSD agent 214 handles the interaction with the corresponding storage device 216. This may include, for example, translating addresses and generating commands to the storage device (e.g., issued on a SATA, SAS, PCIe, or other suitable bus). Thus, the VFS SSD agent 214 operates as an intermediary between the storage device 216 and the VFS backend 206 of the virtual file system. The SSD agent 214 can also communicate with standard network storage devices that support standard protocols such as NVMe-oF (NVMe over Fabrics).
[0032] Figure 3 Another form of a distributed file system according to an exemplary implementation of the present disclosure is shown. Figure 3 In the example, unit 302 indicates that a virtual file system (such as the one described above) resides thereon. Figure 2Element 308 represents one or more physical storage devices 216 that provide long-term storage for the virtual file system.
[0033] like Figure 3 As shown, the physical storage device is organized into multiple distributed fault tolerant address spaces (DFRAS) 518. Each includes multiple groups of blocks 310, which in turn include multiple blocks 312. In some implementations, the blocks 312 are organized into groups of blocks 310 for convenience only, and this is not done in all implementations. Each block 312 stores committed data 316 (which can be in various states, as described below) and / or metadata 314 that describes or references the committed data 316.
[0034] Organizing storage device 308 into multiple DFRAS enables multiple (possibly all) nodes from the virtual file system (e.g., Figure 1 All nodes 1041–104 N , 1061–106 M and 1201–120 J ) high-performance parallel commit. In an exemplary implementation, each node of the virtual file system may own one or more of the multiple DFRASs and have exclusive read / commit access to the DFRASs it owns.
[0035] Each bucket has a DFRAS, so there is no need to coordinate with any other node when writing. Each bucket can be striped across many different chunks on many different SSDs, so each bucket and its DFRAS can choose what "chunk stripe" to write to based on many parameters, and once a chunk is assigned to that bucket, there is no need to coordinate in order to complete that operation. All buckets can effectively write to all SSDs without any coordination.
[0036] Each DFRAS is owned and accessible only by its owner bucket running on a particular node, so that each node of the VFS controls a portion of the storage device 308 without having to coordinate with any other node (e.g., except during initialization or during [re]assignment of the bucket holding the DFRAS after a node failure, which can be performed asynchronously with the actual read / commit of the storage device 308). Therefore, in such an implementation, each node can read / commit the DFRAS of its bucket independently of the work being done by other nodes without achieving any consistency when reading and committing to the storage device 308. In addition, in the event of a failure of a particular node, the fact that a particular node has multiple buckets enables its workload to be more intelligently and efficiently redistributed to other nodes (rather than having to assign the entire workload to a single node, which may create a "hot spot"). In this regard, in some implementations, the number of buckets can be large relative to the number of nodes in the system, so that any one bucket placed on another node can be a relatively small load. This allows fine-grained redistribution of the load of a failed node based on the performance and capacity of other nodes (e.g., a higher proportion of the buckets of a failed node can be given to nodes with higher performance and capacity).
[0037] To enable this, metadata may be maintained that maps each bucket to its current owning node so that reads and commits to storage 308 may be redirected to the appropriate node.
[0038] Load distribution is possible because the entire file system metadata space (e.g., directories, file attributes, ranges of content within files, etc.) can be divided (e.g., partitioned or sharded) into smaller uniform pieces (e.g., shards). For example, a large system with 30k servers can divide the metadata space into 128k or 256k shards.
[0039] Each such metadata shard is maintained in a "bucket". Each VFS node can be responsible for several buckets. When a bucket is serving metadata shards on a given backend, the bucket is considered "active" or the "leader" of the bucket. Typically, there are many more buckets than VFS nodes. For example, a small system with 6 nodes may have 120 buckets, while a large system with 1000 nodes may have 8k buckets.
[0040] Each bucket can be active on a small set of nodes, typically the 5 nodes that form a 5-tuple for that bucket. The cluster configuration keeps all participating nodes up to date on the 5-tuple assignment for each bucket.
[0041] Each quintuple monitors itself. For example, if the cluster has 10k servers and each server has 6 buckets, each server will only need to talk to 30 different servers to maintain the state of its bucket (6 buckets will have 6 quintuples, so 6*5=30). This number is much smaller than the case where the centrally monitored entity must monitor all nodes and maintain cluster-wide state. The use of quintuples enables performance to scale as the cluster becomes larger, because the nodes do not perform more work when the cluster size increases. This may cause the following disadvantages, that is, in "dumb" mode, a small cluster may actually generate more communication than a physical node, but this disadvantage can be overcome by sending only a single heartbeat between two servers and all the buckets they share (as the cluster grows, this will become only one bucket, but if you have a small cluster of 5 servers, it will just include all buckets in all messages, and each server will only talk to 4 other servers). The quintuple can be determined (i.e., reach consistency) using an algorithm similar to the Raft consensus algorithm.
[0042] Each bucket can have a set of compute nodes capable of running it. For example, five VFS nodes can run a bucket. However, at any given moment, only one node in the group is the controller / leader. Furthermore, for sufficiently large clusters, no two buckets share the same group. If there are only 5 or 6 nodes in the cluster, most buckets can share a backend. In fairly large clusters, there are many different node groups. For example, in the case of 26 nodes, there are more than possible five-node groups (i.e., 5-tuples).
[0043] All nodes in the group know and agree (i.e., reach consensus) which node is the actual active controller (i.e., the leader) for the bucket. A node accessing a bucket can remember (cache) the last node among the members of the group (e.g., five) that was the leader for the bucket. If it accesses the leader of the bucket, the leader of the bucket performs the requested operation. If it accesses a node that is not the current leader, the node instructs the leader to "redirect" the access. If access to the cached leader node times out, the node being contacted can try a different node for the same five-tuple. All nodes in the cluster share a common "configuration" of the cluster, which allows the node to know which server can run each bucket.
[0044] Each bucket may have a load / usage value that indicates the extent to which the bucket is used by applications running on the file system. For example, a server node with 11 lightly used buckets may receive metadata for another bucket to run before a server with 9 heavily used buckets, even if this results in an imbalance in the number of buckets used. The load value may be determined based on average response latency, the number of concurrently running operations, memory consumption, or other metrics.
[0045] Even when the VFS node is not faulty, reallocation may occur. If the system identifies that one node is busier than the others based on the tracked load metrics, the system can move one of its buckets (i.e., "failover") to another server that is less busy. However, before the buckets are actually relocated to different hosts, load balancing can be achieved by transferring writes and reads. Because each write can end on a different set of nodes (determined by DFRAS), nodes with higher loads may not be selected in the stripe where the data is being written. The system can also choose not to serve reads from high-load nodes. For example, a "degraded mode read" can be performed, in which blocks in a high-load node are rebuilt from other blocks of the same stripe. Degraded mode reads are reads performed by the remaining nodes in the same stripe, and data is rebuilt through fault protection. When the read latency is too long, a degraded mode read can be performed because the initiator of the read may assume that the node has failed. If the load is high enough to produce longer read latency, the cluster can return to reading the data from other nodes and using degraded mode reads to rebuild the required data.
[0046] Each bucket manages its own distributed erasure coding instance (i.e., DFRAS 518) and does not need to coordinate with other buckets to perform read or write operations. There may be thousands of concurrent distributed erasure coding instances working concurrently, each for a different bucket. This is a major part of the scaling performance because it effectively allows any large file system to be divided into independent parts that do not require coordination, thereby achieving high performance regardless of the scale.
[0047] Each bucket handles all file system operations that fall within its shard. For example, directory structures, file attributes, and file data ranges will fall within the jurisdiction of a specific bucket.
[0048] Any operation done by the frontend is initiated by finding out which bucket owns the operation. The backend leader and node for that bucket is then determined. This determination can be performed by trying the last known leader. If the last known leader is not the current leader, the node can then learn which node is the current leader. If the last known leader is no longer part of the bucket's quintuple, the backend will let the frontend know that it should go back to the configuration to find the members of the quintuple for that bucket. Distributing operations allows complex operations to be handled by multiple servers, rather than a single computer in a standard system.
[0049] If the cluster size is small (e.g., 5) and quintuples are used, there will be buckets that share the same group. As the cluster size grows, the buckets are redistributed so that no two groups are the same.
[0050] Encrypted storage is important for many different applications. For example, financial applications must be protected according to the Federal Information Processing Standard (FIPS), and healthcare applications must be protected according to the Health Insurance Portability and Accountability Act (HIPPA).
[0051] The stored data may need to be encrypted. For example, if one or more storage devices are compromised, encrypting the stored data can prevent recovery of the plaintext data.
[0052] It may also be desirable to have "on-the-fly" encryption for data transmitted over a network. For example, "on-the-fly" encryption prevents eavesdropping and man-in-the-middle attempts to tamper with data.
[0053] Figure 4 An example of a computing device 401 in an encrypted file system 400 is shown. The computing device includes a front end 403 and a back end 405. The front end 403 can encrypt and decrypt data on the client side as the data enters and leaves the system. The front end 403 is operable to encrypt data as it is written to a data file in the system 400. The front end 403 is also operable to decrypt data as it is read from a data file in the system 400.
[0054] The front end 403 encrypts data at least according to the file key 413. The file key 413 can be rotated by copying the data file. The file key 413 (e.g., one key per file) can be provided to the client for front-end encryption and decryption. The file key 413 can be generated by the file system 401 with a FIPS-approved secure random number generator. If the file key 413 is to be saved, the file key 415 can be encrypted by the file system key 415. In addition, the file system id and the index node (inode) id can be used as authentication data for the file key 413. The file key can be rotated by copying the file. When the file is copied, the content will be re-encrypted using the newly generated file key.
[0055] The file system keys 415 (e.g., one key per file system) never leave the boundaries of the file system. The file system keys 415 can be generated by a key management system / server (KMS). If the file system keys 415 are to be saved, they can be encrypted by a cluster key 417. In addition, the cluster id and file system id can be used as authentication data for the file system keys 415. The file system keys 415 can be rotated immediately by the file system 400. For example, the file system 400 can request the KMS to generate a new key for a specified file system. Newly created files will have their file keys 413 encrypted with the latest file system key 415, and updated files can have their file keys 413 re-encrypted with the latest file system key 415.
[0056] The file key 413 may be encrypted by the file system key 415. In addition, the file system key 415 may be encrypted by the cluster key 417. The file key 413 may be re-encrypted when the file system key 415 is rotated, and the file system key 415 may be re-encrypted when the cluster key 417 is rotated.
[0057] The backend includes a plurality of buckets 407a, 407b...407m. The computing device 401 is operably connected to a plurality of storage devices 409a, 409b, 409c, 409d...407n, such as SSDs. The number of buckets may be less than, greater than or equal to the number of storage devices.
[0058] Each bucket 407x in the plurality of buckets 407a, 407b...407m is operable to construct a fault protection stripe 411x comprising a plurality of storage blocks (e.g., block x1, block x2, and block x3). The plurality of storage blocks include encrypted data and error detection and / or error correction coding. For illustration, stripe 411a includes block a1, block a2, and block a3; stripe 411b includes block b1, block b2, and block b3; and stripe 411c includes block c1, block c2, and block c3. The number of blocks in a stripe may be less than, greater than, or equal to three. Each storage block of a particular stripe is located in a different storage device in the plurality of storage devices. All fault protection stripes 411 constructed by the plurality of buckets 407 in the back end 405 are in a common file system 400 associated with a file system key 415.
[0059] Figure 5An example of a cluster 501 of computing devices 401a, 401b, 401c ... 401p is shown. A cluster 501 of computing devices is associated with a cluster key 417. A cluster key 417 (e.g., one key per cluster) never leaves the boundaries of the cluster. The cluster key 417 can be generated by a KMS. The KMS can rotate cluster keys on the fly. Upon rotation, the KMS re-encrypts each file system key 415, and the old cluster key can be destroyed. The file system does not need to be aware of the rotation. When a client (e.g., computing device 401b or 401c) joins the cluster 501, it registers a public key 503 with the leader. Each client node generates a long-term public / private key pair 503 at startup.
[0060] The network encryption key can be negotiated by two nodes (e.g., computing device 401b or 401c) to encrypt the communication between the two nodes. Network interactions between nodes (including system nodes and client nodes) are called remote procedure calls (RPCs). RPCs can mark their parameters, output parameters, and return values as encrypted. RPCs can encrypt their argument payloads using pre-negotiated keys known only to the two RPC endpoints (e.g., nodes). As needed, each client can negotiate a session key 507 for each system node connected to it. A temporary key pair (using, for example, a temporary key pair generator 505) of a long-term key signature (e.g., long-term keys 503b and 503c) generated by each node (e.g., computing node 401b or 401c) when it is added to the cluster 501 can be used to negotiate the session key with perfect forward secrecy.
[0061] System-to-system keys can be used to reduce the time before a newly started server can connect to other servers. After joining, the newly started server can receive the key from the file system leader. The system-to-system key is encrypted and saved in the configuration. An RPC from server A to server A can use the configuration represented as, for example, "config.serverkeys[A]".
[0062] To authenticate a client, an administrator can generate a token or certificate that can be used by the client to join the cluster. The token can be time-limited and can be configured by the administrator based on a challenge-response mechanism. The token can identify a specific machine (e.g., my IP address or instance ID). The token can identify a client class, allowing many machines to join using the same token. Tokens can also be revoked by the administrator. When a token is revoked, new clients will not be able to join using that token, and existing clients that joined using that token will be disconnected unless the existing client has a newer valid token.
[0063] To authorize mounts, administrators can configure which clients (specific machines or groups of machines) can mount which file systems. This authorization is verified when a client attempts to mount a file system. Clients that do not mount the file system can be prevented from accessing the file system. Administrators can specify that the access allowed is read-only, write-only, or read / write. Administrators can also allow clients to mount as root (e.g., rootsquash or non-rootsquash).
[0064] Each file system can have a file name encryption key that can be derived from the file system key. When a client mounts the file system, the client can receive the file name encryption key. The client can use symmetric encryption to create a file with an encrypted file name.
[0065] File data in object storage (e.g., tiered data used for backup and restore) can be encrypted in the same manner as on-disk data. When uploaded to object storage, file data can contain file system parameters, such as encrypted file system keys. For example, the file system key can be encrypted with a special "backup-specific cluster key" available via KMS.
[0066] If the cluster is dropped and a new cluster is created by downloading the file system from the object store, the downloaded file system may be decrypted by the file system key with the backup private key and re-encrypted with the cluster's new cluster key.
[0067] When mounting a snapshot to Simple Storage Service (S3), the file system key can be re-encrypted with a new cluster key. The "stow" command can include the new cluster key ID and the credentials to access it from KMS. The existing file system key can be re-encrypted with the new cluster key and saved in S3 along with the remaining file system metadata. Restoring a mounted snapshot requires access to the same cluster key specified when mounting.
[0068] The system page cache can be encrypted to reduce the possibility that memory (e.g., kernel memory) will be accessed by other processes. To combat this unauthorized access, encrypted data received by the kernel page cache cannot be decrypted immediately when it is received over the network. Because the kernel will store the encrypted data, malicious applications will not be able to access it. When the process that actually owns the file accesses the encrypted data, the kernel driver will decrypt the data with the help of the front end while copying the data to the memory space of the process. Therefore, the system memory will be encrypted until it is legitimately accessed.
[0069] The encryption and decryption processes can take advantage of hardware acceleration. When performing encryption, the file system can look for hardware that can accelerate the encryption process. The file system can sort which hardware is more efficient (e.g., standard CPU opcodes, acceleration within the CPU, encryption on a coprocessor, or a network card) and use the most efficient way to encrypt the decrypted data.
[0070] In addition to the above encryption, self-encrypting drives (SEDs) can also be used. On top of the standard encryption code in the file system, SEDs can encrypt the already encrypted data they are receiving.
[0071] Encryption of stored data can use symmetric encryption keys, where the same key is used to encrypt and decrypt the data. Depending on the configuration of the file system, there are several key management options.
[0072] The same key can be used for all drives (e.g., SSD's). However, if this key is compromised, all drives can be read.
[0073] Different keys can be used for each SSD. These keys can be managed by the file system, which randomly picks and stores different encryption keys for each SSD. Alternatively, these keys can be managed by the client's own key management system, which handles creating and storing encryption keys. The file system can authenticate these keys with the client's KSM and request the correct key for each SSD. The SSD retains the key and can use it until the next reboot.
[0074] Another option for key management is to have each chunk on the SSD have a different key. This allows supporting multiple clients on the same file system. File systems with different distributed fault tolerant address spaces can access the same SSD agent. Each file system can set a different key for each DFRAS. The SSD may not exchange base keys at boot time. Instead, different keys can be registered with the SSD at boot time. If the drive requires an encryption key to be given for each IO, each key can have an index used by each IO. When DFRAS accesses the SSD, DFRAS can remember the index of the encryption key of its SSD and transmit it with the IO request. Therefore, the file system can share several workloads and different DFRAS running on the same SSD without compromising security.
[0075] HIPPA requires three independent copies of data and encryption at rest. Three independent copies can be achieved by using a composite object store backend that writes to two independent object stores. Data sent to the object store can be encrypted first. The file system can be configured to take a snapshot at a predefined time defined by the application (e.g., every 30 minutes) and transfer the snapshot to the object store. By integrating self-encrypting SSD drives with KSM, the cluster can have encryption at rest.
[0076] Figure 6 6 is a flow chart illustrating an exemplary method for reading from a client in an encrypted file system. In block 601, the client opens a file for read access. In block 603, after verifying that the client can access the file system, the system securely sends the file key to the client's computing device. In block 605, the client front end uses the file key to decrypt the data read from the file.
[0077] In block 607, a session key is negotiated using a temporary key pair signed by a long-term key from the client's computing device and the other computing device. In block 609, data is encrypted using the session key. In block 611, the data (encrypted by the session key) is transmitted from the client's computing device to the other computing device.
[0078] In block 613, the file is closed. In block 615, the file key and session key are cleared from memory.
[0079] Figure 7 7 is a flow chart illustrating an exemplary method for writing to a client in an encrypted file system. In block 701, a session key is negotiated using a temporary key pair signed by a long-term key from a computing device of a client and another computing device. In block 703, data is encrypted using the session key. In block 705, the data (encrypted by the session key) is transmitted from the other computing device to the computing device of the client.
[0080] In box 707, the client opens the file for write access. In box 709, after verifying that the client can access the file system, the system securely sends the file key to the client's computing device. In box 711, the client front end uses the session key to decrypt the received data.
[0081] In box 713, the client front end uses the file key to encrypt the decrypted data. In box 715, the data (encrypted by the file key) is written to the open file on the client's computing device.
[0082] In block 717, the file is closed. In block 719, the file key and session key are cleared from memory.
[0083] Figure 8 An exemplary implementation in which two distributed fault-tolerant address spaces reside on multiple solid-state storage disks is shown.
[0084] Block 910 1,1 –910 D,C Can be organized into multiple chunk stripes 9201–920 S (S is an integer). In an exemplary implementation, each chunk stripe 920 s (s is an integer where 1≤s≤S) is individually protected using forward error correction (e.g., erasure coding). Thus, any particular chunk stripe 920 may be determined based on the desired level of data protection. s Block 910 d,c The number of
[0085] For the purpose of illustration, it is assumed that each block stripe 920 s The block 910 includes N=M+K (where each of N, M and K is an integer). d,c , then N blocks 910 d,c The M of the blocks 910 can store data bits (typically binary digits or "bits" for current storage devices), and the N blocks 910 d,c K of them can store protection bits (typically bits). Then, for each stripe 920 s , the virtual file system can assign N chunks 910 from N different failure domains d,c .
[0086] As used herein, a "fault domain" refers to a group of components where the failure of any single one of those components (the component loses power, becomes unresponsive, and / or the like) can result in the failure of all of the components. For example, if a rack has a single top-of-rack switch, the failure of that switch will reduce connectivity to all components on that rack (e.g., compute, storage, and / or VFS nodes). Thus, to the rest of the system, it is equivalent to all components on that rack failing together. A virtual file system according to the present invention may include fewer fault domains than chunk 910.
[0087] In an exemplary implementation where the nodes of the virtual file system are connected and powered in a fully redundant manner, where each such node has only a single storage device 906, the failure domain may be exactly that single storage device 906. Thus, in the exemplary implementation, each chunk stripe 920 s including residing on storage devices 9061–906 D A plurality of chunks 910 on each of N d,c(So D is greater than or equal to N). Fig. 9 An example of such an implementation is shown.
[0088] exist Figure 8 In FIG. 1 , D=7, N=5, M=4, K=1, and the storage is organized into two DFRAS. These numbers are for illustration only and are not intended to be limiting. For illustration purposes, the three block stripes 920 of the first DFRAS are arbitrarily called. The first block stripe 9201 consists of blocks 910 1,1 ,910 2,2 ,910 3,3 ,910 4,5 and 910 5,6 The second block stripe 9202 consists of block 910 3,2 ,910 4,3 ,910 5,3 ,910 6,2 and 910 7,3 The third block stripe 9203 consists of block 910 1,4 ,910 2,4 ,910 3,5 ,910 5,7 and 910 7,5 composition.
[0089] Although in Figure 8 In the simple example of , D=7 and N=5, but in actual implementations, D can be much larger than N (e.g., greater than an integer multiple of 1 and possibly many orders of magnitude higher), and these two values can be selected so that the probability that any two chunk stripes 920 of a single DFRAS reside on the same set of N storage devices 906 (or more generally, reside on the same set of N failure domains) is below a desired threshold. In this way, any single storage device 906 d The failure of D (or more generally, any single failure domain) will result in at most (possibly based on the expected statistics determined by: the chosen values of D and N, the size of the N storage devices 906, and the arrangement of the failure domains) the loss of any particular stripe 920 s A block 910 b,c Furthermore, a double failure will cause most stripes to lose at most a single chunk 910 b,c , only a small number of stripes (determined based on the values of D and N) will lose two chunks in any particular stripe 920s (eg, the number of stripes with two failures may be exponentially less than the number of stripes with one failure).
[0090] For example, if each storage device 906 d is 1 TB, and each block is 128 MB, then storage device 906 dThe failure of will cause (perhaps based on the expected statistics determined by: the selected values of D and N, the size of the N storage devices 906, and the arrangement of the failure domains) 7812 (=1TB / 128MB) chunk stripes 920 to lose one chunk 910. For each such affected chunk stripe 920 s , appropriate forward error correction algorithms and block stripes 920 can be used s The other N-1 chunks are used to quickly reconstruct the lost chunk 910 d,c In addition, due to the affected 7812 chunk stripes on all storage devices 9061–906 D The uniform distribution on the reconstruction loss is 910, so the 7812 blocks d,c will involve (possibly determining the desired statistics based on: the selected values of D and N, the size of the N storage devices 906, and the arrangement of the failure domains) 906 from the storage devices 9061-906 D Each of the 2 storage devices reads the same amount of data (i.e., the burden of rebuilding the lost data is evenly distributed across all storage devices to recover very quickly from failures).
[0091] Next, turn to storage device 9061–906 D In the case of two concurrent failures in the same fault domain (or more generally, two concurrent failures in the same fault domain), since each DFRAS has chunk stripes 9201–920 S On all storage devices 9061–906 D Evenly distributed, with only a very small number of chunk stripes 9201–920 S will lose two of its N chunks. The virtual file system can operate based on indicating chunk stripes 9201-920 S With storage device 9061–906 D 906 to quickly identify such two lost chunk stripes. Once such two lost chunk stripes are identified, the virtual file system can prioritize rebuilding those two lost chunk stripes before starting to rebuild the single lost chunk stripe. The remaining chunk stripes will have only a single lost chunk, and for them (the vast majority of affected chunk stripes), the two storage devices 906 d Concurrent failure with only one storage device 906 d A similar principle applies to three concurrent failures (in the case of two concurrent failures, the number of chunk stripes with three failed blocks will be even less than the number with two failed blocks), and so on. In an exemplary implementation, chunk stripes 920 may be used based on the number of concurrent failures. s The number of losses in the control execution block stripe 920 sThis can be achieved by, for example, controlling the rate at which the reconstructed reads and commits are performed, the rate at which the reconstructed FEC calculations are performed, the rate at which the reconstructed network messages are transmitted, etc.
[0092] Fig. 9 FIG. 9 is a diagram showing a forward error correction scheme that can be used to protect data stored in a non-volatile memory of a virtual file system according to an exemplary implementation of the present disclosure. FIG. 9 is a diagram showing a storage block 902 of a block stripe 9301-9304 of a DFRAS 1,1 –902 7,7 .exist Fig. 9 In the protection scheme of , five blocks of each stripe are used for data bit storage, and two blocks of each stripe are used for protection bit data storage (i.e., M=5, K=2). Fig. 9 The protection bits are calculated using the following formulas (1) to (9):
[0093]
[0094] therefore, Fig. 9 The four stripes 9301-9304 in are part of a multi-strip (in this case, four stripes) FEC protection domain, and the loss of any two or fewer blocks in any of the block stripes 9301-9304 can be recovered from various combinations using equations (1)-(9) above. For comparison, an example of a single-strip protection domain is if D11, D22, D33, D44, D54 are protected only by P1, and D11, D22, D33, D44, D54 are all written to stripe 9301 (9301 is a single-strip FEC protection domain).
[0095] According to an exemplary implementation of the present disclosure, multiple computing devices are communicatively coupled to each other via a network, and each of the multiple computing devices includes one or more of the multiple storage devices. Multiple fault-tolerant address spaces are distributed on the multiple storage devices, so that each of the multiple fault-tolerant address spaces spans the multiple storage devices. Each of the multiple fault-tolerant address spaces is organized into multiple stripes (e.g., Figure 8 and Fig. 9 Each one or more of the plurality of stripes is a plurality of forward error correction (FEC) protection domains (e.g., Figure 8 Each of the plurality of stripes may include a plurality of storage blocks (e.g., a plurality of 912). Each block of a particular one of the plurality of stripes may reside on a different storage device in the plurality of storage devices. A first portion of the plurality of storage blocks (e.g., Fig. 9 902 of strip 9301 1,2 –902 1,6The amount of five) can be used to store data bits, and the second part of the plurality of storage blocks (eg, Fig. 9 The two 902 strips 9301 1,1 and 902 1,7 The amount of ) can be used to store protection bits that are calculated at least in part based on the data bits.
[0096] A plurality of computing devices may be operable to sort a plurality of stripes. The sorting may be used to select which of the plurality of stripes is used to perform the next commit operation on the one of the plurality of fault-tolerant address spaces. The sorting may be based on how many protected and / or unprotected storage blocks exist in each of the plurality of stripes. For any particular stripe in the plurality of stripes, the sorting may be based on a bitmap stored on a plurality of storage devices together with the particular stripe in the plurality of stripes. The sorting may be based on how many blocks of stored data are currently in each of the plurality of stripes. The sorting may be based on committing read and write overhead to each of the plurality of stripes. At any given time, each of the fault-tolerant address spaces may be owned by only one of the plurality of computing devices, and each of the plurality of fault-tolerant address spaces may be read and written by only its owner. Each computing device may own a plurality of fault-tolerant address spaces. A plurality of storage devices may be organized into a plurality of fault domains. Each of the plurality of stripes may span a plurality of fault domains. Each fault-tolerant address space may span all of the multiple fault domains such that upon a failure of any particular one of the multiple fault domains, the workload for rebuilding the lost data is distributed between each of the other fault domains in the multiple fault domains. The multiple stripes may be distributed across the multiple fault domains such that in the event of two concurrent failures in the multiple fault domains, the probability that two blocks of any particular one of the multiple stripes reside on the failed two of the multiple fault domains is exponentially less than the probability that only one block of any particular one of the multiple stripes resides on the failed two of the multiple fault domains.
[0097] The plurality of computing devices are operable to first rebuild any stripe in the plurality of stripes that has two failed blocks, and then rebuild any stripe in the plurality of stripes that has only one failed block. The plurality of computing devices are operable to perform reconstruction of the stripe in the plurality of stripes that has two failed blocks at a higher rate (e.g., a greater percentage of CPU clock cycles dedicated to reconstruction, a greater percentage of network transmission opportunities dedicated to reconstruction, and / or the like) than the reconstruction rate of the stripe in the plurality of stripes that has only one failed block. The plurality of computing devices are operable to determine a rate for rebuilding any particular lost block based on how many other blocks have been lost by the same stripe in the plurality of stripes in the event of a failure in one or more of the failure domains. One or more of the plurality of failure domains include a plurality of storage devices. Each of the plurality of FEC protection domains may span a plurality of stripes in the plurality of stripes.
[0098] Multiple strips can be organized into multiple groups (e.g. Figure 8 The block strips 9201–920 shown in FIG. S ), wherein each of the plurality of groups includes one or more of the plurality of stripes, and the plurality of computing devices are operable to, for each group, sort the one or more of the plurality of stripes in the group. The plurality of computing devices are operable to: perform continuous commit operations on a selected group of the plurality of groups until the one or more of the plurality of stripes in the group no longer satisfies a determined criterion, and select a different one of the plurality of groups when the selected group of the plurality of groups no longer satisfies the determined criterion. The criterion may be based on how many blocks are available for new data to be written.
[0099] Although the present method and / or system has been described with reference to a specific implementation, it will be appreciated by those skilled in the art that various changes may be made and equivalents may be substituted without departing from the scope of the present method and / or system. In addition, many modifications may be made to adapt a particular situation or material to the teachings of the present invention without departing from the scope of the present invention. Therefore, it is intended that the present method and / or system is not limited to the specific implementation disclosed, but that the present method and / or system will include all implementations falling within the scope of the appended claims.
[0100] As used herein, the terms "circuit" and "circuitry" refer to physical electronic components (i.e., hardware), as well as any software and / or firmware ("code") that can configure, be executed by, or otherwise be associated with the hardware. As used herein, for example, a particular processor and memory may include a first "circuitry" when executing a first line or more lines of code, and may include a second "circuitry" when executing a second line or more lines of code. As used herein, "and / or" means any one or more of the items in the list added by "and / or". As an example, "x and / or y" means any element in the three-element set {(x), (y), (x, y)}. In other words, "x and / or y" means "one or both of x and y". As another example, "x, y and / or z" means any element in the seven-element set {(x), (y), (z), (x, y), (x, z), (y, z), (x, y, z)}. In other words, "x, y, and / or z" means "one or more of x, y, and z." As used herein, the term "exemplary" means serving as a non-limiting example, instance, or illustration. As used herein, the terms "for example" and "for example" include a list of one or more non-limiting examples, instances, or illustrations. As used herein, a circuit system is "operable" to perform a function so long as the circuit includes the necessary hardware and code (if necessary) to perform the function, regardless of whether the performance of the function is disabled or not enabled (e.g., by a user-configurable setting, factory trim, etc.).
Claims
1. A system comprising: A front end for encrypting data as it enters the system; A backend that builds a fail-safe stripe across multiple storage devices, where: The plurality of storage devices are distributed such that there are at most a permitted number of storage devices within any particular node of the plurality of nodes.
2. The system according to claim 1, wherein: The front end is used to decrypt the data as it leaves the system.
3. The system according to claim 1, wherein: The front end is used to encrypt the data according to the file key.
4. The system according to claim 3, wherein: The file keys are rotated when files are copied.
5. The system according to claim 3, wherein: All failsafe stripes built from multiple buckets in the backend are associated with a file system key.
6. The system according to claim 5, wherein: The file key is encrypted by the file system key.
7. The system according to claim 5, wherein: The file key is re-encrypted when the file system key is rotated.
8. The system according to claim 1, wherein: The system includes a cluster of computing devices, and wherein the cluster of computing devices is associated with a cluster key.
9. The system according to claim 8, wherein: When the system joins the cluster of computing devices, the front end registers a long-term key with the leader of the cluster.
10. The system according to claim 9, wherein: Prior to transmitting the data, a session key is negotiated using a temporary key pair signed by the long-term key.
11. A method comprising: encrypting the data as it enters the computing device via the front end; Building fault protection stripes across multiple storage devices via the backend; and The plurality of storage devices are distributed such that there are at most a permitted number of storage devices within any particular node of the plurality of nodes.
12. The method according to claim 11, wherein: The method comprises: The data is decrypted via the front end as it leaves the computing device.
13. The method according to claim 11, wherein: The front end encrypts the data according to the file key.
14. The method according to claim 13, wherein: The file keys are rotated when files are copied.
15. The method according to claim 13, wherein: All failsafe stripes built from multiple buckets in the backend are associated with a file system key.
16. The method according to claim 15, wherein: The file key is encrypted by the file system key.
17. The method according to claim 15, wherein: The file key is re-encrypted when the file system key is rotated.
18. The method according to claim 11, wherein: A cluster of computing devices is associated with a cluster key.
19. The method according to claim 18, wherein: The method comprises: When the computing device joins the cluster of computing devices, a long-term key is registered with a leader of the cluster.
20. The method according to claim 19, wherein: The method comprises: Prior to transmitting the data, a session key is negotiated using a temporary key pair signed by the long-term key.
