System and method for hierarchical data policies in distributed storage systems

Abstract

The present application relates to a system and method for hierarchical data policies in a distributed storage system. The system comprises: a plurality of storage devices; and circuitry configured to indicate when a retention period is reached, wherein: each storage device is operable to store one or more data blocks, each data block is associated with a deletion state, when the retention period is reached, each data block not in a highest deletion state is incremented to a next higher deletion state, a data block is reset to a lowest deletion state if the data block is accessed, each storage device of the plurality of storage devices is assigned to one of an upper tier and a lower tier, and the system comprises a processor configured to read a deletion state associated with a particular data block in a storage device assigned to the upper tier, and if the particular data block is in the highest deletion state, demote the particular data block to a storage device assigned to the lower tier.

Description

[0001] This application is a divisional application of the Chinese national phase application, filed on July 13, 2020, with international application number PCT / IB2018 / 001280 and entitled "Hierarchical Data Strategy for Distributed Storage Systems". The Chinese national phase application entered the national phase on October 5, 2018, with application number 201880086411.3 and entitled "Hierarchical Data Strategy for Distributed Storage Systems".

[0002] Cross-reference of related applications

[0003] The full text of U.S. Patent Application No. 15 / 243,519 entitled “Distributed Erasure-Encoded Virtual File System” is incorporated herein by way of its entirety. Technical Field

[0004] This application relates to a hierarchical data strategy for distributed storage systems, and more specifically, to a system and method for a hierarchical data strategy in a distributed storage system. Background Technology

[0005] By comparing such a method with some aspects of the present method and system set forth in the remainder of this disclosure with reference to the accompanying drawings, the limitations and disadvantages of conventional methods of data storage will become apparent to those skilled in the art. Summary of the Invention

[0006] On one hand, this application provides a system comprising: a plurality of storage devices; and circuitry configured to indicate when a retention period is reached, wherein: each storage device is operable for storing one or more data blocks, each data block being associated with a deletion state; when the retention period is reached, each data block not in the highest deletion state is incremented to the next higher deletion state; if a data block is accessed, the data block is reset to the lowest deletion state; each of the plurality of storage devices is allocated to one of an upper layer and a lower layer; and the system includes a processor configured to read the deletion state associated with a specific data block in the storage device allocated to the upper layer, and if the specific data block is in the highest deletion state, demote the specific data block to the storage device allocated to the lower layer.

[0007] On the other hand, this application provides a system comprising: a plurality of storage devices; and circuitry configured to indicate when a retention period is reached, wherein: each storage device is operable for storing one or more data blocks, each data block being associated with a deletion state; when the retention period is reached, each data block not in the highest deletion state is incremented to the next higher deletion state; if a data block is accessed, the data block is reset to the lowest deletion state; each data block is associated with a degradation state; circuitry is configured to indicate when a degradation cycle is reached; when the degradation cycle is reached, each data block not in the highest degradation state is incremented to the next higher degradation state; if a write access is made to a particular data block, the particular data block is reset to the lowest degradation state; and each data block in the highest degradation state is degraded to a storage device allocated to a lower layer.

[0008] On the other hand, this application provides a method comprising: assigning a deletion state to each of a plurality of data blocks; resetting a data block to the lowest deletion state if a data block is accessed; measuring a retention period; after the retention period, incrementing each of the plurality of data blocks not yet in the highest deletion state to the next higher deletion state; associating each of the plurality of data blocks with a degradation state; measuring a degradation cycle; after the degradation cycle, incrementing each data block not in the highest degradation state to the next higher degradation state; resetting a specific data block to the lowest degradation state if a write access is made to a specific data block; and degrading each data block in the highest degradation state to a storage device allocated to a lower layer.

[0009] Methods and systems for hierarchical data strategies in distributed storage systems are provided, the methods and systems being substantially illustrated and / or described in conjunction with at least one figure, and set forth more fully in the claims. Attached Figure Description

[0010] Figure 1 Various example configurations of virtual file systems according to aspects of this disclosure are shown.

[0011] Figure 2 An example configuration of a virtual file system node is shown according to aspects of this disclosure.

[0012] Figure 3 This illustrates another representation of a virtual file system according to an example implementation of this disclosure.

[0013] Figure 4 An example of a hierarchical memory is shown, according to an example implementation of this disclosure.

[0014] Figure 5 This is an example table showing the state of the extension area according to an example implementation of this disclosure.

[0015] Figure 6 This is an example flowchart illustrating the degradation of the extended region or the deletion of read cache entries according to an exemplary implementation of the present invention. (See reference) Figure 6 The flowchart is shown.

[0016] Figure 7 This is an example flowchart illustrating the deletion and reading of an extended area in a cache memory according to an embodiment of the present invention.

[0017] Figure 8 This illustrates an example implementation where two of the distributed fault-tolerant address spaces reside on multiple solid-state storage disks.

[0018] Figure 9 A forward error correction scheme according to an example implementation of this disclosure is shown, which can be used to protect data stored in non-volatile memory of a virtual file system. Detailed Implementation

[0019] Traditionally, file systems use centralized control over metadata structures (e.g., directories, files, attributes, file content). If a local file system is accessible from a single server and that server fails, the data in that file system may be lost due to the lack of further protection. To increase protection, some file systems (such as the NetApp file system) use one or more pairs of controllers in an active-passive manner to replicate metadata across two or more computers. Other solutions use multiple metadata servers in a clustered manner (e.g., provided by IBM GPFS, Dell EMC Isilon, Lustre, etc.). However, such systems cannot scale because the number of metadata servers in traditional clustered systems is limited to a small number.

[0020] The system in this disclosure is suitable for small clusters and can also scale to many tens of thousands of nodes. Example embodiments regarding non-volatile memory (NVM) (e.g., flash memory in the form of solid-state drives (SSDs)) are discussed. The NVM can be divided into 4kB “blocks” and 128MB “chunks”. “Extents” can also be stored in volatile memory, such as RAM for fast access, or can be backed up by the NVM storage. The extent can store pointers to blocks, for example, 256 pointers to 1MB of data stored in a block. In other embodiments, larger or smaller memory partitions can also be used. The metadata functionality in this disclosure can be efficiently distributed across many servers. For example, in the case of a large load pointing to a “hotspot” in a specific part of the file system namespace, this load can be distributed across multiple nodes.

[0021] Figure 1 Various example configurations of a virtual file system (VFS) according to aspects of this disclosure are shown. Figure 1 The diagram shows a local area network (LAN) 102, which includes one or more VFS nodes 120 (integer indices from 1 to J, j≥1), and optionally includes (shown in dashed lines): one or more dedicated storage nodes 106 (integer indices from 1 to M, M≥1); one or more compute nodes 104 (integer indices from 1 to N, N≥1); and / or an edge router connecting LAN 102 to a remote network 118. The remote network 118 optionally includes one or more storage services 114 (integer indices from 1 to K, K≥1); and / or one or more dedicated storage nodes 115 (integer indices from 1 to L, for L≥1).

[0022] 120 per VFS node j (j is an integer, where 1 ≤ j ≤ J) is a network computing device (e.g., a server, personal computer, etc.) that includes circuitry for running the VFS process and optional client processes (or directly in device 104). n On the operating system and / or on device 104 n (In one or more virtual machines running in the system).

[0023] Compute node 104 is a networked device that can run a VFS frontend without a VFS backend. Compute node 104 can run the VFS frontend by placing the SR-IOV into the NIC and using a full processor core. Alternatively, compute node 104 can run the VFS frontend by networking via the Linux kernel networking stack and using kernel process scheduling, thus not requiring a full kernel. This is useful if the user does not want to allocate a full kernel for VFS, or if the networking hardware is incompatible with VFS requirements.

[0024] Figure 2An example configuration of a VFS node according to aspects of this disclosure is shown. The VFS node includes a VFS frontend 202 and drive 208, a VFS memory controller 204, a VFS backend 206, and a VFS SSD agent 214. As used in this disclosure, a “VFS process” is a process that implements one or more of the following: VFS frontend 202, VFS memory controller 204, VFS backend 206, and VFS SSD agent 214. Therefore, in the example implementation, VFS node resources (e.g., processing and memory resources) can be shared between client processes and VFS processes. VFS processes can be configured to require relatively few resources to minimize the impact on client application performance. VFS frontend 202, VFS memory controller 204, and / or VFS backend 206 and / or VFS SSD agent 214 can run on the processor of host 201 or on the processor of network adapter 218. For multi-core processors, different VFS processes can run on different cores and can run different subsets of services. From the perspective of client process 212, the interface with the virtual file system is independent of the specific physical machine running the VFS process. The client process only needs the existence of drive 208 and front end 202 to provide services to them.

[0025] A VFS node can be implemented as a single-tenant server running directly on an operating system (e.g., bare metal), or as a virtual machine (VM) and / or container (e.g., Linux Containers (LXC)) within a bare metal server. A VFS can run as a VM environment within an LXC container. Therefore, within a VM, only an LXC container containing the VFS can run. In a classic bare metal environment, user-space applications exist, and the VFS runs within an LXC container. If the server is running other containerized applications, the VFS may run within an LXC container outside the management scope of a container deployment environment (e.g., Docker).

[0026] A VFS node can be serviced by an operating system and / or a Virtual Machine Monitor (VMM) (e.g., a hypervisor). The VMM can be used to create and run VFS nodes on host 201. Multiple kernels can reside within a single LXC container running VFS, and VFS can run on a single host 201 using a single Linux kernel. Therefore, a single host 201 can include multiple VFS front-ends 202, multiple VFS memory controllers 204, multiple VFS back-ends 206, and / or one or more VFS drives 208. VFS drives 208 can run in kernel space outside the scope of the LXC container.

[0027] A single root input / output virtualization (SR-IOV) PCIe virtualization function can be used to run a network stack 210 in user space 222. SR-IOV allows for the isolation of PCI Express, enabling the sharing of a single physical PCI Express across virtual environments and providing different virtual functions to different virtual components on a single physical server computer. I / O stack 210 enables VFS nodes to bypass the standard TCP / IP stack 220 and communicate directly with network adapter 218. A portable operating system interface for unix (POSIX) VFS functions can be provided to VFS driver 208 via lock-free queuing. SR-IOV or the full PCIe physical function address can also be used to run a non-volatile memory Express (NVMe) driver 214 in user space 222, completely bypassing the Linux I / O stack. NVMe can be used to access non-volatile storage media 216 connected via the PCI Express (PCIe) bus. The non-volatile storage medium 216 may be, for example, flash memory in the form of a solid-state drive (SSD) or a storage class memory (SCM) in the form of a solid-state drive (SSD) or a memory module (DIMM). Other examples may include storage class memory technologies such as 3D-XPoint.

[0028] By coupling the physical SSD 216 with the SSD agent 214 and networking 210, the SSD can be implemented as a networked device. Alternatively, the SSD can be implemented as an NVMe SSD 242 or 244 attached to a network using a networking protocol such as NVMe-oF (structure-based NVMe). NVMe-oF allows access to NVMe devices using redundant network links, providing a higher level of resilience. Network adapters 226, 228, 230, and 232 can include hardware acceleration for connecting to the NVMe SSDs 242 and 244, transforming them into networked NVMe-oF devices without the need for a server. The NVMe SSDs 242 and 244 can each include two physical ports, and all data can be accessed through either of these ports.

[0029] Each client process / application 212 may run directly on the operating system, or it may run in a virtual machine and / or container served by the operating system and / or hypervisor. Client process 212 may read data from and / or write data to memory during the execution of its primary functions. However, the primary function of client process 212 is storage-independent (i.e., the process is only concerned with the reliable storage of its data and its retrieval when needed, regardless of the location, timing, or manner of data storage). Example applications that give rise to such a process include: email servers, web servers, office productivity applications, customer relationship management (CRM), animated video rendering, genomic computing, chip design, software building, and enterprise resource planning (ERP).

[0030] Client application 212 can make system calls to kernel 224, which communicates with VFS driver 208. VFS driver 208 places the corresponding requests on the queue of VFS frontend 202. If several VFS frontends exist, the driver may load balance access to different frontends to ensure that a single file / directory is always accessed through the same frontend. This can be done by “sharding” the frontends based on file or directory IDs. VFS frontend 202 provides an interface for routing filesystem requests to the appropriate VFS backend based on the bucket responsible for the operation. The appropriate VFS backend can be on the same host or on another host.

[0031] The VFS backend 206 hosts multiple buckets, each of which serves the file system requests it receives and performs tasks to additionally manage the virtual file system (e.g., load balancing, logging, maintaining metadata, caching, moving data between layers, deleting outdated data, correcting corrupted data, etc.).

[0032] VFS SSD agent 214 handles interactions with the corresponding storage device 216. This may include, for example, address translation and generating commands to be issued to the storage device (e.g., on SATA, SAS, PCIe, or other suitable buses). Therefore, VFS SSD agent 214 acts as an intermediary between the VFS backend 206 of the virtual file system and the storage device 216. SSD agent 214 can also communicate with standard network storage devices that support standard protocols, such as NVMe-oF (structure-based NVMe).

[0033] Figure 3 This illustrates another representation of a virtual file system according to an example implementation of this disclosure. Figure 3In this context, element 302 represents the memory resources (e.g., DRAM and / or other short-term memory) and processing resources (e.g., x86 processors, ARM processors, NICs, ASICs) of the various nodes (compute, storage, and / or VFS) where the virtual file system resides, for example, as mentioned above regarding... Figure 2 As described. Element 308 represents one or more physical storage devices 216 that provide long-term storage for a virtual file system.

[0034] like Figure 3 As shown, the physical storage is organized into multiple Distributed Failure Resilience Address Spaces (DFRAS) 518. Each DFRAS comprises multiple blocks 310, and each block 310 comprises multiple blocks 312. Organizing blocks 312 into blocks 310 is only a convenience in some implementations and may not be done in all implementations. Each block 312 stores committed data 316 (which may take various states discussed below) and / or metadata 314 describing or referencing the committed data 316.

[0035] Organizing storage device 308 into multiple DFRAS enables high-performance parallel commits from many (potentially all) nodes of the virtual file system (e.g., Figure 1 All nodes 1041–104 N 1061–106 M and 1201–120 J (Concurrent commits can be executed in parallel). In one example implementation, each node of the virtual file system can own one or more corresponding DFRASs and have exclusive read / commit access to the DFRAS it owns.

[0036] Each bucket has its own DFRAS, and therefore does not require coordination with any other nodes when writing to it. Each bucket may create stripes on many different blocks across many different SSDs, so each bucket and its DFRAS can select the "block stripe" to write to based on many parameters, and once the block is assigned to that bucket, writing can proceed without coordination. All buckets can effectively write to all SSDs without coordination.

[0037] Each DFRAS is owned and accessed only by its owner bucket running on a specific node, allowing each node in the VFS to control a portion of storage device 308 without having to coordinate with any other nodes (except during initialization or after a node failure when [re]distributing the buckets holding DFRAS, which can be asynchronous with the actual read / commit operations of storage device 308). Therefore, in such an implementation, each node can read / commit DFRAS to its bucket independently of what other nodes are doing, without needing any consensus when reading and committing to storage device 308. Furthermore, the fact that a particular node owns multiple buckets in the event of a failure allows for a smarter and more efficient redistribution of its workload to other nodes (rather than distributing the entire workload to a single node, which could create "hotspots"). In this respect, in some implementations, the number of buckets may be large relative to the number of nodes in the system, making any single bucket potentially a relatively small load placed on another node. This allows for fine-grained redistribution of the load on the failed node based on the capabilities and capacity of other nodes (e.g., a node with more capabilities and capacity may receive a higher percentage of the failed node's bucket).

[0038] To allow this operation, metadata can be maintained that maps each bucket to the node it currently owns, making it possible to redirect reads and commits to storage device 308 to the appropriate node.

[0039] Load balancing is possible because the entire file system metadata space (e.g., directories, file attributes, ranges of content within files, etc.) can be broken down (e.g., shredded or fragmented) into uniform pieces (e.g., “fragments”). For example, a large system with 30k servers can divide its metadata space into 128k or 256k fragments.

[0040] Each such metadata fragment can be stored in a "bucket". Each VFS node may be responsible for several buckets. When a bucket is serving a metadata fragment on a given backend, that bucket is considered the "active" or "leader" of that bucket. Typically, there are more buckets than VFS nodes. For example, a small system with 6 nodes may have 120 buckets, while a large system with 1,000 nodes may have 8,000 buckets.

[0041] Each bucket can operate on a small group of nodes, typically a 5-tuple consisting of 5 nodes. Cluster configuration ensures that all participating nodes keep up-to-date with the 5-tuple allocation for each bucket.

[0042] Each quintuplet monitors itself. For example, if there are 10,000 servers in the cluster, and each server has 6 buckets, then each server will only need to communicate with 30 different servers to maintain the state of its bucket (6 buckets will have 6 quintuplets, so 6 * 5 = 30). This is far less than a centralized entity having to monitor all nodes and maintain cluster-wide state. Using quintuplets allows performance to scale with larger clusters because nodes don't perform more work as the cluster size increases. A potential drawback is that in "dumb" mode, a small cluster can actually generate more communication than the physical nodes, but this can be overcome by having them share all buckets and sending only a single heartbeat between two servers (this changes to only one bucket as the cluster grows, but if you have a small cluster of 5 servers, it will only include all buckets in all messages, and each server will only communicate with the other 4 servers). Quintuplets can use algorithms similar to the Raft consensus algorithm to make decisions (i.e., reach consensus).

[0043] Each bucket may have a set of compute nodes that can run it. For example, five VFS nodes can run one bucket. However, at any given time, only one node in the group is the controller / leader. Furthermore, for sufficiently large clusters, no two buckets share the same group. If there are only 5 or 6 nodes in the cluster, most buckets can share a backend. In a fairly large cluster, there may be many different groups of nodes. For example, in the case of 26 nodes, there are over 64,000 There are 5 possible five-node pairs (i.e., quintuples).

[0044] All nodes in the group know and agree (i.e., reach consensus) which node is the actual activity controller (i.e., leader) for that bucket. A node accessing a bucket may remember ("caches") the last node among the group's (e.g., five) members that was the leader of that bucket. If it accesses the bucket leader, the bucket leader performs the requested action. If the node it accesses is not the current leader, the node instructs the leader to "redirect" the access. If accessing the cached leader node times out, the contacting node can try using another node from the same five-tuple. All nodes in the cluster share a common cluster "configuration," which allows nodes to know which server can run each bucket.

[0045] Each bucket can have a load / utilization value, which indicates how much the application running on the file system is using the bucket. For example, even if the number of buckets used is unbalanced, a server node with 11 buckets with low utilization can receive another metadata bucket to run before a server with 9 buckets with high utilization. The load value can be determined based on average response latency, the number of concurrent operations, memory consumption, or other metrics.

[0046] Even if a VFS node is not faulty, redistribution can still occur. If the system identifies a node as busier than others based on tracked load metrics, it can move one of its buckets (i.e., "failover") to a less busy server. However, load balancing can be achieved by shifting writes and reads before the bucket is actually relocated to a different host. Since each write may end on a different group of nodes determined by DFRAS, a node with a higher load may not be selected for the stripe where data is written. The system may also choose not to provide reads from high-load nodes. For example, a "degraded-mode read" can be performed, where blocks from a high-load node are reconstructed from blocks in the same stripe. Degraded-mode reads are reads performed by the remaining nodes in the same stripe and the data is reconstructed through failover. Degraded-mode reads may be performed when read latency is too high, as the initiator of the read may assume the node is down. If the load is high enough to create even higher read latency, the cluster can recover by reading the data from other nodes and using degraded-mode reads to reconstruct the required data.

[0047] Each bucket manages its own Distributed Erasure Coding instance (i.e., DFRAS 518) and does not need to cooperate with other buckets to perform read or write operations. There can be thousands of concurrent Distributed Erasure Coding instances working simultaneously, each for a different bucket. This is an integral part of scalability performance because it effectively divides any large file system into independent parts that do not require coordination, thus providing high performance regardless of the scale of the expansion.

[0048] Each bucket handles all file system operations belonging to its fragments. For example, directory structure, file attributes, and file data ranges will fall under the jurisdiction of a specific bucket.

[0049] An operation performed by any frontend begins by determining which bucket owns the operation. Then, the backend leader and nodes for that bucket are determined. This determination can be performed by trying the most recently known leader. If the most recently known leader is not the current leader, the node may know which node is the current leader. If the most recently known leader is no longer part of the bucket's 5-tuple, the backend will inform the frontend that it should go back to the configuration to find the bucket's 5-tuple members. This distribution of operations allows complex operations to be handled by multiple servers instead of a single computer in a standard system.

[0050] If the cluster size is small (e.g., 5) and 5-tuples are used, multiple buckets will share the same group. As the cluster size grows, the buckets will be redistributed so that no two groups are the same.

[0051] Additionally, various embodiments of the present invention may also include a method for classifying files into one of multiple layers, or "layers." This method can determine whether the storage location of a file has changed, whether the file should be cached, and / or whether the file should be prefetched. Layering may depend on various metadata of the file, such as the time or location of the last modification or access, extension, size, etc., and / or user-defined policies (e.g., per directory, per file, per device, etc.).

[0052] Various embodiments of this disclosure may have “local” SSD storage that can be directly connected to the server, and the lower-level “object storage” may be, for example, non-SSD storage. Other embodiments may have more than two tiers. Typically, different embodiments may have different storage speeds in different tiers, and / or the memory in the lower tiers may have greater latency due to its greater distance from the server controlling access to the memory in each tier.

[0053] Figure 4 An example of a hierarchical memory implemented according to this disclosure is shown. (See also...) Figure 4 The diagram illustrates a host 400 that can communicate with the various layers of memory. For example, a given memory might belong to one of four layers: 402, 404, 406, and 408.

[0054] In the four-layer memory, the memory in the main layer 402 can be non-volatile memory (NVM) with the highest performance. The main layer 402 is not bound to a specific type of NVM. Instead, it can support various types of NVM. For example, while flash memory is currently available, other technologies such as FeRAM can be used. The main layer 402 can be local or network-attached storage (NAS) with low-latency support, such as Ethernet, RDMA, NVM Express, etc.

[0055] The next layer below the primary layer can be a low-endurance layer (404). This layer can use an NVM that may be cheaper than the primary NVM, and therefore has slower access times. It can be configured as the primary layer locally via direct connection, or it can be configured as a NAS.

[0056] The next layer below the low-endurance layer 404 can be the object storage layer 406. The object storage layer 406 can also be local or NAS, with an internal bus such as PCIe, SAS, or SATA, and circuitry similar to the main NVM layer that allows access to the NVM over the network. Some object storage layers 406 may use, for example, magnetic technology, but any suitable technology can be used.

[0057] The next lower layer can be referred to as archiving layer 408. This layer can be used for data that is rarely accessed. For example, data may be required for regulatory reasons, and laws or regulations require entities to retain data for a certain period of time. Therefore, archiving layer 408 storage can be located in a remote location, and it may be acceptable to spend hours or days to access the data. For example, archiving layer 408 storage could be cloud-based or stored as tape backups in a building.

[0058] Main layer 402 can store all metadata. Main layer 402 can also be used for two types of caching. Since magnetic media can have high latency when reading data, a read cache for magnetic storage may exist in a lower layer. Therefore, a prefetch algorithm can determine which files are being read and use various heuristics to perform prefetching, along with other techniques, to ensure that future reads of the same type of data will be very fast. Main layer 402 can also use a write cache for writing data to avoid write amplification to NVM, optimize network throughput, and / or reduce latency.

[0059] Various embodiments of this disclosure may allow memory management of one or more of the main layer 402, low-durability layer 404, object storage layer 406, and archive layer 408. A portion of memory management may include, for example, allocating state for data in one or more layers.

[0060] Figure 5 This is an example table illustrating the state of the extension region according to an example implementation of this disclosure. (See also...) Figure 5 The table 500 is shown with 8 rows, where each row represents a possible state of the extended area. The 8 possible states can be identified by 3-digit labels (which may be called "labels").

[0061] The state of each extended region in each layer of memory can be stored in "tags" in the registry, which is a data structure that stores metadata for all files. Each entry in the registry can reference an extended region, and a tag can be part of the entry. Therefore, a tag can point to an extended region of data. The registry can be stored in RAM for fast access.

[0062] Tags can refer to, for example, the three states used to determine when to downgrade a file, the three states used to determine when to delete the read cache, and whether the extended region is entirely on the SSD (“local”). These eight states can be represented by three-bit tags. A downgrade state can be labeled, for example, Local 0, Local 1, Local 2, and deleting the read cache can be labeled, for example, Reserved 0, Reserved 1, Reserved 2. When an extended region is in the Local 2 state, it may be a candidate for downgrading to a lower-level storage. Similarly, when an extended region is in Local 2, it may be a candidate for deleting its data from the read cache.

[0063] A degradation cycle can be the period during which the system waits for writes to the storage area before downgrading (degrading) the extended area to object storage. This may be related to the rate at which files are updated, whereby if a file is not modified at least during the degradation cycle, it can be assumed that it is unlikely to be modified in the near future. Degradation cycles can be counted by appropriate hardware circuitry using a suitable clock signal.

[0064] The retention period can be defined as the time a file is held in the read cache if it is not read before being deleted. Therefore, it relates to the frequency of data reading from the file. If the file is not read after the retention period, it can be assumed that the extended area is unlikely to be read in the near future. The retention period can be counted using appropriate clock signals via suitable hardware circuitry. If the extended area is read, it will be retrieved again from object storage (or a portion thereof) and stored in the read cache again.

[0065] Backpressure is related to the amount of free space a bucket has. If a bucket has too little free space to perform new write operations, it invokes the object storage's backpressure mechanism, allowing the object storage to release data. The first data released might be read cache blocks whose retention period has expired, and then data blocks can be degraded based on the date / time they were last touched.

[0066] Figure 6 This is an example flowchart illustrating the degradation of a memory block according to an exemplary implementation of the present invention. (Refer to...) Figure 6The flowchart 600 shows that a 4K block in the extended region can be in the current state 602. Regarding degradation, the lowest state can be local 0, then the next higher state can be local 1, and the highest state can be local 2. In 601, whenever a write operation is performed on a 4kB block, the degradation state of that block can be reset to the lowest state.

[0067] At 604, a check can be performed to see if the degradation cycle expired without a write to the extended area. If not, the timer can be checked later by returning to 604. If so, the status can be checked at 606 to see if the degradation status is in the highest state, Local 2. If so, there is no need to increment the degradation status to a higher state. If not, the degradation status is incremented at 608 before returning to the current state 602. For example, Local 0 status is incremented to Local 1 status, or Local 1 status is incremented to Local 2 status.

[0068] A block in local state 2 indicates that it has not been updated in at least one downgrade cycle and possibly two downgrade cycles, and therefore can be downgraded to object storage.

[0069] A similar process can be followed to remove blocks from the read cache.

[0070] Figure 7 This is a flowchart illustrating an example implementation of deleting a memory block from a read cache according to an embodiment of the present invention. (See also...) Figure 7 The flowchart 700 shows that a 4K block within the extended region can be in the current state 702. Regarding the deletion of a read cache entry, the lowest state can be Reserved 0, then the next higher state can be Reserved 1, and the highest state could be Reserved 2. At 701, whenever a 4kB block is read, the state of that block for deleting a read cache entry can be reset to the lowest state.

[0071] At 704, a check can be performed to see if the degradation period has expired without accessing the block. If not, this check can be performed later by returning to 704. If yes, a check can be performed at 706 to see if the deletion state is in the highest state, Reserve 2. If yes, there is no need to increment the deletion state to a higher state. If no, at 708, the deletion state is incremented before returning to the current state 702. For example, Reserve 0 is incremented to Reserve 1, or Reserve 1 is incremented to Reserve 2.

[0072] Retention periods and degradation periods can be different or the same. Each of the retention and degradation periods can be dynamically changed, for example, based on the number of memory accesses, the rate of memory accesses during the existing retention and degradation periods, and / or other parameters that the memory management system may track.

[0073] A block in the Retention 2 state indicates that it has not been read for at least one retention period and may not have been read for two retention periods, so it can be removed from the read cache.

[0074] Because the registry resides in fast storage devices, such as RAM, the status of blocks within the registry can be viewed quickly. Therefore, finding blocks that need to be moved to lower-level storage (degraded) or cached data blocks that can be safely removed from the read cache can be done very efficiently without reading and / or writing to data structures on slower storage.

[0075] Typically, when data is stored on an SSD, it may be in a "local" state. "Read cache" data is data stored in object storage and also cached on the SSD for performance improvement. "Object" data can be data in object storage, not data cached on the SSD. The state may not be specific to the extension, but rather to each of the 256 4k blocks within that extension, as data can be overwritten.

[0076] As an example of different blocks within an extended region, a "local" extended region can be downgraded to object storage. The state of the extended region (all data blocks within the extended region) can then change from "local" to "read cache" because the local copy is still available. If some 4k blocks in the extended region are modified, those 4k blocks will again be in the "local" state. If some read cache blocks are no longer on the system due to backpressure or server failure, the state of those blocks will change to "object". Therefore, a single extended region may have different blocks in different states.

[0077] Regarding object storage, object names on an object storage system can be derived from the extent ID, but the name can also contain a version ID. A version ID might be needed for two reasons. First, the extent may be downgraded several times (because it may be overwritten), so there can be multiple versions of the extent, especially if snapshots are also implemented.

[0078] Secondly, when the object storage system is consistent, and if two PUT (write) calls are made to the same object in a short sequence, the first PUT might actually save the data, rather than the second PUT following the first. If the order of PUTs is not preserved, it's possible that data from the first PUT might be saved while data from the second PUT is not.

[0079] As an example, if degradation occurs along with the first PUT, and further writes are made on the same extended area, then the server fails before the first PUT is acknowledged, the new server might have already considered those writes. However, since the degradation PUT hasn't been acknowledged, it might be "held back." Therefore, the file system will assume that subsequent PUTs on objects are accepted by the changed blocks, and thus the file system will be out of sync with the object storage. However, this might not happen if an index is incremented after each PUT. The first PUT will increment a counter and log it, and after a failover, the next PUT will be made. Because it's unknown whether the first PUT was successful, but there are different indexes, in the worst case, the file system might "waste" objects, but without losing data integrity.

[0080] This allows the file system to efficiently manage very large amounts of data, while keeping all relevant metadata in RAM and making the selection of the extended area for PUT very simple. When the backpressure mechanism is activated, it also allows the system to very efficiently determine which blocks are good candidates for degradation or deletion of the read cache.

[0081] In contrast to block-based systems with a file system layered on top of them, various embodiments of this disclosure may be able to perform efficient prefetching and thus know the storage location of files because the file system manages the files. However, a file system operating independently of a block system may not be able to associate block locations with files. Moreover, if a block must be retrieved from a low-performance medium, the block system may not know where the next read prefetch will occur because the file system manages the placement of data, but the block system typically manages the prefetching of data independently of the file system.

[0082] Various embodiments of this disclosure may also downgrade or delete the entire extended region, for example, depending on how many blocks in the extended region are ready to be downgraded or deleted from the read cache.

[0083] In various embodiments of this disclosure, an extended region can notify other extended regions that it is reading data from object storage, allowing these extended regions to choose to prefetch data from object storage. For example, an extended region can notify the next five extended regions. Once an extended region receives notification of a read occurring in an increasingly nearby extended region, it can decide to notify more distant extended regions so that more data can be read in parallel.

[0084] Various embodiments of this disclosure can provide other management mechanisms. For example, specific files or portions of files may be marked as "fixed" so that these files are always stored on the SSD without degradation. Specific files or portions of files may be marked as "always read cached" so that even if they are degraded, there will be read cached copies of these files. If a node failure occurs, the files can be retrieved from object storage immediately after the node is recovered.

[0085] CLI commands can be provided to retrieve (hydrate) the file list, demote the file list to object storage, or clear the cache of the file list read from the SSD to free up SSD space. Each file in the file list can be an entire file or a part of a file.

[0086] Various embodiments of this disclosure can support a variety of backends for object storage, including backend storage services. However, it should be noted that some backend storage services may have a very limited number of buckets. For example, some backend storage services may allow only about 1MB of objects per bucket. The SWIFT storage system can be one such backend storage service. However, since various embodiments of this disclosure can support more extents than 1MB, the 1MB object limit may be too small for some purposes.

[0087] Therefore, to circumvent the limit on the number of objects per bucket, a large number of buckets can be created. For example, creating a 1M bucket might support 10... 12 One object. Creating more buckets will support more objects.

[0088] The file system can then use the object's hash to select a bucket. The first few bits of the key hash can be used to determine which bucket to use for that object.

[0089] If one of the buckets is full, the first option might be to start with a new list of buckets while still remembering the old list. An index of the bucket configuration can then be stored on the extension area to efficiently retrieve the new configuration. This representation allows the extension area to maintain a counter of the bucket configurations and has the advantage of not requiring data movement.

[0090] The second option could be to ask the user to double the number of buckets and add another bit to the "fragmentation". New objects might be written based on the larger bucket size, and depending on the extra bit added, there might be an ongoing process of migrating objects to the new bucket. As long as the background process is still running, the file system will first try to retrieve (read) the object using the extra bit at its final position; if it doesn't exist, it will try to read the object from the old bucket. This might have the advantage of the file system using a single backend configuration, but the disadvantage is that it must move data between buckets every time the number of buckets doubles.

[0091] Very large namespaces that can span more than a single object storage space can also be extended. For example, ten 50PB object storage systems can be aggregated to support a 1 / 2Exabyte system. This aggregation can be used, for example, with the first option of storing the key in an extension area that indicates which backend object it is actually mapped to.

[0092] One approach could be to have a 1:1 mapping between the extension area and the object. However, some backend object stores may not handle small objects well, and a 1MB file might still be considered small.

[0093] Various implementations of the system may allow for optimal object sizes, such as 4MB-32MB, to facilitate higher throughput in the object storage system. These sizes will be multiples of the extension size, which can be 1MB.

[0094] Then, during downgrading and retrieval, the file system can use the new size as the object size. Objects can all be aligned to the requested size, so if the object size is 10MB, all objects can start with a 10MB aligned offset.

[0095] The first extended region that is aligned can be called the "controller" of the object, and the bucket that owns the extended region can collect all the data from all extended regions, including data from other buckets and possibly from other nodes, and write that data.

[0096] If the registry scan is not one of the controller's extended areas, and a decision is made that degrading may be necessary, it will send an RPC to the controller to initiate the degrading process. Similarly, if data needs to be retrieved from object storage, it can be read through the controller's extended areas and distributed to the remaining buckets.

[0097] Various embodiments of this disclosure have higher internal throughput than object storage systems, thus allowing the use of some embodiment resources so that object storage solutions can provide greater overall throughput performance.

[0098] During initial configuration, the file system can decide how to allocate extensions among themselves. If the file system determines early on that object storage requiring larger objects is already being utilized, adjacent extensions can be grouped into the same bucket, thus reducing network traffic associated with aggregating extensions into a single object.

[0099] As disclosed, embodiments of this disclosure may include multiple storage devices, each of which may be assigned to an upper or lower layer. Each storage device may be used to store one or more data blocks, and each data block may be associated with a state. A processor may be configured to read the current state associated with a particular data block and, depending on the current state, perform one of the following: demote each particular data block to a second storage device in the lower layer; and remove each particular data block from the read cache.

[0100] Multiple degradation states may exist, and the system may include appropriate circuitry to calculate degradation cycles. When the degradation cycle count ends, each data block not in the highest degradation state can be incremented to the next higher degradation state. If a write access is made to any data block, that data block is reset to the lowest degradation state. Any data block in the highest degradation state can be degraded to the next lower level.

[0101] Data blocks can be, for example, 4KB.

[0102] Multiple deletion states may exist, and the system may include appropriate circuitry to calculate the retention period. When the retention period count expires, each data block not in the highest deletion state can be incremented to the next higher deletion state. If any data block is read from, those blocks can be reset to the lowest deletion state. Any data block in the highest deletion state can be removed from the read cache.

[0103] Embodiments of this disclosure may further include performing memory management on each of a plurality of storage devices, each of which may be assigned to an upper or lower layer, wherein each of the plurality of storage devices includes data blocks. Each data block may have a corresponding state. The processor may read the state of each data block and demote the data block to a lower layer (if indicated by the state of the data block), or may delete the data block from the read cache (if indicated by the state of the data block).

[0104] Multiple degradation states may exist, and the method may include calculating degradation cycles. When the count of degradation cycles ends, each data block not in the highest degradation state can be incremented to the next higher degradation state. If a write access is made to any data block, those blocks can be reset to the lowest degradation state. Any data block in the highest degradation state can be degraded to the next lower level.

[0105] Multiple deletion states can exist, and the method can include calculating a retention period. When the retention period expires, each data block not in the highest deletion state can be incremented to the next higher deletion state. If any data block is read from, those blocks can be reset to the lowest deletion state. Any data block in the highest deletion state can be deleted from the read cache.

[0106] Figure 8 This illustrates an example implementation where two of the distributed fault-tolerant address spaces reside on multiple solid-state storage disks. (Tie 510) 1,1 Up to 510 D,C It can be organized into multiple chunk stripes 5201-520 S (S is an integer). In the example implementation, forward error correction (e.g., erasure coding) is used to protect each chunk stripe 520 separately. s (s is an integer, where 1 ≤ s ≤ S). Therefore, any particular chunk stripe 520 can be determined based on the desired data protection level. s Block 510 in d,c The quantity.

[0107] For illustrative purposes, assume each block strip is 520. s Includes N = M + K (where each of N, M, and K is an integer) blocks 510 d,c Then there are N blocks 510 d,c The M bytes can store data codes (typically binary numbers or "bits" for current storage devices) and N blocks 510 d,c K bits in the array can store protection codes (again, typically bits). Then, the virtual file system can allocate 520 bits to each stripe. s Assign N blocks from N different fault domains 510 d,c .

[0108] As used herein, a "fault domain" refers to a set of components, where a failure in any one component (a component losing power, becoming unresponsive, etc.) could cause all components to fail. For example, if a rack has a single top-of-rack switch, a failure of that switch would disconnect all components on that rack (e.g., compute, storage, and / or VFS nodes). Therefore, for the rest of the system, this is equivalent to all components on that rack failing together. The virtual file system according to this disclosure may include fewer fault domains than block 510.

[0109] In the example implementation where each virtual file system node is connected and powered to a single storage device 506 in a fully redundant manner for each such node, the fault domain can be limited to that single storage device 506. Therefore, in the example implementation, each block stripe 520 s Including storage devices 5061-506 D Multiple blocks on each of the N ones 510 d,c (Therefore, D is greater than or equal to N). An example of this implementation is... Figure 7 It is displayed in the middle.

[0110] exist Figure 8 In this configuration, D=7, N=5, M=4, K=1, and the storage devices are organized into two DFRAS. These numbers are for illustrative purposes only and are not intended as limitations. The three block stripes 520 of the first DFRAS are arbitrarily selected for illustrative purposes. The first block stripe 5201 consists of block 510. 1,1 510 2,2 510 3,3 510 4,5 and 510 5,6 Composition; the second block strip 5202 is composed of block 510 3,2 510 4,3 510 5,3 510 6,2 and 510 7,3 Composition; the third block strip 5203 is composed of block 510 1,4 510 2,4 510 3,5 510 5,7 and 510 7,5 composition.

[0111] Although in the actual implementation Figure 8 In the simple example, D=7 and N=5, but D can be much larger than N (e.g., an integer multiple of 1, and possibly several orders of magnitude higher), and two values ​​can be chosen so that the probability of any two block stripes 520 of a single DFRAS residing on the same group of N storage devices 506 (or more generally, on the same group of N fault domains) is below a desired threshold. In this way, any single storage device 506 d (Or more generally, a failure in any single fault domain) will result in (expected statistics may be determined based on: selected values ​​of D and N, the size of N storage devices 506, and the arrangement of the fault domain) any particular stripe 520 s At most one block 510 b,c The loss. Furthermore, a double failure will cause the vast majority of stripes to lose at most a single block of 510. b,cAnd only a small number of bands (determined based on the values ​​of D and N) will be drawn from any given band 520. s Two segments are lost (for example, the number of stripes in two failures may be exponentially reduced compared to the number of stripes in one failure).

[0112] For example, if each storage device has 506 d If it's 1TB and each block is 128MB, then the storage device is 506. d The failure will result in (expected statistics may be determined based on the following: selected values ​​of D and N, the size of N storage devices 506, and the arrangement of the failure domain) 7812 (=1TB / 128MB) block stripes 520 losing one block 510. For each such affected block stripe 520 s Appropriate forward error correction algorithms and chunking stripes 520 can be used. s The other N-1 chunks are used to quickly reconstruct the lost chunks. d,c Furthermore, since the affected 7812 block stripes are evenly distributed across all storage devices 5061-506D, the reconstruction of the lost 7812 blocks 510 is possible. d,c The statistics to be involved (the desired statistics may be determined based on the following: selected values ​​of D and N, the size of N storage devices 506, and the layout of the fault domain) will be from each storage device 5061-506. D Reading the same amount of data (i.e., the burden of reconstructing lost data is evenly distributed across all storage devices 5061-506) D In order to recover very quickly from a failure.

[0113] Next, we turn to two storage devices, 5061–506. D Concurrent failure scenarios (or more generally, concurrent failures in two fault domains), due to the block stripes 5201-520 of each DFRAS. S In all storage devices 5061-506 D The surface is uniformly distributed, with only a very few clusters of bands 5201-520. S Two out of N chunks will be lost. The virtual file system is operable to quickly identify such a double-lost chunk stripe based on metadata indicating chunk stripe 5201-520. S With storage devices 5061-506 D The mapping between them. Once such two-times-lost-block stripes are identified, the virtual file system can prioritize rebuilding those stripes before initiating a single-time-lost-block striping reconstruction. The remaining block stripes will have only a single lost block, and for them (the vast majority of the affected block stripes), the two storage devices 506 dConcurrent failures with only one storage device 506 d The faults are the same. A similar principle applies to three concurrent faults (in a two-fault scenario, the number of chunk stripes with three fault blocks will be far less than the number with two fault blocks), and so on. In the example implementation, it can be based on chunk stripe 520. s The number of missing items in the control block stripe 520 is used to control the execution of the stripe. s The rate of reconstruction. This can be achieved, for example, by controlling the rate at which reads and commits are performed for reconstruction, the rate at which FEC calculations are performed for reconstruction, the rate at which network messages for reconstruction are transmitted, etc.

[0114] Figure 9 A forward error correction scheme according to an example implementation of this disclosure is shown, which can be used to protect data stored in non-volatile memory of a virtual file system. Storage block 902 of block stripes 5301–5304 of DFRAS is shown. 1,1 -902 7,7 .exist Figure 8 In the protection scheme, five blocks in each strip are used to store data codes, and two blocks in each strip are used to store protection codes (i.e., M=5 and K=2). Figure 9 In the middle, the protection code is calculated using the following formulas (1)-(9):

[0115]

[0116]

[0117]

[0118]

[0119]

[0120]

[0121]

[0122]

[0123]

[0124] therefore, Figure 9The four stripes 5301-5304 are part of a multi-strip (in this case, four stripes) FEC protection domain, and the loss of any two or fewer blocks in any block stripe 5301-5304 is recovered by using various combinations of equations (1)-(9) above. For comparison, an example of a single-strip protection domain is: if only P1 protects D11, D22, D33, D44, D54, and writes D11, D22, D33, D44, D54, and P1 all into stripe 5301 (5301 is a single-strip FEC protection domain).

[0125] According to an example implementation of this disclosure, multiple computing devices are communicatively coupled to each other via a network, and each of the multiple computing devices includes one or more of a plurality of storage devices. Multiple failover address spaces are distributed across the multiple storage devices, such that each of the multiple failover address spaces spans multiple storage devices. Each of the multiple failover address spaces is organized into multiple stripes (e.g., as shown in the example implementation). Figure 8 and Figure 9 The multiple 530s shown). Each or more of the multiple stripes are multiple forward error correction (FEC) protection domains (e.g., such as...). Figure 9 A portion of a corresponding domain in a multi-strip FEC domain. Each of the multiple stripes may include multiple storage blocks (e.g., multiple 512). Each block of a particular stripe in the multiple stripes may reside on different storage devices in multiple storage devices. The first portion of the multiple storage blocks (e.g., by...) Figure 9 The 5301 band of 902 1,2 -902 1,6 The five components) can be used to store data codes, while the second part of multiple storage blocks (e.g., Figure 9 The two 902 stripes of 5301 1,1 and 902 1,7The number of data stripes can be used to store protection codes calculated at least in part based on data codes. Multiple computing devices can be operable to sort multiple stripes. This sorting can be used to select which of the multiple stripes is used for the next commit operation to one of the multiple failover address spaces. The sorting can be based on the number of protected and / or unprotected storage blocks in each of the multiple stripes. For any of the multiple stripes, the sorting can be based on a bitmap stored on multiple storage devices having a particular one of the multiple stripes. The sorting can be based on the number of blocks currently storing data in each of the multiple stripes. The sorting can be based on the read and write overhead for committing to each of the multiple stripes. Each failover address space can be owned by only one of the multiple computing devices at any given time, and each of the multiple failover address spaces can only be read and written by its owner. Each computing device can own multiple failover address spaces. Multiple storage devices can be organized into multiple failover domains. Each of the multiple stripes can span multiple failover domains. Each fault resilience address space can span all multiple fault domains, such that when any particular fault domain fails, the workload for reconstructing lost data is distributed among each of the other fault domains in the multiple fault domains. Multiple stripes can be distributed across multiple fault domains such that, in the event that two fault domains in the multiple fault domains fail simultaneously, the probability of two blocks appearing in any particular stripe of multiple stripes across the multiple fault domains is less than the probability of only one block appearing in any particular stripe of multiple stripes across the multiple fault domains. Multiple computing devices can be used to first reconstruct any one of the multiple stripes with two fault blocks, and then reconstruct any one of the multiple stripes with only one fault block. Multiple computing devices can be used to perform the reconstruction of multiple stripes with two fault blocks at a higher rate than the reconstruction rate of multiple stripes with only one fault block (e.g., a larger percentage of CPU clock cycles dedicated to reconstruction, a larger percentage of network transfer opportunities dedicated to reconstruction, etc.). Multiple computing devices can be used to determine the rate at which to reconstruct any particular lost block based on the number of other blocks in the same stripe across the multiple stripes that are lost, in the event that one or more fault domains fail. One or more of the multiple fault domains may include multiple storage devices. Each of the multiple FEC protection domains may span multiple stripes. The multiple stripes may be organized into multiple groups (e.g., Figure 8 Block strips 5201-520 in SThis involves multiple groups, each comprising one or more stripes, and multiple computing devices operable to sort one or more stripes within each group. The multiple computing devices are operable to perform successive commit operations on a selected group of groups until one or more stripes in the group no longer meet a defined criterion, and if one of the selected groups no longer meets the defined criterion, another group is selected. The criterion may be based on the number of blocks available for new data writing.

[0126] Although this method and / or system has been described with reference to certain implementations, those skilled in the art will understand that various changes and substitutions can be made without departing from the scope of this method and / or system. Furthermore, many modifications can be made to adapt particular situations or materials to the teachings of this invention without departing from the scope of the invention. Therefore, it is intended that this method and / or system be limited to the specific implementations disclosed, but rather that this method and / or system will include all implementations falling within the scope of the appended claims.

[0127] As used herein, the terms “circuit” and “circuit system” refer to physical electronic components (i.e., hardware) and any software and / or firmware (“code”) that can configure, be executed by, and / or be associated with the hardware. As used herein, for example, a particular processor and memory may include a first “circuit” when executing one or more lines of code, and may include a second “circuit” when executing a second or more lines of code. As used herein, “and / or” refers to any one or more items in a list connected by “and / or”. As an example, “x and / or y” represents any element in the three-element set {(x), (y), (x, y)}. In other words, “x and / or y” means “one or both of x and y”. As another example, “x, y and / or z” represents the seven-element set {(x), (y), (z), (x, y), (x, z), (y, z), (x, y, z)}. In other words, “x, y and / or z” means “one or more of x, y, and z”. As used herein, the term "exemplary" means used as a non-limiting example, instance, or illustration. As used herein, the terms "for example" and "e.g." introduce a list of one or more non-limiting examples, instances, or illustrations. As utilized herein, a circuit is "operable" to perform a function as long as it contains the necessary hardware and code (if necessary) required to perform that function, regardless of whether the function's performance is disabled (e.g., by user-configurable settings, factory settings, etc.).

Claims

1. A system comprising: Multiple storage devices; as well as A circuit is configured to indicate when the retention period is reached, wherein: Each storage device is capable of operating to store one or more blocks of data. Each of the data blocks is associated with a deletion status. When the retention period is reached, each data block that is not in the highest deletion state is incremented to the next higher deletion state. If a data block is accessed, the data block is reset to the least deleted state, and each of the plurality of storage devices is assigned to one of the upper and lower layers. The system includes a processor configured to read the deletion status associated with a specific data block allocated to the upper-level storage device, and if the specific data block is in the highest deletion status, to demote the specific data block to the lower-level storage device.

2. The system according to claim 1, wherein, The system includes a processor configured to read the deletion status associated with a specific data block and delete the specific data block if the specific data block is in the highest deletion status.

3. The system according to claim 1, wherein, If a specific data block is read, that specific data block is reset to the minimum deletion state.

4. The system according to claim 1, wherein, Each of the data blocks is associated with a degraded state.

5. The system according to claim 4, wherein, The circuit is configured to indicate when a degradation period is reached.

6. The system according to claim 5, wherein, When the degradation period is reached, each data block not in the highest degradation state is incremented to the next higher degradation state, and If a write access is made to a specific data block, that specific data block is reset to the lowest degraded state.

7. The system according to claim 6, wherein, Each data block in the highest degraded state is degraded to the storage device allocated to the lower layer.

8. The system according to claim 1, wherein, Each of the data blocks is a 4KB data block.

9. The system according to claim 1, wherein, Each data block in the highest deletion state is deleted from the read cache.

10. A system comprising: Multiple storage devices; as well as A circuit is configured to indicate when the retention period is reached, wherein: Each storage device is capable of operating to store one or more blocks of data. Each of the data blocks is associated with a deletion status. When the retention period is reached, each data block that is not in the highest deletion state is incremented to the next higher deletion state. If a data block is accessed, it is reset to the lowest deletion state, with each data block associated with a degradation state. The circuit is configured to indicate when a degradation cycle is reached. When the degradation cycle is reached, each data block that is not in the highest degradation state is incremented to the next higher degradation state. If a write access is made to a specific data block, that specific data block is reset to the lowest degraded state, and Each data block in the highest degradation state is downgraded to a storage device allocated to the lower layer.

11. The system according to claim 10, wherein, The system includes a processor configured to read the deletion status associated with a specific data block and delete the specific data block if the specific data block is in the highest deletion status.

12. The system according to claim 10, wherein, If a specific data block is read, that specific data block is reset to the minimum deletion state.

13. The system according to claim 10, wherein, Each of the data blocks is a 4KB data block.

14. The system according to claim 10, wherein, Each data block in the highest deletion state is deleted from the read cache.

15. A method comprising: Delete the status for each of the multiple data blocks; If a data block among the plurality of data blocks is accessed, the data block is reset to the least deleted state; Measurement retention period; After the retention period, each of the multiple data blocks that is not yet in the highest deletion state is incremented to the next higher deletion state; Associate each of the plurality of data blocks with a degraded state; The degradation cycle was measured. After the degradation cycle, each data block that is not in the highest degradation state is incremented to the next higher degradation state; If a write access is made to a specific data block, that specific data block is reset to the lowest degraded state; and Each data block in the highest degradation state is downgraded to a storage device allocated to the lower layer.

16. The method according to claim 15, wherein, The method includes deleting each data block that is in the highest deletion state.

17. The method according to claim 15, wherein, The method includes resetting a specific data block to a minimum deletion state if the specific data block is read.

18. The method according to claim 15, wherein, Each of the data blocks is a 4KB data block.

19. The method according to claim 15, wherein, The method includes: if a specific data block is in the highest deletion state, then deleting the specific data block from the read cache.

Images