Systems and methods for atomic storage operations
The method addresses inefficiencies in storage systems by using transaction identifiers and metadata binding to ensure efficient and crash-resistant storage operations, particularly in asymmetric media, thereby maintaining file consistency and reducing latency.
Patent Information
- Application Number
- DE112014003152
- Authority / Receiving Office
- DE · DE
- Patent Type
- Patents
- Current Assignee / Owner
- Priority Date
- 2014-06-24
- Filing Date
- 2014-07-29
- Publication Date
- 2024-11-21
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
Existing storage systems face challenges in maintaining file consistency and efficiency when handling non-contiguous and disjoint logical identifier areas in non-volatile storage media, particularly with asymmetric storage media that have different latencies for different operations, leading to inefficiencies like write amplification and delayed data deletion.
A method and apparatus for implementing atomic memory operations by associating transaction identifiers with target identifiers, redirecting memory operations, and storing data with metadata to bind it to these identifiers, using a redirection module to map logical identifiers, and a protocol storage module to perform operations efficiently, while maintaining a persistent note to ensure consistency and crash-proof operations.
This approach ensures efficient and consistent storage operations by avoiding write amplification, reducing latency, and ensuring that data operations are completed persistently and crash-resistant, even in the face of system failures.
Smart Images

Figure 00000000_0000_ABST
Abstract
Description
TECHNICAL FIELD
[0001] This disclosure relates to storage systems, and more particularly to systems and methods for maintaining file consistency.
[0002] US 2013 / 0166855 A1 relates to systems and methods for vector input / output operations, wherein data of a vector memory request concerning disjoint, non-adjacent and / or non-contiguous logical identifier ranges are stored contiguously in a log structure in a non-volatile storage medium. SUMMARY OF THE INVENTION
[0003] According to the invention, a method for implementing an atomic memory request, an apparatus for implementing an atomic memory request, and a system having the features of the independent claims are provided; dependent claims relate to preferred embodiments. Embodiments of a method for implementing atomic memory operations are disclosed herein. Embodiments of the disclosed method may include linking transaction identifiers to target identifiers of an atomic memory request, redirecting memory operations of the atomic memory requests to the transaction identifiers, and / or moving data of the redirected memory operations from the transaction identifiers to the target identifiers in response to completion of the memory operations of the atomic memory request.
[0004] The storage operations may include appending data to a storage medium with metadata configured to associate the appended data with one or more of the transaction identifiers. Moving the appended data may include storing metadata on the storage medium configured to associate the appended data with the target identifiers.
[0005] The storage operations include storing data on a non-volatile storage medium with metadata configured to bind the stored data to one or more intermediate identifiers associated with the respective transaction identifiers. Moving the stored data may include storing a persistent note on the non-volatile storage medium configured to bind the one or more intermediate identifiers to the respective target identifiers. In some embodiments, the method further includes acknowledging the atomic storage request in response to moving the data of the redirected storage operations.
[0006] Disclosed herein are embodiments of an apparatus for implementing atomic storage operations. The disclosed apparatus may include a redirection module configured to map a second set of logical identifiers to a first set of logical identifiers of a storage request, a log storage module configured to perform storage operations of the storage request on a storage device within the second set of identifiers, and / or an atomic storage module configured to move the storage operations of the storage request to the first set of logical identifiers. The atomic storage module may be configured to store the storage operations by storing persistent metadata on the storage device.The persistent metadata may be configured to bind data segments stored on the storage device in association with the second set of identifiers to the first set of identifiers. The atomic storage module may be further configured to acknowledge completion of the storage request in response to a determination that the persistent metadata is stored on the storage device. The first set of identifiers may comprise a plurality of disjoint sets of logical identifiers, and the persistent metadata may be a single persistent annotation configured to bind the data segments to respective logical identifiers within the plurality of disjoint sets of logical identifiers.
[0007] In some embodiments, the redirection module is configured to allocate the second set of identifiers within an address space different from an address space of the first set of identifiers. Alternatively, the redirection module may be configured to allocate the second set of identifiers within a designated sub-region of an address space of the first set of identifiers. The second set of identifiers may correspond to a transaction address space, and the disclosed apparatus may further include a reconstruction module configured to invalidate data segments associated with identifiers of the transaction address space.
[0008] The atomic storage module may be configured to invalidate one of the identifiers within the first set of identifiers in response to a move of a storage operation configured to invalidate data of one of the identifiers in the second set of identifiers. The log storage module may be configured to append data segments to an ordered log of data segments on the storage device. The log storage module may be further configured to append a data segment that is not related to the storage request between two or more data segments of the storage request within the ordered log of data segments.
[0009] Disclosed herein are embodiments of a system for implementing atomic memory operations.The system may include means for receiving an atomic storage request relating to a set of logical destination identifiers, means for associating a set of transaction identifiers with the destination set of logical identifiers, means for appending a plurality of data packets relating to the atomic storage request to a sequential storage log on a storage device, wherein the data packets comprise persistent metadata configured to associate the data packets with respective transaction identifiers, and / or means for appending a persistent annotation to the storage log, wherein the persistent annotation is configured to associate the plurality of data packets comprising persistent metadata configured to associate the data packets with respective transaction identifiers with respective logical destination identifiers.
[0010] The disclosed system may further include means for mapping logical identifiers of the set of logical destination identifiers to respective transaction identifiers. In some embodiments, the system includes means for binding one or more of the transaction identifiers to a storage location associated with a logical identifier linked to the respective one or more transaction identifiers. Embodiments of the disclosed system may further include means for providing access to the data packets comprising the persistent metadata configured to associate the data packets with respective transaction identifiers using respective logical destination identifiers in response to appending the persistent annotation.
[0011] The logical destination identifiers may correspond to a logical address space, and the transaction identifiers may correspond to a different transaction address space. The system may further include means for reconstructing an index of mappings between logical identifiers of the logical address space and the data packets on the storage device using persistent metadata of the data packets and / or means for invalidating data packets associated with the transaction identifiers corresponding to the different transaction address space. BRIEF DESCRIPTION OF THE DRAWINGS Fig. 1A is a block diagram of one embodiment of a system for open-to-close consistency; Fig. 1B shows embodiments of storage metadata; Fig. 1C is a block diagram showing one embodiment of a memory array; Fig. 1D shows one embodiment of a data packet format; Fig. 1E shows one embodiment of a storage protocol; Fig. 2 is a block diagram of another embodiment of a system for open-to-close consistency; Fig. 3A is a block diagram of one embodiment of a system including a storage layer configured to efficiently implement cloning, moving, merging, and other higher-level storage operations; Fig. 3B shows embodiments of range cloning operations; Fig. 3C shows further embodiments of region cloning operations; Fig. 3D shows further embodiments of region cloning operations; Fig. 3E shows further embodiments of region cloning operations; Fig. 4A is a block diagram of another embodiment of a system for open-to-close consistency; Fig. 4B shows embodiments of range clone operations implemented using a reference map; Fig. 4C shows further embodiments of range clone operations implemented using a reference map; Fig. 4D shows further embodiments of range cloning operations implemented by using a reference map; Fig. 4E shows further embodiments of range clone operations implemented using a reference map; Fig. 5A is a block diagram of one embodiment of a system including an aggregation layer; Fig. 5B shows embodiments of range clone operations implemented using an aggregation layer; Fig. 6 shows embodiments of deduplication operations; Fig. 7 is a block diagram illustrating one embodiment of a system including a storage layer configured to efficiently implement storage snapshot operations; Fig. 8A-E show embodiments of range shift operations; Fig. 9A is a block diagram of a system including a storage layer configured to implement efficient file management operations; Fig. 9B shows an embodiment of a storage layer configured to implement mmap checkpoints; Fig. 9C shows embodiments of range clone and range merge operations implemented by a storage layer; Fig. 9D shows further embodiments of range clone and range merge operations; Fig. 9E shows further embodiments of range clone and range merge operations; Fig. 9F is a block diagram of one embodiment of a system including a storage layer configured to implement efficient open-to-close file consistency; Fig. 9G shows further embodiments of close-to-open file consistency; Fig. 10A shows an embodiment of a system including a memory layer configured to implement atomic memory operations; Fig. 10B shows embodiments of atomic memory operations; Fig. 11 is a flow diagram of one embodiment of a method for managing a logical data storage interface in a context-dependent format on a non-volatile storage medium; Fig. 12 is a flow diagram of one embodiment of a method for managing a logical interface of context-dependent data; Fig. 13 is a flow diagram of another embodiment of a method for managing a logical interface of context-dependent data; Fig. 14 is a flow diagram of one embodiment of a method for managing range merge operations; Fig. 15 is a flow diagram of another embodiment of a method for managing range clone operations; Fig. 16 is a flow diagram of another embodiment of a method for managing range merge operations; Fig. 17 is a flow diagram of one embodiment of a method for atomic memory operations; Fig. 18 is a flow diagram of another embodiment of a method for atomic memory operations; and Fig. 19 is a flow diagram of another embodiment of a method for atomic memory operations. DETAILED DESCRIPTION
[0012] Fig. 1A is a block diagram of one embodiment of a computing system 100 including a storage layer 130 configured to provide storage services to one or more storage clients 106. The storage layer 130 may be configured to provide open-to-close file services, as disclosed in more detail herein. The computing system 100 may include any computing device, including, but not limited to, a server, laptop, embedded system, mobile device, and / or the like. In some embodiments, the computing system 100 may include multiple computing devices, such as a cluster of server computing devices. The computing system 100 may include processing resources 101, volatile storage resources 102 (e.g., random-access memory (RAM), non-volatile storage resources 103, and a communications interface 104.The processing resources 101 may include, but are not limited to, general-purpose central processing units (CPUs), application-specific integrated circuits (ASICs), and programmable logic elements such as field-programmable gate arrays (FPGAs), programmable logic gates (PLGs), and the like. The non-volatile storage resources 103 may comprise a non-transitory machine-readable storage medium such as a magnetic disk, a solid-state storage medium, an optical storage medium, and / or the like. The communication interface 104 may be configured to communicatively couple the computing system 100 to a network 105.The network 105 may comprise any suitable communications network, including, but not limited to, a Transmission Control Protocol / Internet Protocol (TCP / IP), a Local Area Network (LAN), a Wide Area Network (WAN), a Virtual Private Network (VPN), a Storage Area Network (SAN), a Public Switched Telephone Network (PSTN), the Internet, and / or the like.
[0013] Computing system 100 may form a storage layer 130 that may be configured to provide storage services to one or more storage clients 106. Storage clients 106 may include, but are not limited to, operating systems (including bare metal operating systems, guest operating systems, virtual machines, virtualization environments, and the like), file systems, database systems, remote storage clients (e.g., storage clients communicatively coupled to computing system 100 and / or storage layer 130 through network 105), and / or the like.
[0014] The storage layer 130 (and / or modules thereof) may be implemented in software, hardware, or a combination thereof. In some embodiments, portions of the storage layer 130 are embodied as executable instructions, such as computer program code, that may be stored on a persistent, non-transitory storage medium, such as the non-volatile memory resources 103. The instructions and / or computer program code may be configured for execution by the processing resources 101. Alternatively or additionally, portions of the storage layer 130 may be embodied as machine components, such as general-purpose and / or application-specific components, programmable hardware, FPGAs, ASICs, hardware controllers, memory controllers, and / or the like.
[0015] The storage layer 130 may be configured to perform storage operations 140 on a storage medium. The storage medium 140 may comprise any storage medium capable of persistently storing data. As used herein, "persistent" refers to storing information on a persistent, non-volatile storage medium. The storage medium 140 may include non-volatile storage media such as solid-state storage media in one or more solid-state storage devices or solid-state drives (SSDs), hard disk drives (e.g., integrated drive electronics (IDE) drives, small computer system interface (SCSI) drives, serial-attached SCSI (SAS) drives, serial-attached SATA (SATA) drives, etc.), tape drives, writable optical drives (e.g., CD drives, DVD drives, Blu-ray drives, etc.), and / or the like.
[0016] In some embodiments, storage medium 140 comprises non-volatile solid-state memory, which may include, but is not limited to, NAND flash memory, NOR flash memory, Nano-RAM (NRAM), magnetoresistive RAM (MRAM), phase-change RAM (PRAM), racetrack memory, memristor memory, nanocrystal wire-based memory, silicon oxide-based sub-10 nanometer process memory, graphene memory, silicon oxide-nitride-oxide-silicon (SONOS) memory, resistive random-access memory (RAM), programmable metallization cell (PMC), conductive bridging RAM (CBRAM), and / or the like. Although particular embodiments of storage medium 140 are disclosed herein, the teachings of this disclosure could be applied to any suitable form of memory, including both non-volatile and volatile types.Accordingly, although certain embodiments of the storage layer 130 are disclosed in the context of non-volatile solid-state storage devices 140, the storage layer 130 may be used with other storage devices and / or storage media.
[0017] In some embodiments, storage medium 140 includes volatile memory, which may include, but is not limited to, RAM, dynamic RAM (DRAM), static RAM (SRAM), synchronous dynamic RAM (SDRAM), etc. Storage medium 140 may correspond to the memory of processing resources 101, such as a CPU cache (e.g., L1, L2, L3 cache, etc.), graphics memory, and / or the like. In some embodiments, storage medium 140 is communicatively coupled to storage layer 130 using an interconnect 127. Interconnect 127 may include, but is not limited to, Peripheral Component Interconnect (PCI), PCI express (PCI-e), Serial Advanced Technology Attachment (Serial ATA or SATA), Parallel ATA (PATA), Small Computer System Interface (SCSI), IEEE 1394 (FireWire), Fibre Channel, Universal Serial Bus (USB), and / or the like.Alternatively, storage medium 140 may be a remote storage device communicatively coupled to storage layer 130 through network 105 (and / or another communication interface, such as a storage area network (SAN), a virtual storage area network (VSAN), and / or the like). Therefore, interconnect 127 may include a remote bus, such as a PCE-e bus, a network interconnect (e.g., Infiniband), a storage network, a Fibre Channel Protocol (FCP) network, HyperSCSI, and / or the like.
[0018] The storage layer 130 may be configured to manage storage operations on the storage medium 140 through the use of, among other things, a storage controller 139. The storage controller 139 may include, but is not limited to, software and / or hardware components, including one or more drivers and / or other software modules operating on the computing system 100, such as storage drivers, I / O drivers, filter drivers, and / or the like; hardware components such as hardware controllers, communication interfaces, and / or the like; and so on. The storage medium 140 may be embodied on a storage device 141. Portions of the storage layer 130 (e.g., the storage controller 139) may be implemented as hardware and / or software components (e.g., firmware) of the storage device 141.
[0019] The storage controller 139 may be configured to implement storage operations at specific storage locations of the storage medium 140. As used herein, a storage location refers to a storage unit of a storage resource (e.g., a storage medium and / or device) capable of persistently storing data; storage locations may include, but are not limited to, pages, groups of pages (e.g., logical pages and / or offsets within a logical page), storage sections (e.g., physical erase blocks, logical erase blocks, etc.), sectors, locations on a magnetic disk, battery-backed storage locations, and / or the like. The storage locations may be addressable within a memory address space 144 of the storage medium 140. Memory addresses may correspond to physical addresses, media addresses, back-end addresses, address offsets, and / or the like.Memory addresses may correspond to any memory address space 144, memory addressing scheme, and / or arrangement of memory locations.
[0020] The storage layer 130 may include an interface 131 through which the storage clients 106 may access storage services provided by the storage layer 130. The storage interface 131 may include one or more of the following components: a block device interface, a virtualized storage interface, one or more virtual storage units (VSUs), an object storage interface, a database storage interface, and / or another suitable interface and / or an application programming interface (API).
[0021] The storage layer 130 may provide referencing of the storage resources through a front-end storage interface. As used herein, a "front-end storage interface" refers to an interface and / or namespace through which the storage clients 106 can reference storage resources of the storage layer 130. A storage interface may correspond to a logical address space 132. The logical address space 132 may include a group, set, collection, range, and / or extension of identifiers. As used herein, an "identifier" or "logical identifier" (LID) refers to an identifier for referencing a source resource; LIDs may be names (e.g.,File names, distinguished names and / or the like), data identifiers, references, links, LIDs, front-end identifiers, logical addresses, logical block addresses (LBAs), LUN (Logical Unit Number) addresses, VUN (Virtual Unit Number) addresses, virtual storage addresses, storage addresses, physical addresses, media addresses, back-end addresses and / or the like.
[0022] The logical capacity of the logical address space 132 may correspond to the number of LIDs in the logical address space 132 and / or the size and / or granularity of the storage resources referenced by the LIDs. In some embodiments, the logical address space 132 may be "underprovisioned." As used herein, an underprovisioned logical address space 132 refers to a logical address space 132 having a logical capacity that exceeds the physical storage capacity of the underlying storage resources (e.g., exceeds the storage capacity of the storage medium 140). In one embodiment, the storage layer 130 is configured to provide a 64-bit logical address space 132 (e.g., a logical address space comprising 2^26 unique LIDs), which may exceed the physical storage capacity of the storage medium 140.The large, under-provisioned logical address space 132 may enable storage clients 106 to efficiently allocate and / or reference contiguous ranges of LIDs while reducing the possibility of name conflicts.
[0023] The translation module 134 of the storage layer 130 may be configured to map LIDs of the logical address space 132 to storage resources (e.g., data stored within the memory address space 144 of the storage medium 140). The logical address space 132 may be independent of the back-end storage resources (e.g., the storage medium 140); accordingly, there may be no fixed or predetermined mappings between the LIDs of the logical address space 132 and the memory addresses of the memory address space 144. In some embodiments, the logical address space 132 is sparse, underprovisioned, and / or overprovisioned, such that the size of the logical address space 132 is different from the memory address space 144 of the storage medium 140.
[0024] The storage layer 130 may be configured to maintain storage metadata 135 related to storage operations performed on the storage medium 140. The storage metadata 135 may include, but is not limited to, a forward map that includes mappings from any element to any element between LIDs of the logical address space 132 and memory addresses within the memory address space 144, a reverse map related to the contents of the memory locations of the storage medium 140, validity bitmaps, reliability tests and / or status metadata, status information (e.g., miss rate, retirement status, etc.), cache metadata, and / or the like. Portions of the storage metadata 135 may be maintained within the volatile storage resources 102 of the computing system 100. Alternatively or additionally, portions of the storage metadata 135 may be stored on non-volatile storage resources 103 and / or the storage medium 140.
[0025] Fig. 1B shows one embodiment of any-element-to-any-element mappings 150 between LIDs of the logical address space 132 and back-end identifiers (e.g., memory addresses) within the memory address space 144. The any-element-to-any-element mappings 150 may be maintained in one or more data structures of the memory metadata 135. As in Fig. 1B, the translation module 134 may be configured to map any memory resource identifier (any LID) to any back-end memory location. As further illustrated, the size of the logical address space 132 may be different than that of the underlying memory address space 144. In the embodiment of Fig. 1B, the logical address space 132 may be under-provisioned and, as such, may include a larger range of LIDs than the range of memory addresses in the memory address space 144.
[0026] As disclosed above, storage clients 106 may reference storage resources by the LIDs of logical address space 132. Accordingly, logical address space 132 may correspond to a logical interface 152 of the storage resources, and mappings to specific memory addresses in memory address space 144 may correspond to a back-end interface 154 of the storage resources.
[0027] The storage layer 130 may be configured to maintain the mappings from any element to any element 150 between the logical interface 152 and the back-end interface 154 in a forward map 160. The forward map 160 may comprise any suitable data structure, including, but not limited to, an index, a map, a hash table, a tree, a range-encoded tree, a B-tree, and / or the like. The forward map 160 may include entries 162 corresponding to LIDs assigned for use in referencing data stored on the storage medium 140. The entries 162 of the forward map 160 may associate LIDs 164A-D with corresponding memory addresses 166A-D within the memory address space 144.The forward map 160 may be sparsely populated and, as such, may omit entries corresponding to LIDs that are not currently assigned by a storage client 106 and / or are not currently in use to reference valid data stored on the storage medium 140. In some embodiments, the forward map 160 includes a range-encoded data structure such that one or more of the entries 162 may correspond to a plurality of LIDs (e.g., a range, an extension, and / or a set of LIDs). In the embodiment of FIG. Fig. 1B, the forward map 160 includes an entry 162 corresponding to a range of LIDs 164A that is mapped to a corresponding range of memory addresses 166A. The entries 162 may be indexed by LIDs. In the embodiment of Fig. 1B, the entries 162 are arranged by respective links in a tree data structure. However, the disclosure is not limited in this respect and could be adapted to use any suitable data structure and / or indexing mechanism.
[0028] On Fig. 1C, in some embodiments, the storage medium 140 may include a solid-state memory array 115 including a plurality of solid-state memory elements 116A-Y. As used herein, a solid-state memory array (or memory array) 115 refers to a set of two of the more independent columns 118. A column 118 may include one or more solid-state memory elements 116A-Y communicatively coupled to the memory layer 130 in parallel using, among other things, interconnect 127. Rows 117 of the array 115 may include physical memory units of the respective columns 118 (solid-state memory elements 116A-Y). As used herein, a solid-state memory element 116A-Y includes, but is not limited to, solid-state memory resources formed as a package, die, chip, plane, printed circuit board, and / or the like.The solid-state storage elements 116A-Y comprising array 115 may be capable of operating independently. Accordingly, a first of the solid-state storage elements 116A may be capable of performing a first storage operation while a second solid-state storage element 116B performs a different storage operation. For example, the solid-state storage element 116A may be configured to read data at a first physical address while another solid-state storage element 116B reads data at a different physical address.
[0029] A solid-state storage array 115 may also be referred to as a logical storage element (LSE). As disclosed in more detail herein, the solid-state storage array 115 may include logical storage units (rows 117). As used herein, a "logical storage unit" or row 117 refers to a combination of two or more physical storage units, where each physical storage unit is on a respective column 118 of the array 115. A logical erase block refers to a set of two or more physical erase blocks, a logical page refers to a set of two or more pages, and so on. In some embodiments, a logical erase block may include erase blocks within respective logical storage elements 115 and / or banks. Alternatively, a logical erase block may include erase blocks within a plurality of different arrays 115 and / or may span multiple banks of solid-state storage elements.
[0030] Back to Fig. Referring to Figure 1A, the storage layer 130 may further include a protocol storage module 136 configured to store data on the storage medium 140 in a protocol-structured storage configuration (e.g., in a storage protocol). As used herein, a "storage protocol" or „"Protocol structure" refers to an ordered arrangement of data within the storage address space 144 of the storage medium 140. Data in the storage protocol may include and / or be associated with persistent metadata. Accordingly, the storage layer 130 may be configured to store data in a context-dependent, self-describing format. As used herein, a context-dependent or self-describing format refers to a data format in which data is stored in association with persistent metadata. In some embodiments, the persistent metadata may be configured to identify the data and, as such, may include and / or reference the logical interface of the data (e.g., may include the LID(s) associated with the data).The persistent metadata may contain other information, including, but not limited to, information related to the owner of the data, access controls, data types, the relative position or offset of the data, information related to storage operations associated with the data (e.g., atomic storage operations, transactions, and / or the like), log sequence information, data storage parameters (e.g., compression algorithm, encryption, etc.), and / or the like.
[0031] Fig. 1D illustrates one embodiment of a context-dependent data format. The packet format 110 of Fig. 1D includes a data segment 112 and persistent metadata 114. The data segment 112 may have any length and / or size. The persistent metadata 114 may be embodied as one or more header fields of the data packet 110. As disclosed above, the persistent metadata 114 may include the logical interface of the data segment 112 and, as such, may include the LID(s) associated with the data segment 112. Although Fig. 1D depicts a packet format 110, the disclosure is not limited in this regard and could associate data (e.g., data segment 112) with context-dependent metadata in other ways, including, but not limited to, an index on storage medium 140, a storage section index, and / or the like. Data packets 110 may be associated with sequence information 113. The sequence information may be used to determine the relative order of the data packets in the storage protocol. In some embodiments, data packets are appended sequentially within storage sections of storage medium 140. The storage sections may correspond to erase blocks, logical erase blocks, or the like. Each storage section may be capable of storing a large number of data packets 110. The relative position of the data packets 110 within a storage section may determine the order of the packets within the storage protocol.The order of the memory slices may be determined, among other things, by memory slice sequence information 113. Memory slices may be assigned respective sequence information 113 at the time the memory slice is initialized for use (e.g., erased), programmed, closed, or the like. The memory slice sequence information 113 may determine an ordered sequence of memory slices within the memory address space 144. Accordingly, the relative order of a data packet 110 within the memory protocol may be determined by the following: a) the relative position of the data packet 110 within a particular memory slice, and b) the order of the memory slice relative to other memory slices in the memory address space 144.
[0032] In some embodiments, the storage layer 130 may be configured to manage an asymmetric, write-once storage medium 140, such as a solid-state storage medium, flash storage medium, or the like. As used herein, a "write-once" storage medium refers to a storage medium that is reinitialized (e.g., erased) each time new data is written or programmed to it. As used herein, an "asymmetric" storage medium refers to a storage medium that has different latencies for different types of storage operations. For example, in some embodiments, read operations may be faster than write / program operations, and write / program operations may be much faster than erase operations (e.g., reading the media may be hundreds of times faster than erasing, and some tens of times faster than programming the storage medium).Storage medium 140 may be partitioned into storage sections that can be erased as a group (e.g., erase blocks). As such, modifying a single data segment "in situ" may require erasing the entire erase block containing the data and rewriting the modified data into the erase block along with the original, unaltered data. This may result in inefficient "write amplification," which may excessively wear out the media. Therefore, in some embodiments, storage layer 130 may be configured to write data "elsewhere." As used herein, writing data "elsewhere" refers to updating and / or overwriting data at other storage locations rather than overwriting the data "in situ" (e.g., overwriting the original physical storage location of the data).Updating and / or overwriting data elsewhere can avoid write amplification because existing, valid data on the erase block containing the data to be modified does not need to be erased and re-copied. Furthermore, writing data elsewhere can remove an erase from the latency path of many storage operations, so erase latency is not part of the "critical path" of write operations.
[0033] The storage layer 130 may be configured to perform storage operations elsewhere using, among other things, the log storage module 136. The log storage module 136 may be configured to append data to a current append point within the memory address space 144 in a manner that maintains the relative order of storage operations performed by the storage layer 130, thereby forming a "storage log" on the storage medium 140. Fig. 1E illustrates one embodiment of append-only memory operations performed within the memory address space 144 of the storage medium 140. As disclosed above, the memory address space 144 includes a plurality of memory sections 170A-N (e.g., erase blocks, logical erase blocks, or the like), each of which can be initialized (e.g., erased) for use in storing data. The memory sections 170A-N may include respective memory locations, which may correspond to pages, logical pages, and / or the like, as disclosed herein. The memory locations may be assigned respective memory addresses (e.g., memory address 0 through memory address N).
[0034] The log storage module 136 may be configured to store data sequentially from an append point 180 within the physical address space 144. In the embodiment of Fig. 1E, data may be appended at append point 180 within storage location 182 of storage section 170A, and when storage location 182 is filled, append point 180 may advance 181 to the next available storage location. As used herein, an "available" storage location refers to a storage location that has been initialized and has not yet been programmed (e.g., that has been erased). As disclosed above, some types of storage media can only be reliably programmed once after an erase operation. Accordingly, an available storage location may refer to a storage location within a storage section 170A-N that is in an initialized (or erased) state.
[0035] In the embodiment of Fig. 1E, logical erase block 170B may be unavailable for storage because, among other things, it is not in an erased state (e.g., includes valid data), is out of service due to high error rates, or the like. Therefore, after filling storage location 182, log storage module 136 may skip unavailable storage section 170B and advance append point 180 to the next available storage section 170C. Log storage module 136 may be configured to continue appending data to storage locations 183-185, at which point append point 180 continues to a next available storage section 170A-N, as disclosed above.
[0036] After storing data at the "last" location within memory address space 144 (e.g., location N 189 of memory section 170N), protocol storage module 136 may advance append point 180 by wrapping back to the first memory section 170A (or to the next available memory section if memory section 170A is unavailable). Accordingly, protocol storage module 136 may treat memory address space 144 as a loop or cycle.
[0037] As disclosed above, sequential appending of data in the memory address space 144 may create a memory log on the storage medium 140. In the embodiment of Fig. 1E, the storage protocol may include the ordered sequence of storage operations performed by sequentially storing data packets (and / or other data structures) from the append point 180 within the memory address space 144. The append-only storage format may be used to modify and / or overwrite data elsewhere, as disclosed above. Performing storage operations elsewhere may avoid write amplification because existing valid data in the memory sections 170A-N that include the data being modified and / or overwritten does not need to be erased and / or re-copied. Furthermore, writing data elsewhere may remove an erase from the latency path of many storage operations (the erase latency is no longer part of the "critical path" of a write operation).
[0038] In the embodiment of Fig. 1E, a data segment X0 corresponding to LID A may be stored at a storage location 191. The data segment X0 may be stored in the self-describing packet format 110 disclosed above. The data segment 112 of the packet 110 may include the data segment X0, and the persistent metadata 114 may include the LID(s) associated with the data segment (e.g., LID A). A storage client 106 may request an operation to modify and / or overwrite the data associated with LID A, which may include exchanging the data segment X0 with a data segment X1. The storage layer 130 may perform this operation elsewhere by appending a new data packet 110 comprising the data segment X1 at a different storage location 193 on the storage medium 144, rather than modifying the existing data packet 110 in situ at the storage location 191.The storage operation may further include updating the storage metadata 135 to associate the LID A with the memory address of the storage location 193 and / or invalidate the stale data X0 at the storage location 191. As shown in FIG. Fig. 1E, updating the memory metadata 135 may include updating an entry of the forward map 160 to associate the LID A 164E with the memory address of the modified data segment X1.
[0039] Performing storage operations elsewhere (e.g., appending data to the storage log) may result in outdated or invalid data remaining on the storage medium 140 (e.g., data that has been deleted, modified, and / or overwritten elsewhere). As described in Fig. As illustrated in Figure 1E, modifying the data of LID A by appending the data segment X1 to the storage log, as opposed to overwriting and / or replacing the data segment X0 in situ at storage location 191, results in the outdated version of the data segment X0 being retained on the storage medium 140. The outdated version of the data segment X0 cannot be immediately removed (e.g., erased) from the storage medium 140 because, as disclosed above, erasing the data segment X0 may involve erasing an entire storage portion 170A and / or relocating valid data within the storage portion 170A, which is a time-consuming operation and may result in write amplification. Similarly, data that is no longer in use (e.g., erased or subjected to a TRIM operation) may not be immediately removed. As such, the storage medium 140 may accumulate a substantial amount of "invalid" data over time.
[0040] The storage layer 130 may identify invalid data by using the storage metadata 135 (e.g., the forward map 160), such as the data segment X0 at the storage location 191. The storage layer 130 may determine that storage locations not associated with valid identifiers (LIDs) in the forward map 160 include data that does not need to be retained on the storage medium 140. Alternatively or additionally, the storage layer 130 may maintain other storage metadata 135, such as validity bitmaps, reverse maps, and / or the like, to efficiently identify data that has been deleted, TRIMed, is stale, and / or is otherwise invalid.
[0041] The storage layer 130 may be configured to reclaim storage resources occupied by invalid data. The storage layer 130 may be further configured to perform other media management operations, including, but not limited to, refreshing data stored on the storage medium 140 (to prevent error conditions due to data degradation, write disturbances, read disturbances, and / or the like), monitoring media reliability conditions, and / or the like. As used herein, reclaiming storage resources, such as a storage section 170A-N, refers to erasing the storage section 170A-N so that new data can be stored / programmed thereon. Reclaiming a storage section 170A-N may include relocating valid data in the storage section 170A-N to a new storage section.The memory layer 130 may identify memory sections 170A-N for reclaim based on one or more factors, including, but not limited to, the amount of invalid data in the memory section 170A-N, the amount of valid data in the memory section 170A-N, wear levels (e.g., the number of program / erase cycles), time since programming or refreshing the memory section 170A-N, etc.
[0042] The storage layer 130 may be configured to reconstruct the storage metadata 135, including the forward map 160, by using the contents of the storage log on the storage medium 140. In the embodiment of Fig. 1E, the current version of the data associated with LID A can be determined based on the relative protocol order of the data packets 110 at storage locations 191 and 193, respectively. Because the data packet at storage location 193 is ordered after the data packet at storage location 191 in the storage protocol, storage layer 130 can determine that storage location 193 contains the most recent, current version of the data corresponding to LID A. Storage layer 130 can reconstruct forward map 160 to associate LID A with the data packet at storage location 193 (rather than with the outdated data at storage location 191).
[0043] Fig. 2 shows another embodiment of a system 200 including a storage layer 130. The storage medium 140 may include a plurality of independent banks 119A-N, each of which may include one or more memory arrays 115A-N. Each independent bank 119A-N may be coupled to the memory controller 139 via interconnect 127.
[0044] The memory controller 139 may include a memory request receiver module 231 configured to receive memory requests from the memory layer 130 via a bus 127. The memory request receiver 231 may be further configured to transfer data to / from the memory layer 130 and / or the memory clients 106. Accordingly, the memory request receiver module 231 may include one or more direct memory access (DMA) modules, remote DMA modules, bus controllers, bridges, buffers, etc.
[0045] The storage controller 139 may include a write module 240 configured to store data on the storage medium 140 in response to requests received via the request module 231. The storage requests may include and / or reference the logical interface of the data to which the requests refer. The write module 240 may be configured to store the data in a self-describing storage protocol, which, as disclosed above, may include sequentially appending data packets 110 within the storage address space 144 of the storage medium 140. The data packets 110 may include and / or reference the logical interface of the data (e.g., may include the LID(s) associated with the data). The write module 240 may include a write processing module 242 configured to process data for storage.Processing the data for storage may include one or more of the following: a) compression processing, b) encryption processing, c) encapsulating data into respective data packets 110 (and / or other containers), d) performing error-correcting code (ECC) processing, etc. Write buffer 244 may be configured to buffer data for storage on storage medium 140. In some embodiments, write buffer 244 may include one or more synchronization buffers configured to synchronize a clock domain of storage controller 139 with a clock domain of storage medium 140 (and / or interconnect 127).
[0046] The log storage module 136 may be configured to select a memory location(s) for data storage operations and may provide addressing and / or control information to the memory arrays 115A-N of the independent banks 119A-N. As disclosed herein, the log storage module 136 may be configured to append data sequentially in a log format within the memory address space 144 of the storage medium 140.
[0047] Storage operations for writing data may include: a) appending one or more data packets to the storage protocol on the storage medium 140; and b) updating the storage metadata 135 to associate LID(s) of the data with the storage addresses of the one or more data packets. In some embodiments, the storage metadata 135 may be maintained on storage resources of the storage controller 139 (e.g., on dedicated volatile storage resources of the storage device 141 that includes the storage medium 140). Alternatively or additionally, portions of the storage metadata 135 may be maintained within the storage layer 130 (e.g., on a volatile memory 112 of the computing device 110 of Fig. 1A). In some embodiments, storage metadata 135 may be maintained by storage layer 130 in volatile memory and may be periodically stored on storage medium 140.
[0048] The storage controller 139 may further include a data read module 241 configured to read data from the storage protocol on the storage medium 140 in response to requests received via the storage request module 231. The requests may include LID(s) of the requested data, a memory address of the requested data, and / or the like. The read module 241 may be configured to: a) determine the memory address(es) of the data packet 110(s) containing the requested data using, among other things, the forward map 160, b) read the data packet 110(s) 110 from the determined memory address(es) on the storage medium 140, and c) process the data for use by the requesting device. Data read from the storage medium 140 may flow into the read module 241 via the read buffer 245.Read buffer 245 may include one or more read synchronization buffers for synchronizing clock domains, as described above. Read processing module 243 may be configured to process data read from storage medium 144, which may include, but is not limited to, one or more of the following: a) decompression processing, b) decryption processing, c) extracting data from one or more data packets 110 (and / or other containers), d) performing ECC processing, etc.
[0049] The memory controller 139 may further include a bank controller 252 configured to selectively route data and / or commands from the write module 240 and / or the read module 241 to / from certain independent banks 119A-N. In some embodiments, the memory controller 139 is configured to interleave memory operations between the independent banks 119A-N. For example, the memory controller 139 may read from the memory array 115A of bank 119A to the read module 241 while programming data from the write module 240 to the memory array 115B of bank 119B. Further embodiments of multi-bank memory operations are described in U.S. Patent Application No. 08 / 2008. No. 11 / 952,095, entitled "Apparatus, System, and Method for Managing Commands for Solid-State Storage Using Bank Interleave," filed December 12, 2006, to David Flynn et al., which is incorporated herein by reference.
[0050] The write processing module 242 may be configured to encode data packets 110 into ECC codewords. As used herein, an ECC codeword refers to data and corresponding error detection and / or correction information. The write processing module 242 may be configured to implement any suitable ECC algorithm and / or generate ECC codewords of any suitable type, which may include, but are not limited to, data segments and associated ECC syndromes, ECC symbols, ECC blocks, and / or other structured and / or unstructured ECC information. ECC codewords may comprise any error correction coding, including, but not limited to, block ECC coding, convolutional ECC coding, low-density parity check (LDPC) coding, Gallager coding, Reed-Solomon coding, Hamming codes, multidimensional parity coding, cyclic error correction codes, BCH codes, and / or the like.The write processing module 242 may be configured to generate ECC codewords of a predetermined size. Accordingly, a single packet may be encoded into a plurality of different ECC codewords, and / or a single ECC codeword may comprise portions of two or more packets. Alternatively, the write processing module 242 may be configured to generate ECC codewords of any size. Further embodiments of error-correcting code processing are disclosed in U.S. Patent Application No. 13 / 830,652, entitled "Systems and Methods for Adaptive Error-Correction Coding," filed March 14, 2013, to Jeremy Fillingim et al., which is incorporated herein by reference.
[0051] In some embodiments, storage layer 130 utilizes logical address space 132 to efficiently implement high-level storage operations. Storage layer 130 may be configured to implement "clone" or "logical copy" operations. As used herein, "cloning" or "logical copying" refers to operations for efficiently copying or replicating data managed by storage layer 130. A clone operation may include creating a set of "cloned" LIDs that may correspond to the same data as a set of "original" LIDs. A clone operation may therefore include referencing the same set of storage locations using two (or more) different logical interfaces (e.g., different sets of LIDs). A clone operation may therefore modify the logical interface of one or more data packets 110 stored on storage medium 140.A "logical move" may refer to an operation to modify the logical data interface managed by storage layer 130. A logical move operation may include changing the LIDs used to reference data stored on storage medium 140. A "merge" operation may include merging different portions of logical address space 132. As described in more detail herein, clone and / or move operations may be used to efficiently implement higher-level storage operations such as deduplication, snapshots, logical copies, atomic operations, transactions, and / or the like.
[0052] On Fig. Referring to Figure 3A, the storage layer 130 may include a logical interface management module 334 configured to manage logical interface operations related to data managed by the storage layer 130, such as clone operations, move operations, merge operations, and so on. Cloning LIDs may include modifying the logical interface of data stored on the storage medium 140 to, among other things, enable the data to be referenced using two or more different sets of LIDs. Accordingly, creating a clone may include: a) allocating a set of LIDs in the logical address space 132 (or dedicated portions thereof); and b) associating the allocated LIDs with the same storage location(s) as an "original" set of LIDs using, among other things, the storage metadata 135.Creating a clone may therefore include adding one or more entries to a forward map 160 configured to associate the new set of cloned LIDs with a particular set of storage locations.
[0053] The logical interface management module 334 may be configured to implement clone operations according to a clone synchronization strategy. A clone synchronization strategy may be used to determine how operations performed with respect to one of a plurality of clones or copies are propagated to the other clones or copies. For example, clones may be synchronized with respect to mapping operations such that a request to expand one of the clones involves expanding the other clones and / or copies. As used herein, expanding a file (or other data segment) refers to increasing a size, range, and / or extension of the file, which may include adding one or more logical identifiers to the clone, modifying one or more of the logical identifiers associated with the clone, and / or the like.The clone synchronization strategy may include a merge strategy that may, among other things, determine how differences between clones are managed when the clones are joined in a merge and / or fold operation (disclosed in additional detail below).
[0054] Fig. 3A shows an embodiment of a range clone operation 130 implemented by the storage layer. The range clone operation of Fig. 3A may be implemented in response to a request from a storage client 106. In some embodiments, interface 131 of storage layer 130 may be configured to provide interfaces and / or APIs for performing clone operations. Alternatively or additionally, the range clone operation may be performed as part of a higher-level operation, such as an atomic operation, transaction, snapshot, logical copy, file management operation, and / or the like.
[0055] As in Fig. 3A, the forward map 160 of the storage layer 130 includes an entry 362 configured to bind the LIDs 1024-2048 to media storage locations 3453-4477. Other entries are in Fig. 3A to avoid obscuring the details of the illustrated embodiment. As disclosed herein, entry 362 and its bindings may define a logical interface 311A through which storage clients 106 may reference the corresponding data (e.g., data segment 312); storage clients 106 may access and / or reference data segment 312 (and / or portions thereof) through storage layer 130 using LIDs 1024-2048. Accordingly, LIDs 1024-2048 define, among other things, logical interface 311A of data segment 312.
[0056] As disclosed herein, the storage layer 130 may be configured to store data in a context-dependent format on a storage medium 140 (e.g., packet format 110). In the embodiment of Fig. 3A, the data packet 310 includes a data segment 312 at storage locations 3453-4477. The data packet 310 further includes persistent metadata 314 indicating the logical interface of the data segment 312 (e.g., associating the data segment 312 with LIDs 1024-2048). As disclosed above, storing data in conjunction with descriptive, persistent metadata may enable the storage layer 130 to reconstruct the forward map 160 (and / or other storage metadata 135) from the contents of the storage log. In the embodiment of Fig. 3A, entry 362 can be reconstructed by associating the data stored at memory addresses 3453-4477 with LIDs 1024-2048 referenced by persistent metadata 314 of package 310. Although Fig. 3A depicts a single package 310, the disclosure is not limited in this regard. In some embodiments, the data of entry 362 may be stored in multiple different packages 310, each of which includes respective persistent metadata 314 (e.g., a separate package for each storage location, etc.).
[0057] The logical interface management module 334 may be configured to clone the entry 362 by, among other things, allocating a new set of LIDs corresponding to the original LIDs to be cloned and binding the new LIDs to the locations of the original source LIDs. As in Fig. 3B, creating the clone of LIDs 1024-2048 may include the logical interface management module 334 allocating an equivalent set of LIDs 6144-7168 and binding the cloned set of identifiers to memory addresses 3453-4477. Therefore, creating the clone may include modifying the storage metadata 135 to extend the logical interface 311B of the data segment 312 to include the LIDs 6144-7168 without requiring the underlying data segment 312 to be copied and / or replicated on the storage medium 140.
[0058] The modified logical interface 311B of the data segment 312 may be inconsistent with the context-dependent format of the corresponding data packet 310 stored at storage locations 3453-4477. As disclosed above, the persistent metadata 314 of the data packet 310 references the LIDs 1024-2048, but does not include and / or reference the cloned LIDs 6144-7168. The context-dependent format of data segment 312 may be updated to be consistent with the modified logical interface 311B (e.g., updated to associate the data with LIDs 1024-2048 and 6144-7168, as opposed to only LIDs 1024-2048), which may include rewriting the data segment in a packet format that associates the data segment with both sets of LIDs. If storage device 141 is a write-in-place random access storage device, persistent metadata 314 may be updated in-place.In other embodiments that include a write-once asymmetric storage medium 140, such in-situ updates may be inefficient. Therefore, the storage layer 130 may be configured to maintain the data in the inconsistent context-dependent format until the data is relocated in a media management operation, such as storage rebuild, relocation, and / or the like (by the media management module 370). Updating the context-dependent format of the data segment 312 may involve relocating and / or rewriting the data segment 312 on the storage medium 140, which may be a time-consuming process and may be particularly inefficient if the data segment 312 is large and / or the clone includes a large number of LIDs.Therefore, in some embodiments, storage layer 130 may defer updating the context-dependent format of cloned data segment 312 and / or may update the context-dependent format in one or more background operations. In the meantime, storage layer 130 may be configured to provide access to data segment 312 while it is stored in the inconsistent context-dependent format (data packet 310).
[0059] The storage layer 130 may be configured to confirm the completion of the clone operations before updating the context-dependent format of the corresponding data segment 312. The data may subsequently be rewritten (e.g., relocated) to the storage medium 140 in the updated context-dependent format. The update may occur outside the "critical path" of the clone operation and / or other foreground storage operations. In some embodiments, the data segment 312 is relocated by the media management module 370 as part of one or more of the following: a storage rebuild process, a data update operation, and / or the like. Accordingly, the storage clients 106 may access the data segment 312 through the modified logical interface 311B (e.g.,with respect to LIDs 1024-2048 and / or 6144-7168) without waiting for the context-dependent format of data segment 312 to be updated in accordance with the modified logical interface 311B.
[0060] Until the context-dependent format of the data segment 312 is updated on the storage medium 140, the modified logical interface 311B of the data segment 312 can only be stored in the storage metadata 135 (e.g., the Fig. ) exist. Therefore, if the forward map 160 is lost due to, among other things, power failure or data corruption, the clone operation may not be reflected in the reconstructed storage metadata 135 (the clone operation may not be persistent and / or crash-safe). As illustrated above, the persistent metadata 314 of the data packet 310 indicates that the data segment 312 is associated only with the LIDs 1024-2048, not with 6144-7168. Therefore, only the entry 362 is reconstructed (as in Fig. 3A) and entry 364 is omitted; as a result, subsequent attempts to access data segment 312 through modified logical interface 311B (e.g., through 6144-7168) may fail.
[0061] In some embodiments, the clone operation may further include storing a persistent note on the storage medium 140 to make a clone operation persistent and / or crash-safe. As used herein, a "persistent note" refers to metadata stored on the storage medium 140. Persistent notes 366 may correspond to a protocol order and / or be stored in a packet format, as disclosed herein. The persistent note 366 may include an indication of the modified logical interface 311B of the data segment 312. In the embodiment of Fig. 3B, the persistent annotation 366 corresponding to the clone operation shown may be configured to associate the data stored at memory addresses 3453-4477 with both ranges of LIDs 1024-2048 and 6144-7168. During reconstruction of the forward map 160 from the contents of the storage medium 140, the persistent annotation 366 may be used to reconstruct both entries 362 and 364 to associate the data segment 312 with both LID ranges of the updated logical interface 311B. In some embodiments, the storage layer 130 may acknowledge the completion of the clone operation in response to updating the storage metadata 135 (e.g., creating entry 364) and storing the persistent annotation 366 on the storage medium 140.In response, the persistent annotation 366 may be invalidated and / or marked for removal from the storage medium 140, wherein the context-dependent format of the data segment 312 is updated to be consistent with the updated logical interface 311B (e.g., wherein the data segment 312 is relocated and / or rewritten as disclosed above).
[0062] In some embodiments, the updated context-dependent format of the data segment 312 may include associating the data segment 312 with both the LID ranges 1024-2048 and 6144-7168. Fig. 3C shows an embodiment of an updated context-dependent format (data packet 320) for the data segment 312. As in Fig. As illustrated in Figure 3C, the persistent metadata 324 of the data packet 320 associates the data segment 312 with both the LID ranges 1024-2048 and 6144-7168 of the updated logical interface 311B. The data packet 320 may be written to a different location, at different memory addresses (64432-65456), than the original data packet 310, which may be reflected in updated entries 362 and 364 of the forward map 160. In response to appending the data packet 320 to the memory log, the corresponding persistent annotation 366 may (if applicable) be invalidated (removed and / or marked for subsequent removal from the storage medium 140). In some embodiments, removing the persistent note 366 may include issuing one or more TRIM messages indicating that the persistent note 366 no longer needs to be maintained on the storage medium 140.Alternatively or additionally, portions of the forward map 160 may be stored in a persistent, crash-safe location (e.g., on the non-volatile memory resources 103 and / or the storage medium 140). In response to the persistence of the forward map 160 (e.g., entries 362 and 364), the persistent annotation 366 may be invalidated, as disclosed above, even if the data segment 312 has not yet been rewritten in an updated context-dependent format.
[0063] The logical interface management module 334 may be configured to implement clone operations according to one or more different modes, including a copy-on-write mode. Fig. 3D shows one embodiment of a memory operation performed on a cloned region in a copy-on-write mode. In a copy-on-write mode, memory operations that occur after a clone is created may cause the clones to diverge from each other (e.g., entries 362 and 364 may point to different memory addresses, regions, and / or extensions). In the embodiment of Fig. 3D, the storage layer 130 has written the data segment 312 in the updated contextual data format (packet 320) that is configured to associate the data segment 312 with both LID ranges 1024-2048 and 6144-7168 (as in Fig. 3C). A storage client 106 may then issue one or more storage requests to modify and / or overwrite data corresponding to LIDs 6657-7168. In the embodiment of Fig. 3D, the storage request includes modifying and / or overwriting the data of LIDs 6657-7168. In response, storage layer 130 may store the new and / or modified data on storage medium 130, which may include appending a new data packet 340 to the storage protocol, as disclosed above. Data packet 340 may associate data segment 342 with LIDs 6657-7424 (e.g., by using persistent metadata 344 of packet 340). The forward map 160 may be updated to associate the LIDs 6657-7424 with the data segment 342, which may include splitting the entry 364 into an entry 365 configured to continue to reference the unmodified portion of the data in the data segment 312, and an entry 367 that references the new data segment 342 stored at memory addresses 78512-79024. In Fig. In the copy-on-write mode shown in Figure 3D, entry 362 corresponding to LIDs 1024-2048 may be unchanged and continue to reference data segment 312 at memory addresses 64432-65456. Although in Fig. 3D, modifications within the range 1024-2048 may result in similarly divergent changes affecting entry 362. Furthermore, the memory request(s) is / are not limited to modifying and / or overwriting data. Other operations may include extending the set of LIDs (appending data), removing LIDs (deleting, shortening, and / or truncating data), and / or the like.
[0064] In some embodiments, the storage layer 130 may support other cloning modes, such as a "synchronized clone" mode. In a synchronized clone mode, changes made within a cloned range of LIDs may be reflected in one or more other corresponding ranges. In the embodiment of Fig. 3D, implementing the described memory operation in a "synchronized clone" mode may include updating entry 362 to reference the new data segment 342 as disclosed herein, which may include, among other things, splitting entry 362 into an entry configured to associate LIDs 1024-1536 with portions of the original data segment 312 and adding an entry configured to associate LIDs 1537-2048 with the new data segment 342.
[0065] Again, the “Copy on Write” version of Fig. 3D, the logical interface management module 334 may be further configured to manage clone merge operations. As used herein, a "merge" or „ Clone Merge is an operation to combine two or more different sets and / or ranges of LIDs. In the embodiment of Fig. 3D, a range merge operation may include merging entry 362 with the corresponding cloned entries 365 and 367. Logical interface management module 334 may be configured to implement range merge operations according to a merge strategy, such as: a write job strategy where newer changes overwrite older changes; a priority-based strategy based on the relative priority of storage operations (e.g., based on characteristics of storage client(s) 106, applications, and / or users associated with the storage operations); a completion indicator (e.g., completion of an atomic store operation, failure of an atomic store operation, or the like); fadvise parameters; ioctrl parameters; and / or the like.
[0066] Fig. 3E shows an embodiment of a range merge operation. The range merge operation of Fig. 3E may include merging the range 6144-6656 into the range 1024-2048. Accordingly, the range merge operation may include selectively applying changes made within the LID range 6144-6656 to the LID range 1024-2048 in accordance with the merge strategy. The range merge operation may therefore include updating the LID range 1024-2048 to associate the LIDs 1537-2048 with the memory addresses 78512-79024 comprising the new / modified data segment 342. The update may include splitting the entry 362 in the forward map 160; entry 372 may be configured to associate LIDs 1024-1536 with portions of the original data segment 312, and entry 373 may be configured to associate LIDs 1537-2048 with the new data segment 342.Portions of data segment 312 that are no longer referenced by LIDs 1537-2048 may be invalidated, as disclosed herein. The LID range 6144-7168 that was merged into the original source range may be deallocated and / or removed from forward mapping 160.
[0067] The range merge operation that Fig. 3E, may result in a change in the logical interface 311C to portions of the data. The context-dependent format of the data segment 342 (the data packet 340) may associate the data segment 342 with the LIDs 6657-7168 instead of the merged LIDs 1537-2048. As disclosed above, the storage layer 130 may provide access to the data segment 342 stored in the inconsistent context-dependent format. The storage layer 130 may be configured to store the data segment 342 in an updated context-dependent format in which the data segment 342 is associated with the LIDs 1537-2048 in one or more background operations (e.g., memory recovery operations).In some embodiments, the range merge operation may further include storing a persistent annotation 366 on the storage medium 140 to associate the data segment 342 with the updated logical interface 311C (e.g., to associate the data segment 342 at memory addresses 78512-79024 with LIDs 1537-2048). As disclosed above, the persistent annotation 366 may be used to ensure that the range merge operation is persistent and crash-safe. The persistent annotation 366 may be removed in response to relocating the data segment 342 in a context-dependent format consistent with the logical interface 311C (e.g., associating the data segment 342 with LIDs 1537-2048), persistently storing the forward map 160, and / or the like.
[0068] The in connection with Fig. 3A-E can be used to implement other logical operations, such as a range shift operation. Back to Fig. Referring to Figures 3A-C, a clone operation to replicate entry 362 of forward map 160 may include modifying the logical interface associated with data segment 312 to associate data segment 312 with both the original set of LIDs 1024-2048 and a new set of cloned LIDs 6144-7168 (from entry 364). The clone operation may further include storing a persistent annotation 366 indicating the updated logical interface 311B of data segment 312 and / or rewriting data segment 312 according to the updated logical interface 311B in one or more backend storage operations.
[0069] The logical interface management module 334 may be further configured to implement "scope move" operations. As used herein, a "scope move" operation refers to modifying the logical interface of one or more data segments to associate the data segments with different sets of LIDs. A scope move operation may therefore include updating the storage metadata 135 (e.g., the forward map 160) to associate the one or more data segments with the updated logical interface, storing a persistent annotation 366 on the storage medium 140 indicating the updated logical interface of the data segments, and rewriting the data segments in a context-dependent format (packet format 310) consistent with the updated logical interface, as disclosed herein.Accordingly, the storage layer 130 may implement range shift operations using the same mechanisms and / or processing steps as those described above in connection with the . Fig. 3A-E revealed.
[0070] The Fig. 3A-E may impose certain restrictions on the storage layer 130. As disclosed above, storing data in a context-dependent format may include associating the data with each LID that references the data. In the embodiment of Fig. 3C, the persistent metadata 324 includes references to both LID ranges 1024-2048 and 6144-7168. Increasing the number of references to a data segment may therefore impose a corresponding increase in the overhead of the context-dependent data format (e.g., increasing the size of the persistent metadata 324). In some embodiments, the size of the persistent metadata 314 may be limited, which may limit the number of references and / or clones that can reference a particular data segment 312. Furthermore, including multiple references to different LID(s) may complicate memory reclamation operations. The number of forward map entries that must be updated when a data segment 312 is relocated may vary depending on the number of LIDs that reference the data segment 312. Fig. Referring to Figure 3C, relocating the data segment 312 in a cleanup and / or reclaim operation may include updating two separate entries 362 and 364. Relocating a data segment referenced by N different LIDs (e.g., N different clones) may include updating N different entries in the forward map 160. Similarly, saving the data segment may include writing N entries to the persistent metadata 314. This variable overhead may reduce the performance of background reclaim operations and may limit the number of concurrent clones and / or references that can be supported.
[0071] In some embodiments, the logical interface management module 334 may include and / or employ an intermediate mapping layer to reduce the overhead imposed by clone operations. The intermediate mapping layer may include "reference entries" configured to enable efficient clone operations (as well as other operations as disclosed in more detail herein). As used herein, a "reference entry" refers to an entry of a mapping data structure used to reference other entries within the forward map 160 (and / or other storage metadata 135). A reference entry may only exist while being referenced by one or more other entries within the logical address space 132. In some embodiments, reference entries may not be accessible to the storage clients 106 and / or may be immutable.Storage layer 130 may use reference entries to allow storage clients to reference the same data set through multiple different logical interfaces via a single reference entry interface. The context-dependent format of data on storage medium 140 (data referenced by multiple LIDs) may be simplified to associate the data with the reference entries, which in turn are associated with N other logical interfaces through other persistent metadata (e.g., persistent annotations 366). Relocating cloned data may therefore involve updating a single mapping between the reference entry and the new storage address of the data segment.
[0072] Fig. 4A is a block diagram of another embodiment of a system 400 for efficient open-to-close consistency. System 400 includes a storage layer 130 configured to implement range clone operations using an intermediate mapping layer. Storage metadata 135 may include a forward map 160 related to logical address space 132. Forward map 160 (and / or other storage metadata 135) may include information related to logical address space mappings by storage clients 106, bindings between LIDs and storage addresses within storage address space 144, and so forth, as disclosed above.
[0073] In the embodiment of Fig. 4A, the logical interface management module 334 may include a reference module 434 configured to manage clone operations using a reference map 460. The reference map 460 may include reference entries corresponding to data referenced by one or more logical interfaces of the logical address space 132 (e.g., one or more sets of LIDs). The reference module 434 may be configured to remove reference entries that are no longer used to reference valid data and / or are no longer referenced by entries within the forward map 160. As shown in Fig. 4A, reference entries may be maintained separately from the forward map 160 (e.g., in a separate reference map 460). The reference entries may be identified using reference identifiers, which may be maintained in a separate namespace from the logical address space 132. Accordingly, the reference entries may be part of an intermediate, "virtual" or "reference" address space 432 that is separate and distinct from the logical address space 132 directly accessible to the storage clients 106 via the storage layer interface 131. Alternatively, in some embodiments, reference entries may be assigned LIDs selected from predetermined ranges and / or portions of the logical address space 132 that are not directly accessible to the storage clients 106.
[0074] The logical interface management module 334 may be configured to implement cloning operations by linking one or more LID entries in the forward map 160 to reference entries in the reference map 460. The reference entries may be bound to the memory address(es) of the cloned data. Accordingly, LIDs associated with cloned data may indirectly reference the underlying data through the reference map 460 (e.g., the LID(s) may map to reference entries, which in turn map to memory addresses). Accordingly, entries in the forward map 160 that correspond to cloned data may be referred to as "indirect entries." As used herein, an "indirect entry" refers to an entry in the forward map 160 that references and / or is linked to a reference entry in the reference map 460.Indirect entries may be assigned an LID within the logical address space 132 and may be accessible to the storage clients 106.
[0075] As disclosed above, after cloning a particular set of LIDs, the storage clients 106 may perform storage operations within one or more of the cloned regions, which may result in the clones diverging from each other (according to the clone mode). In a "copy on write" mode, changes made to a particular clone may not be reflected in the other cloned regions. In the embodiment of Fig. 4A, changes made to a clone may be reflected in "local" entries associated with an indirect entry. As used herein, a "local entry" refers to a portion of an indirect entry that maps directly to one or more memory addresses of storage medium 140. Accordingly, local entries may be configured to reference data that has been modified in a particular clone and / or differs from the content of other clones. Local entries may therefore correspond to data that is unique to a particular clone.
[0076] The translation module 134 may be configured to access data associated with cloned data using, among other things, the reference map 460 and / or the reference module 434. The translation module 134 may implement a cascaded lookup, which may include traversing local entries first and, if the leading target identifier(s) are not found in the local entries, continuing the traversal in the reference entries to which the indirect entry is linked.
[0077] The log storage module 136 and the media management module 370 may be configured to manage the context-dependent format of the cloned data. In the embodiment of Fig. 4A, the cloned data (data referenced by two or more LID ranges in the forward map 160) may be stored in a context-dependent format that associates the data with one or more reference entries of the reference map 460. The persistent metadata stored with such cloned data segments may correspond to a single reference entry, rather than identifying each LID associated with the data segment. Therefore, creating a clone may include updating the context-dependent format of the cloned data in one or more background operations, including by using the media management module 370, as disclosed above.
[0078] Fig. Figure 4B shows an embodiment of a clone operation using a reference map 460. In state 413A, an entry having LID 10 extension 2 in logical address space 132 (in Fig. 4B (labeled 10.2) directly references data at memory address 20000 on storage medium 140. Other entries are in Fig. 4B to avoid obscuring the details of the disclosed embodiment. In state 413B, the storage layer 130 implements an operation to clone the range 10,2. Cloning the range 10,2 may include: a) Assigning a new range of LIDs (in Fig. 4B as 400,2) in the logical address space and b) allocating reference entries in the reference map 460, through which the entries 10,2 and 400,2 can reference the cloned data at memory address 20000 (in Fig. 4B as 100000,2). The clone operation may further include associating entries 10,2 and 400,2 with reference entry 100000,2, as illustrated at state 413C. As disclosed above, associating entries 10,2 and 400,2 with reference entry 100000,2 may include indicating that entries 10,2 and 400,2 are indirect entries. State 413C may further include storing a persistent annotation 366 on storage medium 140 to associate the data at memory address 20000 with reference entry 100000,2 and / or to associate entries 10,2 and 400,2 with reference entry 100000,2 in reference map 460.
[0079] The storage layer 130 may provide access to the data segment at memory address 20000 through either LID 10 or 400 (through the reference entry 100000,2). In response to a request referring to LID 10 or 400, the translation module 134 may determine that the corresponding entry in the forward map 160 is an indirect entry associated with an entry in the reference map 460. In response, the reference module 434 performs a cascade to determine the memory address using local entries within the forward map 160 (if applicable) and the corresponding reference entries in the reference map 460 (e.g., reference entry 100000,2).
[0080] Creating the clone at step 413C may include modifying the logical interface of the data segment stored at step 20000 to associate the data with both LID ranges 10,2 and 400,2. However, the context-dependent format of the data may only associate the data with LIDs 10,2. As disclosed above, creating the clone may further include storing a persistent annotation 366 on storage medium 140 to associate the data segment with LIDs 10,2 and 400,2 through reference entry 100000,2. The data segment may be rewritten in an updated context-dependent format in one or more background operations performed by media management module 370. The data can be stored with persistent metadata 314 that associates the data segment with the reference entry 100000,2 instead of with the separate LID ranges 10,2 and 400,2.Therefore, relocating the data segment (as shown in state 413D) may require updating only a single entry in the reference map 460, rather than multiple entries corresponding to each LID range that references the data (e.g., multiple entries 10,2 and 400,2). Furthermore, any number of LID ranges in the forward map 160 may reference the data segment without increasing the size of the persistent metadata 314 associated with the data on the storage medium 140 and / or complicating the operation of the media management module 370.
[0081] Fig. 4C shows another embodiment of a clone operation implemented using reference entries. In response to a request to create a clone of LIDs 1024-2048 and / or data segment 312, logical interface management module 334 may be configured to allocate a reference entry 482 in reference map 460 to represent data segment 312. Any number of LID(s) in forward map 160 may reference the data through reference entry 482 without increasing the overhead of the persistent metadata associated with data segment 312 and / or complicating the operation of media management module 370. As shown in Fig. 4C, the reference entry 482 may be bound to the memory addresses of the data segment 312 (memory addresses 64432-65456). Entries 462 and 472 in the forward map 160 may reference the memory addresses indirectly, through the reference entry 482 (e.g., may be linked to the reference entry 482, as shown in Fig. 4C illustrated).
[0082] In the embodiment of Fig. 4C, identifiers 0Z-1024Z are assigned to reference entry 482. The identifier(s) of reference entry 482 may correspond to a particular portion of logical address space 132 or may correspond to another, separate namespace. Storage layer 130 may link entries 462 and 472 to reference entry 482 using, among other things, metadata associated with entries 462 and / or 472. Alternatively or additionally, indirect entries 462 and / or 472 may replace storage address metadata with references and / or links to reference entry 482. Reference entry 482 may not be directly accessible to storage clients 106 via storage layer 130.
[0083] The clone operation may further include modifying the logical interface 311D of the data segment 312; the modified logical interface 311D may allow the data segment 312 to be referenced by the LIDs 1024-2048 of the indirect entry 462 and / or 6144-7168 of the indirect entry 472. Although the reference entry 482 may not be accessible to the storage clients 106, the reference entry 482 may be used to access the data by the translation module 134 (via the indirect entries 462 and 472) and, as such, may be considered part of the modified logical interface 311B of the data segment 312.
[0084] The clone operation may further include storing a persistent note 366A on the storage medium 140. As disclosed above, storing the persistent note(s) 366A and / or 366B may ensure that the clone operation is persistent and crash-safe. The persistent note 366A may be configured to identify the reference entry 482 associated with the data segment 312. Accordingly, the persistent note 366A may associate the memory addresses 64432-65456 with the reference entry identifier(s) 0Z-1024Z. The clone operation may further include storing another persistent note 366B configured to associate the LIDs of the entries 462 and / or 472 with the reference entry 482. Alternatively, metadata relating to the association between entries 462, 472 and 482 may be contained in a single persistent annotation.The persistent annotations 366A and / or 366B may be retained on the storage medium 140 until the data segment 312 is relocated in an updated context-dependent format and / or the forward map 160 (and / or the reference map 460) is persistently stored.
[0085] The modified logical interface 311D of the data segment 312 may be inconsistent with the context-dependent format of the original data packet 410A; the persistent metadata 314A may reference the LIDs 1024-2048 instead of the reference entry 482 and / or the cloned entry 472. The storage layer 130 may be configured to store the data segment 312 in an updated context-dependent format (packet 410B) consistent with the modified logical interface 311D; the persistent metadata 314B may associate the data segment 312 with the reference entry 482 instead of separately identifying the LID(s) in each cloned region (e.g., entries 462 and 472). Accordingly, the use of the indirect entry 482 allows the logical interface 311D of the data segment 312 to include any number of LIDs, regardless of size limitations of the persistent metadata 314A-B.Furthermore, additional clones of the reference entry 482 may be made without updating the context-dependent format of the data segment 312; these updates may be made by associating the new LID ranges with the reference entry 482 in the forward map 160 and / or by using, among other things, the persistent annotations 366.
[0086] As disclosed above, the indirect entries 462 and / or 472 may initially reference the data segment 312 through the reference entry 482. Storage operations performed after the clone operation may be reflected in the forward map 160 using local entries. After the clone operation is complete, the storage layer 130 may modify data associated with one or more of the cloned LID(s). In the embodiment of Fig. 4D, a storage client 106 modifies and / or overwrites data corresponding to LIDs 1024-1052 of indirect entry 462, which may include appending a new data segment 412 to the storage log (in data packet 420 at memory addresses 7823-7851).
[0087] The data segment 412 may be stored in a context-dependent format (data packet 420) that includes persistent metadata 414A configured to associate the data segment 412 with the LIDs 1024-1052. The storage layer 130 may be configured to associate the data segment 412 with the LIDs 1024-1052 in a local entry 465. The local entry 465 may directly reference the updated data, rather than referencing the data through the indirect entry 462 and / or the reference entry 482.
[0088] In response to a request for the data 1024-1052 (or a subset thereof), the logical interface management module 334 may search for references to the requested LIDs in a cascaded lookup operation, which may include searching for references to local entries (if available) followed by the reference entries. In the embodiment of Fig. 4D, the local entry 465 can be used to satisfy requests related to the LID range 1024-1052 (memory addresses 7823-7851) instead of 64432-64460 for the reference entry 462. Requests for LIDs not found in a local entry (e.g., LIDs 1053-2048) can still be served by the reference entry 482. The logical interface 311E of the data related to the range 1024-2048 can therefore include one or more local entries 465, one or more indirect entries 462, and / or one or more reference entries 482.
[0089] In a further embodiment, in Fig. 4E, a storage layer 130 may modify data of the clone by another of the LIDs of the logical interface 311E (e.g., LIDs 6144-6162); the logical interface delimiters are shown in Fig. 4E to avoid obscuring the details of the illustrated embodiment. The modified data may be referenced using a local entry 475, as disclosed above. In the embodiment of Fig. 4E, each of the areas 462 and 472 has its own respective local version of the data formerly referenced by identifiers 0Z-52Z of the reference entry 482. As such, neither entry 462 nor 472 contains a reference to the area 0Z-52Z. The reference module 434 may determine that the corresponding data (and reference identifiers) are no longer referenced and may, as such, be marked (e.g., invalidated) for removal from the storage medium 140. As in Fig. 4E, invalidating the data may include removing the references to the data from the reference map 460 by, among other things, modifying the reference entry 482 to remove the range 0Z-52Z. Invalidating the data may further include updating other storage metadata 135, such as a reverse map, validity bitmaps, and / or the like (e.g., to indicate that the data stored at memory addresses 64432-64484 does not need to be retained). The ranges of the entries 462 and 472 may continue to diverge until neither references any portion of the reference entry 482, at which time the reference entry 482 may be removed and the data referenced thereby invalidated, as disclosed above.
[0090] Although Fig. 4D and Fig. 4E show local entries 465 and 475 that include overlapping LID ranges with the corresponding indirect entries 462 and 472, the disclosure is not limited in this regard. In some embodiments, the storage operation of Fig. 4D by creating the local entry 465 and modifying the indirect entry 462 to reference only the LIDs 1053-2048. Similarly, the operation of Fig. 4E may include creating the local entry 475 and modifying the indirect entry 472 to reference a truncated LID range 6163-7168.
[0091] Back on Fig. Referring to Figure 4A, the reference module 434 may be configured to manage or "clean" the reference map 460. In some embodiments, each entry in the reference map 460 includes metadata including a reference count. The reference count may be incremented as new references or links are added to the reference entry and may be decremented in response to removal of references to the entry. In some embodiments, reference counts may be maintained for each reference identifier in the reference map 460. Alternatively, reference counts may be maintained for reference entries as a whole. When a reference entry's reference count reaches 0, the reference entry (and / or a portion thereof) may be removed from the reference map 460.Removing a reference entry (or a portion of a reference entry) may include invalidating the corresponding data on storage medium 140 as disclosed herein (indicating that the data no longer needs to be retained).
[0092] In another embodiment, the reference module 434 may remove reference entries using a mark-and-sweep approach. The reference module 434 (or another process such as the translation module 134) may periodically check references to entries in the reference map 460 by, among other things, following links to the reference entries from indirect entries (or other types of entries) in the forward map 160. Reference entries not accessed during the mark-and-sweep process may be removed, as disclosed above. The mark-and-sweep process may operate as a background process and may periodically perform a mark-and-sweep operation to identify and remove reference entries that are no longer in use.
[0093] In some embodiments, the reference map 460 disclosed herein may be created on request (e.g., in response to creating a clone or other indirect data reference). In other embodiments, all data storage operations may be performed through intermediate maps. In such embodiments, the storage clients 106 may allocate indirect virtual address space (VAS) identifiers (VIDs), which may be linked to and / or referenced by an intermediate mapping layer such as the logical address space 132. The VAS may add an intermediate mapping layer between the storage clients 106 and the storage medium 140.Storage clients 106 may reference data using virtualized address space VIDs, which map to logical identifiers of logical address space 132 and which, in turn, are associated with storage addresses on a respective storage device(s) 141 and / or storage medium 140. As used herein, a VAS may include, but is not limited to, a Logical Unit Number (LUN) address space, a Virtual LUN (vLUN) address space, and / or the like.
[0094] Fig. 5A illustrates one embodiment of an aggregation layer 530 configured to implement, among other things, efficient range clone operations using a virtualized address space 532. The aggregation layer 530 may be configured to present a VAS 532 to storage clients 106 via an interface 531. Like the interface 131 disclosed herein, the interface 531 may include one or more of: a block device interface, a virtual storage interface, a cache interface, and / or the like. Storage clients 106 may perform storage operations related to storage resources managed by the aggregation layer 530 by referencing VIDs of the VAS 532 through the interface 531.
[0095] The aggregation layer 530 may further include a VAS translation module 534 configured to map VIDs through one or more intermediate storage layers (e.g., storage layer 130) to storage resources. Accordingly, the VAS metadata 535 of the aggregation layer 530 may include a VAS forward mapping 560 comprising mappings from any element to any element between VIDs of the VAS 532 and LIDs of the VAS 532. Although not described in Fig. 5A, the VAS translation module 534 and / or the VAS forward map 560 may be configured to aggregate a plurality of logical address spaces 132 of a plurality of different storage tiers 130 into a single VAS 532. Accordingly, in some embodiments, a VAS 532 may correspond to a plurality of different logical address spaces 132, each comprising a separate set of LIDs and each corresponding to a respective storage tier 130, a respective storage device 141, and / or storage medium 140.
[0096] Although Fig. While Figure 5A shows the aggregation layer 530 separate from the storage layer 130, the disclosure is not limited in this regard. In some embodiments, the VAS 532, the VAS forward mapping 560, the VAS translation module 534, and / or other modules of the aggregation layer 530 may be implemented as part of the storage layer 130.
[0097] The aggregation layer 530 may be configured to utilize the intermediate virtual address space provided by the VAS 532 to implement, among other things, efficient range cloning, moving, merging, and / or other high-level operations. Alternatively or additionally, the intermediate mapping layer(s) may be employed to enable efficient cloning operations on write-in-situ random-access storage devices such as hard disks and / or the like.
[0098] The storage clients 106 may perform storage operations with respect to VIDs of the VAS 532. Accordingly, storage operations may include two (or more) translation layers. The VAS forward map 560 may include a first translation layer between VIDs of the VAS 532 and identifiers of the logical address space 132 of the storage layer 130. The forward map 160 of the storage layer 130 may implement a second translation layer between LIDs and storage address(es) on the storage medium 140.
[0099] The aggregation layer 530 may be configured to manage mappings in the VAS 532 using, among other things, the VAS metadata 535, the VAS forward map 560, and / or the VAS translation module 534. In some embodiments, mapping a VID in the VAS 532 may include mapping one or more corresponding LIDs in the logical address space 132 (and / or identifiers of one or more other storage layers). Accordingly, each VID mapped in the VAS 532 may correspond to one or more LIDs of the logical address space 132. The mappings from any element to any element between the VIDs of the aggregation layer 530 and the logical address space 132 may be sparse and / or from any element to any element, as disclosed herein.Furthermore, in some embodiments, the aggregation layer 530 may be configured to maintain any-element-to-any-element mappings and / or range-managed mappings between VIDs and a plurality of different logical address spaces 132. Accordingly, the aggregation layer 530 may aggregate and / or combine the logical address spaces of a plurality of different storage devices 141, each managed by different storage layers 130, into a single, aggregated VAS 532.
[0100] In the embodiment of Fig. 5A, the logical address space 132 may not be directly accessible, and as such, the storage clients 106 may reference storage resources using VIDs via the interface 531. Therefore, performing a storage operation by the aggregation layer 530 with respect to one or more VIDs may include: a) identifying the storage layer 130 corresponding to the VIDs, b) determining the LID(s) of the storage layer 130 that map to the VIDs using the VAS translation module 534 and / or the VAS forward map 560; and c) implementing the storage operation using the storage layer 130 with respect to the determined LID(s).
[0101] Fig. 5B illustrates one embodiment of a clone operation implemented using the aggregation layer 530. As disclosed above, the VAS forward map 560 may correspond to a VAS 532 indirectly mapped to memory addresses by a logical address space 132 of a storage layer 130. Fig. 5B illustrates the addressing layers used to implement memory operations by the aggregation layer 530. The VIDs of the VAS 532 may be the upper-level addressing layer accessible to the storage clients 106 via, among other things, the interface 531 of the aggregation layer 530. The logical address space 132 of the storage layer 130 may include an intermediate addressing layer. The VAS forward map 560 may include mappings from any element to any element between VIDs and LIDs. The LIDs may be mapped to memory addresses within the memory address space 144 using the forward map 160. Accordingly, VIDs may be mapped to the memory address space 144 through the intermediate logical address space of the storage layer 130.
[0102] As in Fig. 5B, the VAS forward map 560 in state 563A may include an entry 10,2 representing two VIDs (10 and 11) in the VAS 532. The VAS forward map 560 associates the VID entry 10,2 with LIDs of the logical address space 132. In the embodiment of Fig. 5B, VAS forward mapping 560 binds VID entry 10,2 to LIDs 100000 and 100001 (entry 100000,2). Entry 10,2 may be associated with a particular storage client 106 that may perform storage operations with respect to the VIDs. In state 563A, storage layer 130 may be configured to map entry 100000,2 to one or more storage addresses on storage medium 140 (storage address 20000).
[0103] In state 536B, the aggregation layer 530 may implement a clone operation to clone the VID entry 10,2. The clone operation may include: a) allocating a new VID entry 400,2; and b) associating the new VID entry 400,2 with the corresponding entry 100000,2 in the VAS forward map 560. The corresponding entry 100000,2 in the forward map 160 may remain unchanged. Alternatively, a reference counter (or other indicator) of the entry 100000,2 in the forward map 160 may be updated to indicate that the entry is referenced by multiple VID ranges. The context-dependent format of the data stored at memory address 20000 can remain unchanged (e.g., continue to associate the data with the logical interface 100000,2).The clone operation may further include storing a persistent annotation 366 on storage medium 140 to indicate the association between VID entry 400,2 and entry 100000,2 in forward map 160. Alternatively or additionally, the clone operation may be made persistent and / or crash-safe by persistently storing VAS forward map 560 (and / or portions thereof).
[0104] In state 536C, the data at memory address 20000 may be relocated to memory address 40000. The relocation may occur in a standard storage medium maintenance operation and may not update the context-dependent format of the cloned data. Relocating the data may involve updating a single entry in forward map 160. VAS forward map 560 may remain unchanged. Changes to the various versions of VID ranges 10,2 and 400,2 may be managed by the intervening logical address space. A modification to VID 10 may involve: a) allocating a new LID in logical address space 132, b) storing the modified data in association with the new LID, and c) mapping the new LID to VID 10 in VAS forward map 560.
[0105] The embodiments for implementing range clone, move, and / or merge operations disclosed herein may be used to efficiently implement other higher-level storage operations, such as snapshots, deduplication, atomic operations, transactions, file system management functionality, and / or the like. Fig. 4A, the storage layer 130 may include a deduplication module 374 configured to identify duplicate data on the storage medium 140. Duplicate data may be identified using any suitable mechanism. In some embodiments, duplicate data is identified by: a) scanning the contents of the storage medium 140, b) generating signature values for various data segments, and c) comparing the data signature values to identify duplicate data. The signature values may include, but are not limited to, cryptographic signatures, hash codes, cyclic codes, and / or the like. Signature information may be stored in the storage metadata 135, such as the forward map 160 (e.g., in metadata associated with the entries), and / or may be maintained and / or indexed in one or more separate data structures of the storage metadata 135.The deduplication module 374 may compare data signatures and, upon detecting a signature match, perform one or more deduplication operations. The deduplication operations may include verifying the signature match (e.g., performing a byte-by-byte comparison) and performing one or more range clone operations to reference the duplicate data by two or more LID ranges.
[0106] Fig. 6 shows one embodiment of a deduplication operation. The forward map 160 may include entries 662 and 672 that reference duplicate data stored at different memory addresses 3453-4477 and 7024-8048, respectively. Entries 662 and 672 may correspond to different respective logical interfaces 663 and 673, corresponding to LIDs 1024-2048 and 6144-6656, respectively. The duplicate data segment (data segment 612) may be identified and / or verified by the deduplication module 374, as disclosed above. Alternatively, the duplicated data may be identified when data is received for storage at the storage layer 130. Accordingly, the data may be deduplicated before an additional copy of the data is stored on the storage medium 140.
[0107] In response to identifying and / or verifying that entries 662 and 672 reference duplicate data, storage layer 130 may be configured to deduplicate the data, which may include creating one or more range clones to reference a single copy of the duplicate data through two different sets of LIDs. As disclosed above, creating a range clone may include modifying the logical interface(s) 663 and 673 of a data segment. In the embodiment of Fig. 6, the duplicate data is stored as a data segment 612 in a packet 610 at memory locations 3453-4477 and 7024-8048, respectively. The clone operation may include modifying the logical interface of one of the data segments (or a new version and / or copy of the data segment) so that the data segment can be referenced by both entries 663 and 673.
[0108] The range clone operation may be implemented using any of the cloning embodiments disclosed herein, including the range cloning embodiments of Fig. 3A-E, the reference entry embodiments of Fig. 4A-E and / or the intermediate embodiments of Fig. 5A-B. In the deduplication embodiment of Fig. 6, both LID ranges 1024-2048 and 6144-7168 may be modified to reference a single version of the data segment 612 through a reference entry 682 (the other data segment may be invalidated). As such, the deduplication operation may include creating a reference entry 682 to represent the deduplicated data segment 612 (to reference the packet 610). The deduplication operation may further include modifying and / or converting the entries 662 and 672 into respective indirect entries 665 and 675, which may be mapped to the data segment 612 through the reference entry 682, as disclosed above. The deduplication operations may further include modifying the logical interface 669 of the data segment 612 to associate the data segment 612 with both sets of LIDs 1024-2048 and 6144-7168 (as well as the reference entry 682).The deduplication operations may further include storing a persistent note 366 on the storage medium 140, as disclosed above.
[0109] The deduplication operation may further include updating the context-dependent format of the data segment 612 to be consistent with the modified logical interface 669, as disclosed above. Updating the context-dependent format may include appending the data segment 612 in an updated context-dependent format (data packet 610) to the storage log (e.g., at locations 84432-85456) in one or more background operations. The updated data packet 610 may include persistent metadata 614 associating the data segment 612 with the updated logical interface 669 (e.g., LIDs 1024-2048 and 6144-6656 through reference identifiers 0Z-1023Z).
[0110] Although Fig. 6 illustrates cloning and / or deduplicating a single entry or range of LIDs, the disclosure is not limited in this regard. In some embodiments, a plurality of leading identifier ranges may be cloned in a single clone operation. This type of clone operation may be used to create a "snapshot" of an address space (or the entire logical address space 132). As used herein, a snapshot refers to the state of a storage device (or set of LIDs) at a particular point in time. The snapshot may maintain an "original" state of an LID range regardless of changes that occur within the range after the snapshot operation is completed.
[0111] Fig. 7 is a block diagram illustrating one embodiment of a system 700 including a storage layer 130 configured to efficiently implement storage snapshot operations. The embodiment of Fig. 7 refers to an address space within a logical address space 132. However, the disclosure is not limited in this regard and could be adapted for use with other types of address spaces, such as ranges and / or extensions within a VAS 532, as disclosed above. The storage layer 130 may include a snapshot module 736 and a timekeeping module 738 configured to implement snapshot operations as disclosed herein.
[0112] In state 773A, the storage layer 130 may be configured to create a snapshot of an LID region FR1. Creating the snapshot may include obtaining the state of the LID region FR1 at a particular time. The snapshot operation may further include obtaining the LID region FR1 while allowing subsequent storage operations to be performed in the LID region.
[0113] As disclosed above, the storage layer 130 may be configured to store data in a storage log on the storage medium 140, including through the use of the log storage module 136. The log order of storage operations may be determined using sequence information associated with data packets, such as the sequence indicators 113 on the storage sections 170A-N and / or sequential storage locations in the memory address space 144 of the storage medium 144 (as described in connection with Fig. 1D and Fig. 1E).
[0114] The storage layer 130 may be further configured to maintain other types of ordering and / or timing information, such as the relative temporal ordering of data in the log. In some embodiments, the log order of data may not accurately reflect the timing information, due, among other things, to the relocation of data in the storage device during media management operations. Relocating data may include reading the data from its original storage location on the storage medium 140 and appending the data at a current append point in the storage log. As such, older, relocated data may be stored with newer, current data in the storage log. Therefore, although the storage log may maintain the relative log order of data operations related to particular LIDs, the storage log may not accurately reflect absolute timing information.
[0115] In some embodiments, the log storage module 136 is configured to associate data with timing information that can be used to establish relative timing information of the storage operations performed on the storage medium 130. In some embodiments, the timing information can include respective timestamps (maintained by the timing module 738) that can be applied to each data packet stored on the storage medium 140. The timestamps can be stored within persistent metadata 314 of the data packets 310. Alternatively or additionally, the timing module 738 can be configured to track timing information at a coarser level of granularity. In some embodiments, the timing module 738 maintains one or more global timing indicators (an epoch identifier).As used herein, an "epoch identifier" refers to an identifier used to determine the relative timing of storage operations performed by the storage layer 130. The log storage module 136 may be configured to include an epoch indicator 739 in the data packets 710. The epoch indicator 739 may correspond to the current epoch maintained by the timing module 738 (e.g., the global timing indicator). The epoch indicator 739 may correspond to the epoch in which the corresponding data segment 712 was written to the storage log. The epoch indicator 739 may be stored within the persistent metadata 714 of the packet 710, and as such, may remain associated with the data packet 710 during the relocation operation.The timing module 738 may be configured to increment the global epoch identifier in response to certain events, such as the creation of a new snapshot, a user request, and / or the like. The epoch indicator 739 of the data segment 712 may remain unchanged by relocation and / or other media management operations. Accordingly, the epoch indicator 739 may correspond to the original storage time of the data segment 712, regardless of the relative position of the data packet 710 in the storage log.
[0116] A snapshot operation may involve obtaining the state of a particular LID range (FR1) at a particular time. A snapshot operation may therefore involve obtaining data related to FR1 on the storage medium 140. Obtaining the data may involve: a) identifying data related to a particular time frame (epoch), and b) preserving the identified data on the storage medium 140 (e.g., preventing the identified data from being removed from the storage medium 140 in, among other things, storage recovery operations). Data related to a snapshot may be retained even though it may be invalidated by subsequent storage operations (e.g., operations that overwrite, modify, TRIM, and / or otherwise bypass the data).The data that needs to be obtained for a particular snapshot can be identified by using the epoch indicator 739 disclosed above.
[0117] In state 773A (time t1, indicated by epoch indicator e0), the storage layer 130 may receive a request to implement a snapshot operation. In response to the request, the snapshot module 736 may determine the current value of the epoch identifier maintained by the timekeeping module 738. The current value of the epoch identifier may be referred to as the current "snapshot epoch." In the embodiment of Fig. 7, the snapshot epoch is 0. The snapshot module 736 may be further configured to cause the timekeeping module 738 to increment the current global epoch indicator (e.g., increment the epoch identifier to 1). Creating the snapshot may further include storing a persistent annotation 366 on the storage medium configured to indicate the current, updated epoch indicator. The persistent annotation 366 may be further configured to indicate that data related to the snapshot epoch is to be preserved (e.g., identify the particular range of LIDs F1 as being preserved in the snapshot operation). The persistent annotation 366 may be used during metadata reconstruction operations to: a) determine the current epoch identifier and / or b) configure the snapshot module 736 and / or the media management module 370 to use a particular snapshot epoch (e.g.,Epoch e0) to obtain associated data.
[0118] The snapshot module 736 may be further configured to instruct the media management module 370 to obtain data associated with the snapshot epoch. In response, the media management module 370 may be configured to: a) identify data to be obtained for the snapshot (snapshot data), and b) prevent the identified data from being removed from the storage medium 140, among other things, in storage recovery operations. The media management module 370 may identify snapshot data by using the epoch indicators 739 of the data packets 710. As described in connection with Fig. 1E, data may be written elsewhere on the storage medium 140. The most recent version of data associated with a particular LID may be determined based on the order of the corresponding data packets 710 in the protocol. The media management module 370 may be configured to identify the most recent version of data in the snapshot epoch as data that needs to be preserved. Data that has been rendered obsolete by other data in the snapshot epoch may be removed. The embodiment of Fig. Referring to Figure 1E, if data X0 and X1 (associated with the same LID A) were both marked with snapshot epoch 0, the media management module 370 would identify the most recent version of the data at epoch 0 as X1 and would mark the data X0 for removal. However, if the data X0 was marked with snapshot epoch 0 and X1 was marked with a later epoch (e.g., epoch 1, after the snapshot operation), the media management module 370 may obtain the data X0 on the storage medium 140 to obtain the snapshot data.
[0119] In state 773B, the snapshot module 738 may be configured to obtain data related to the snapshot FR1 (data associated with epoch e0) while allowing storage operations to continue during subsequent epochs (e.g., epoch e1). Obtaining FR1 may include cloning FR1 to preserve the original state of the LID range at epoch e0 (FR1(e0)) while allowing storage operations related to FR1 to continue. The clone operation may be implemented as disclosed above using one or more of the duplicate entries, reference entries, and / or an intermediate mapping layer. The storage operations may include appending data to the storage log on the storage medium 140 related to the LIDs FR1. The cloned LIDs corresponding to the snapshot FR1(e0) may be immutable.Accordingly, the snapshot of FR1 (e0) may be retained despite changes to the LID range. Data stored in state 773B may be saved with an epoch indicator 739 of the current epoch (e1). The snapshot module 736 may be configured to retain data that has been rendered obsolete and / or invalidated by storage operations performed during epoch e1 (and subsequent epochs). Referring back to the embodiment of FIG. Fig. Referring to Figure 1E, the media management module 370 may identify the data X0 as data to be preserved for the snapshot FR1 (the data X1 may have been saved after the snapshot operation was performed). The snapshot module 738 and / or the media management module 370 may be configured to preserve the data X0 even if the data was subsequently made obsolete by the data X1 in epoch e1. The data X0 may be retained even if the LID A is deleted, trimmed, or the like.
[0120] The snapshot of FR1 (e0), including the LID range FR1 (e0) and the data e0 marked with the epoch indicator, may be retained until the corresponding snapshot is deleted. The snapshot may be deleted in response to a request received through interface 131. As indicated in state 773C, epoch 0 may be retained on storage medium 140 even after other, intervening epochs (epochs e1-eN) have been created and / or deleted. Deleting epoch e0 may include configuring snapshot module 738 and / or media management module 370 to remove the invalid / stale data associated with epoch e0.
[0121] Storage operations performed after taking the snapshot at state 773A may modify the logical address space 132, and in particular, the forward map 160. The modifications may include updating storage address bindings in response to appending the data to the storage medium 140, adding and / or removing LIDs to FR1, and so on. In some embodiments, the snapshot module 736 is configured to maintain the snapshot region FR1 (e0) in separate storage metadata 135, such as a separate subregion of the logical address space 132, in a separate namespace, in a separate map, and / or the like. Alternatively, the snapshot module 736 may allow the changes to occur in the forward map 160 without maintaining the original version of FR1 at time e0.The snapshot module 736 may be configured to reconstruct the forward map 160 for e0 (time t1) using the snapshot data obtained on the storage medium 140. The forward map 160 at time t1 may be reconstructed as disclosed above, which may include sequentially accessing data stored on the storage medium 140 (in log order) and creating forward map entries based on persistent metadata 714 associated with the data packets 710. In the embodiment of FIG. Fig. 7, the forward map 160 corresponding to epoch e0 can be reconstructed by referencing data packets 710 marked with epoch indicator 739 e0 (or lower). Data associated with epoch indicators 739 greater than e0 can be ignored (since such data corresponds to operations after the creation of snapshot FR1 (e0)).
[0122] The storage layer 130 disclosed herein may be further configured to implement efficient range shift operations. Fig. 8A shows one embodiment of a move operation implemented by the storage layer 130 disclosed herein. The forward map 160 includes entries 862 configured to bind the LIDs 1023-1025 to respective data segments on the storage medium 140. The entries 862 are shown separately to better illustrate the details of the embodiment; however, the entries 862 could be included in a single entry encompassing the full range of LIDs 1023-1025. The entries 862 may define a logical interface 863 of the data stored at memory addresses 32, 3096, and 872. As disclosed above, the data stored at memory addresses 32, 3096, and 872 may be stored in a context-dependent format that associates the data with the corresponding LID(s) 1023, 1024, and 1025.
[0123] Storage layer 130 may be configured to move entries 862 to LIDs 9215-9217, among other things, by replacing the association between LIDs 1023, 1024, and 1025 and the data at the respective media storage orders 32, 3096, and 872 with a new logical interface 863B corresponding to the new set of LIDs (e.g., 9215, 9216, and 9217). The move operation may be performed in response to a request received via interface 131 and / or as part of a higher-level storage operation (e.g., a request to rename a file, operations to rebalance and / or defragment the forward map 160, or the like).
[0124] The move operation may be implemented according to one or more of the cloning embodiments disclosed above. In some embodiments, the move operation may include associating the memory addresses mapped to LIDs 1023, 1024, and 1025 with destination LIDs 9215, 9216, and 9217, which may result in modifying the logical interface 863A of the data in accordance with the move operation. The move operation may further include storing a persistent note 366 on the storage medium 140 to ensure that the move operation is persistent and crash-safe. The data stored at memory addresses 32, 872, and 3096 may be rewritten in accordance with the updated logical interface 836B in one or more background operations, as disclosed above.
[0125] Fig. Figure 8B shows one embodiment of a move operation. As above, the move operation may include moving the data associated with LIDs 1023-1025 to LIDs 9215-9217. The move operation of Fig. 8B can use the reference entries as in connection with Fig. 4A-E. Accordingly, the move operation may include creating reference entries 882 in a reference map 460 to represent the move operation. The move operation may further include allocating new indirect entries 866 to reference the data through reference entries 882. The reference entries 882 may include pre-move LIDs 1023, 1024, 1025, which may be associated with addresses 32, 3096, and 872. The new logical interface 863C of the data may therefore include the indirect entries 866 and the corresponding reference entries 882. The move operation may further include storing a persistent notation 366 on the storage medium to ensure that the move operation is persistent and crash-safe, as disclosed above.
[0126] The context-dependent format of the data stored at memory addresses 32, 3096, and 872 may be inconsistent with the updated logical interface 863C; the context-dependent format of the data may associate the respective data segments with LIDs 1023, 1024, and 1025 instead of 9215, 9216, and 9217 (and / or the reference entries). The persistent annotation 366 may include the updated logical interface 863C of the data so that the storage metadata 135 (e.g., forward map 160 and / or reference map 460) can be correctly reconstructed if necessary.
[0127] Storage layer 130 may provide access to the data in the inconsistent context-dependent format via modified logical interface 863C (LIDs 9215, 9216, and 9217). The data may be rewritten and / or relocated (outside the path of the move operation and / or other storage operations) in a context-dependent format consistent with modified logical interface 863C after the move operation. In some embodiments, the data at memory addresses 32, 3096, and / or 872 may be rewritten by a media management module 370 in one or more background operations, as described above. Therefore, the move operation may complete (and / or return an acknowledgment) in response to updating forward map 160 and / or saving persistent note 366.
[0128] As in Fig. 8C, the forward map 160 and / or other memory metadata 135 may be updated in response to a rewrite of the data of the move operation. In the embodiment of Fig. 8C, the data segment 812A stored at the media storage location 32 may be relocated in a storage recovery operation, which may include storing the data in a context-dependent format (data packet 810A) consistent with the modified logical interface 863C. The data packet 810A may include persistent metadata 814A associating the data segment 812A with the LID 9215. The forward map 160 may be updated to reference the data in the updated context-dependent format, which may include modifying the indirect entry of the LID 9215 to reference the data packet 810A directly instead of the reference entry. The entry corresponding to the LID 9215 may fall back from an indirect entry to a local default entry, and the reference entry for the LID 1023 may be removed from the reference map 460.
[0129] On Fig. Referring to Figure 8D, a storage client 106 may modify data associated with LID 9217, which may include storing a data segment elsewhere (e.g., at memory address 772). The data segment may be written in a context-dependent format consistent with the modified logical interface 863C (e.g., associating the data with LID 9217). In response, the forward map 160 may be updated to associate the entry for LID 9217 with the memory address of the data segment (e.g., memory address 772) and remove the reference entry for LID 1025 from the reference map 460, as disclosed above.
[0130] In some embodiments, the reference map 460 may be maintained separately from the forward map 160 so that the entries there (e.g., entries 882) cannot be directly referenced by the storage clients 106. This separation may allow the storage clients 106 to operate more efficiently. For example, instead of pausing operations until data is rewritten and / or relocated in the updated context-dependent format, the data operations may continue while the data is rewritten in one or more background processes. Fig. Referring to Figure 8E, following the move operation disclosed above, a storage client 106 may store data associated with LID 1024. The reference entry 882 corresponding to LID 1024 may be included in reference map 460, among other things because the data at memory address 3096 is not yet rewritten in the updated context-dependent format. However, because reference map 460 is maintained separately from forward map 160, no name collision may occur, and the store operation may complete. Forward map 160 may include a separate entry 864 comprising the logical interface for the data stored at media storage location 4322, while continuing to provide access to the data formerly bound to LID 1024 via logical interface 863C (and reference map 460).
[0131] If the indirect entries are no longer linked to reference entries of the reference map 460, due to, among other things, rewriting, relocating, modifying, deleting, and / or overwriting the corresponding data, the reference entries may be removed in the disclosed move operation, and the indirect entries may be converted back into direct, local entries. Additionally, the persistent note 366 associated with the move operation may be invalidated and / or removed from the storage medium 140, as disclosed above.
[0132] Back on Fig. 1A, the interface 131 of the storage layer 130 may be configured to provide APIs and / or interfaces for performing the storage operations disclosed herein. The APIs and / or interfaces may be provided through one or more of: block interfaces, an extended storage interface, and / or the like. The block interface may be extended to include additional APIs and / or functionality using interface extensions, such as fadvise parameters, I / O control parameters, and the like. The interface 131 may provide APIs to perform range clone operations, range move operations, range merge operations, deduplication, snapshot, and other higher-level operations disclosed herein. The interface 131 may allow the storage clients 106 to apply attributes and / or metadata to LID ranges (e.g.,freeze a range), manage range snapshots, and so on. As disclosed herein, a range clone operation comprises creating a logical copy of a set of one or more source LIDs. Range clone, move, and / or merge operations may be implemented using any of the cloning embodiments disclosed herein, including those described in . Fig. 3A-E, the reference entry embodiments of Fig. 4A-E and / or the intermediate imaging layer embodiments of Fig. 5A-B, but not limited thereto.
[0133] The range clone, move, and / or merge operations disclosed herein may be used to implement higher-level operations such as deduplication, snapshots, efficient file copy operations (logical file copies), file consistency management, address space management, mmap checkpointing, atomic writes, and the like. These higher-level operations may also be provided through the interface 131 of the storage layer 130. The disclosed operations may be employed by various different storage clients 106, such as operating systems, file systems, database systems, and / or the like.
[0134] Fig. 9A shows one embodiment of a system 900A including a storage layer 130 configured to implement file management operations. System 900A may include a file system 906 that may be configured to employ the functionality of storage layer 130 to reduce complexity, overhead, and the like. File system 906 may be configured to employ the range clone, move, snapshot, deduplication, and / or other functionality disclosed herein to implement efficient file-level snapshot and / or copy operations. File system 906 may be configured to implement such operations in response to client requests (e.g., a copy command, a file snapshot ioctrl, or the like).The file system 906 may be configured to implement efficient file copy and / or file-level snapshot operations on a source file by, among other things, a) flushing modified pages of the source file (if any), b) creating a new destination file to represent the copied file and / or the file-level snapshot, and c) instructing the storage module 130 to perform a range clone operation configured to clone the source file to the destination file.
[0135] Fig. 9A illustrates various embodiments for implementing range clone operations for a file system 906. In some embodiments, and as shown in state 911A, the storage layer 130 may be configured to maintain a logical address space 132 in which LIDs of the source file (the file to be cloned) are mapped to file data on the storage medium using the forward mapping 160. The corresponding range clone operation, shown in state 911B, may include: a) allocating a set of LIDs for the destination file, and b) mapping the LIDs of the source file and the destination file to the file data on the storage medium 140. The range clone operation may further include storing a persistent annotation 366 on the storage medium 140 to indicate that the file data is associated with both the source file and the destination file LIDs.The range clone operation may further include rewriting the file data in accordance with the updated context-dependent format as disclosed herein.
[0136] In other embodiments, the storage layer 130 may employ a reference map 460 to implement range cloning operations (e.g., as shown in Fig. 4A-E). Prior to the range clone operation, in state 911C, the source file LIDs may be mapped directly to the corresponding file data in the forward map 160. Creating the range clone in state 911D may include associating one or more reference entries in the reference map 460 with the file data and linking indirect entries corresponding to the source file LIDs and the destination file LIDs with the reference entry. The range clone operation may further include storing a persistent annotation 366 on the storage medium 140 and / or updating the context-dependent format of the file data, as disclosed herein.
[0137] In some embodiments, the storage layer 130 may be configured to implement range cloning operations using an intermediate mapping layer (as in Fig. 5A-B). As indicated in state 911E, the source file may correspond to a set of VIDs of a VAS 532 that can be mapped to file data on the storage medium 140 via an intermediate address space (e.g., the logical address space 132 of the storage layer 130). Performing the range clone operation may include: a) allocating VIDs in the VAS 532 for the destination file, and b) associating the VIDs of the destination file with the LIDs of the intermediate mapping layer (e.g., the same set of LIDs mapped to the source file VIDs). The range clone operation may further include storing a persistent annotation 366 on the storage medium 140 indicating that the destination VIDs are associated with the file data LIDs. Since the file data is already bound to the intermediate identifiers, the context-dependent format of the file data may not need to be updated.
[0138] The file system 906 may be further configured to use the storage layer 130 for checkpointing mmap operations. As used herein, a „An "mmap" operation refers to an operation in which the contents of files are accessed as memory pages using standard load and store operations rather than the standard read / write interfaces of the file system 906. An "msync" operation refers to an operation to flush the changed pages of the file (if any) to the storage medium 140. The use of mmap operations can make file checkpointing difficult. File operations are performed in memory, and an msync command is issued when the state needs to be saved. However, the state of the file after an msync represents the current state in memory, and the last saved state may be lost. Therefore, if the file system 906 were to crash during an msync operation, the file could be left in an inconsistent state.
[0139] In some embodiments, the file system 906 is configured to perform checkpointing of the state of a file on which an mmap operation was performed during calls with msync. Checkpointing the file may include creating a file-level snapshot (and / or a range clone), as disclosed above. The file-level snapshot may be configured to save the state of the file before the changes are applied. When the msync command is issued, another clone may be created to reflect the changes applied in the msync operation. As in Fig. As shown in Figure 9B, File 1 in state 913A (before the mmap operation) may be associated with LIDs 10-13 and corresponding memory addresses P1-P4 on storage medium 140. In response to the mmap operation, file system 906 may perform a range clone operation through interface 131 of storage layer 130, which may include creating a clone of File 1 (referred to as File 1.1). File 1.1 may be associated with a different set of LIDs 40-43 that reference the same file data (e.g., the same memory addresses P1-P4). In other embodiments, File 1 may be cloned using a reference map 460 and / or an intermediate translation layer, as disclosed above.
[0140] In response to an msync call, system 906 may perform another range clone operation (using storage layer 130). As illustrated in state 913C, the range clone operation associated with the msync operation may include updating File 1 with the contents of one or more modified pages (memory addresses P5 and P6) and cloning the updated File 1 as File 1.2. File 1.1 may reflect the state of the file prior to the msync operation. Accordingly, in the event of a failure, file system 906 may be able to reconstruct the previous state of File 1.
[0141] As disclosed above, the storage layer 130 may be configured to implement range cloning and range merging operations, which may be employed to implement higher-level operations such as file consistency (e.g., close-to-open file consistency, as disclosed in more detail herein), atomic operations, and the like. These operations may include: a) cloning a particular sub-region of the logical address space 132, b) performing memory operations in the cloned sub-region, and c) selectively merging and / or folding the cloned sub-region into another portion of the logical address space 132. As used herein, merging and / or folding sub-regions of the logical address space 132 refers to combining two or more LID ranges by, among other things, incorporating changes implemented in one of the ranges into one or more other ranges.A merge operation may be implemented according to a merge strategy that may be configured to resolve conflicts between different LID ranges. The merge strategy may include, but is not limited to, an "overwrite" mode in which the contents of one LID range "overwrite" the contents of another LID range; an "OR" mode in which the contents of the LID ranges are combined (e.g., in a logical OR operation); a copy-on-conflict mode in which conflicts are resolved by creating separate, independent copies of one or more LID ranges; and / or the like. In overwrite mode, the LID range that overwrites the contents of the one or more other LID ranges may be determined based on any suitable criteria, including delivery time (e.g.,newer operations overwrite earlier operations), priority and / or similar, but is not limited to this.
[0142] Fig. 9C shows embodiments of range merging operations implemented using the storage layer 130. In the embodiment of Fig. 9C, the storage layer 130 may be configured to clone the identifier range 914, which may be represented by one or more entries in the forward map 160. The LIDs 072-083 within the range 914 may be bound to memory addresses 95-106. The range cloning and / or merging operations disclosed herein may be performed using any of the range cloning and / or move embodiments disclosed herein. Fig. 3A-E, the reference entry embodiments of Fig. 4A-E and / or the intermediate imaging layer embodiments of Fig. 5A-B. Accordingly, in some embodiments, LIDs 072-083 may be bound to memory addresses 95-106 through one or more reference entries and / or intermediate mapping layers.
[0143] The storage layer 130 may be configured to clone the range 914, which, as illustrated at state 941A, may include binding a new range of LIDs 924 to memory addresses 95-106. The ranges 914 and / or 924 may include respective metadata 984 and / or 994 configured to indicate that the ranges 914 and 924 are related (e.g., bound to the same set of memory addresses). The metadata 984 and / or 994 may be configured to link the LIDs 972-983 to 972-983 such that modifications related to one of the LID ranges may be correlated with LIDs in the other range (e.g., data written in connection with LID 972 may be associated with the corresponding LID 972, and so on). Metadata 984 and / or 994 may indicate a synchronization strategy for the cloned LID ranges, which, as disclosed above, may indicate whether mapping operations should be synchronized between clones.Metadata 984 and / or 994 may further include and / or reference a merge strategy that may specify how merge conflicts are to be managed. The merge strategy may be specified through interface 131 of storage layer 130, may be determined based on a global and / or default merge strategy, may be specified via request parameters (e.g., fadvise, ioctrl, etc.), and / or the like. The clone operation may further include appending a persistent annotation 366 to storage medium 140 configured to associate the data at memory addresses 95-106 with the LID range 972-983 (and / or rewrite the data in an updated context-dependent format), as disclosed above.
[0144] The storage layer 130 may perform storage operations in one or more of the ranges 914 and / or 924 in response to store requests from one or more storage clients 106. As illustrated in state 914B, a store operation may modify data associated with the LIDs 972-973, which may include associating the identifiers 972-973 with a new set of memory addresses 721-722. Following the store operation(s) of state 941B, the storage layer 130 may perform a range merge operation to merge the LID range 972-983 with the range 721-783. The range merge operation may include incorporating the modifications made to the LID range 924 into the LID range 914 in accordance with a merge strategy. The merge strategy may specify that modifications made in the cloned area 924 overwrite data in the source area 914.Accordingly, the result of the merge operation illustrated in state 941C may include binding the LIDs 072-073 of the source range 914 to the modified data at memory addresses 721-722. The range merge operation may further include releasing the cloned LID range 972-983, storing a persistent annotation 366 configured to associate the data at memory addresses 756-757 with the LIDs 072-073, and / or rewriting the data at memory addresses 721-722 in an updated context-dependent format, as disclosed herein. Data stored at memory addresses 95-96 that was bypassed by the new data at 721-722 may be invalidated as disclosed above.
[0145] Store operations performed in ranges 914 and / or 924 may result in conflicts. In some embodiments, the merging strategy associated with the LID ranges may anticipate conflicts. As disclosed in further detail herein, in an atomic store operation, the storage layer 130 may lock one or more LID ranges while atomic store operations in one or more corresponding ranges are completed. However, in other implementations, the storage layer 130 may allow store operations to be performed concurrently within cloned ranges. In state 941D, the storage layer 130 may implement a store operation(s) configured to overwrite and / or modify data associated with the LIDs 972-973 and 982-983 in range 924. The storage layer 130 may perform a different store operation(s).implement other memory operations configured to overwrite and / or modify data associated with LIDs 072-073 of range 914. The memory operation(s) relating to LIDs 072-073 and 972-973 may create a merge conflict between ranges 914 and 924. The merge conflict may be resolved in accordance with a merge strategy as disclosed above. In some embodiments, the merge strategy may include applying the most recent modification based on, among other things, the relative order of the memory operations in the memory log. In other implementations, the merge strategy may resolve conflicts based on the relative priority of the memory clients 106 (processes, applications, and / or the like) that requested the respective memory operations.In another implementation, the merge strategy may resolve conflicts by creating two (or more) versions of areas 914 and / or 924 to represent the different conflicting versions.
[0146] State 941E illustrates one embodiment of a result of a merge operation configured to include the operation(s) associated with LIDs 072-073 rather than the conflicting modifications associated with LIDs 972-973. Therefore, in state 941E, LIDs 072-073 are bound to memory addresses 756-757 corresponding to the memory operation(s) performed with respect to LIDs 072-073, rather than to memory addresses 721-722 corresponding to the memory operation(s) performed with respect to LIDs 972-973.
[0147] State 941F illustrates one embodiment of a result of a merge operation configured to include the modifications of range 972-973 instead of the conflicting modifications made with respect to LIDs 072-073. Accordingly, in state 941F, identifiers 072-073 are bound to memory addresses 721-722 corresponding to the memory operation(s) performed with respect to LIDs 972-973, rather than to memory addresses 756-757 associated with LIDs 072-073.
[0148] State 941G illustrates one embodiment of a merge operation result configured to manage merge conflicts by creating separate region copies or versions. Region 914 may include the non-conflicting modifications made with respect to identifiers 982-983 and may retain the result of the conflicting store operations with respect to identifiers 072-073 (rather than including memory addresses 721-722). The other LID region 924 may retain the modifications of state 941D without including the results of the conflicting store operation(s) made with respect to identifiers 072-073.Although state 941G shows the copies using the original cloned LID ranges 072-083 914 and 974-981 924, the disclosure is not limited in this regard and could be configured to create the range copies and / or versions within any sub-region of the logical address space 132. The range merge operations disclosed with respect to states 941E-G may further include attaching one or more persistent annotations 366 to the storage medium 140 to associate the data stored at memory addresses 721-722, 756-757, and / or 767-768 with the corresponding LIDs and / or rewriting the data in one or more backend storage operations, as disclosed herein.
[0149] In some embodiments, operations within one or more of the cloned LID ranges 914 and / or 924 may include modifying the LID ranges 914 and / or 924 by, among other things, expanding the ranges 914 and / or 924, shrinking the ranges 914 and / or 924, or the like. Expanding one of the ranges 914 and / or 924 may include a corresponding expansion to the other range, and as such, mapping operations may be based on mapping additional LID(s) in both ranges 914 and 924.
[0150] The region merging operations disclosed herein may be performed using any of the region cloning and / or movement embodiments of Fig. 3A-E, the reference entry embodiments of Fig. 4A-E and / or the intermediate embodiments of Fig. 5A-B should be implemented. Fig. 9D illustrates one embodiment of a region merge operation using a reference map 460. As shown in state 943A, cloning region 914 may include allocating an LID region 924 in logical address space 132, linking regions 914 and 924 (using, among other things, metadata 984 and / or 994), and associating regions 914 and 924 with reference identifiers 934 in reference map 460. The region clone operation may further include storing a persistent annotation 366 on storage medium 140 configured to associate region 934 in reference map 460 with indirect regions 914 and / or 924, as disclosed above. Region 934 in reference map 460 may be bound to memory addresses 95-106. Accordingly, both regions 914 and 924 may indirectly reference the same data at the same memory addresses.
[0151] A memory operation in range 924 configured to modify data corresponding to LIDs 982-983 may include allocating new LIDs within range 924 and binding the new local entry 982-983 to the corresponding memory addresses 767-768, as shown in state 943B. Merging ranges 914 and 924 may include incorporating the modified data at memory addresses 767-768 into range 914 in accordance with a merge strategy, as disclosed above. In the embodiment of Fig. 9D, the range merge operation of state 943C may include removing reference entry 934 and updating LIDs 081-083 of range 914 to reference the updated data at memory addresses 767-768. The merge operation may further include storing a persistent annotation 366 and / or rewriting the data at memory addresses 767-768 in an updated context-dependent format, as disclosed above.
[0152] Fig. 9E shows further embodiments of range clone and range merge operations implemented by storage layer 130. Fig. Figure 9E illustrates range clone and range merge operations in embodiments that include an intermediate address space, as described in connection with the Fig. 5A-B. In state 947A, VIDs 072-083 comprising VID range 914 are bound to memory addresses 95-106 by intermediate identifiers 272Z-283Z in VAS forward map 560. The intermediate identifiers may be part of a separate intermediate address space 2136 (e.g., logical address space 132 of memory layer 130).
[0153] As illustrated in state 947B, cloning the VID range 914 may include allocating a new VID range 924 comprising VIDs 972-983 and associating the range 924 with the intermediate identifiers 272Z-283Z in the VAS forward map 560. The clone operation may further include storing a persistent note 366 on the storage medium 140 configured to associate the VID range 924 with the intermediate addresses 272Z-283Z. Storage operations may be performed with respect to the VID ranges 914 and / or 924 as disclosed herein. Modifications to VID ranges 914 and / or 924 may be reflected in updated mappings between the respective VID ranges 914 and / or 924 and the intermediate address space 2136. In state 947C, a memory operation that modifies data of VIDs 982-983 is reflected in updated mappings between VIDs 982-983 and intermediate identifiers 984Z-985Z, and memory addresses 456-457.Merging VID ranges 914 and 924 may include updating the VID mappings of range 914 to reference the updated data (through intermediate addresses 984Z-985Z), as illustrated in state 947D. The merge operation may further include resolving merge conflicts (if any), as disclosed above. The merge operation may further include appending one or more persistent annotations 366 to storage medium 140 to associate VIDs 082-083 with intermediate addresses 984Z-985Z.
[0154] In some embodiments, the storage layer 130 may employ the range clone, move, and / or merge operations disclosed herein to provide file consistency functionality to storage clients 106, such as file systems, databases, and / or the like. Fig. Referring to Figure 9F, a file system 906 may employ storage layer 130 to implement a close-to-open file consistency model following the Network File System (NFS) version 3 protocol and / or other file system implementations and / or protocols. The close-to-open file consistency model may be configured to allow multiple processes and / or applications (file system clients) to work on the same file concurrently. File modifications are committed at the time the file is closed; other clients working on the file in parallel will not see the changes until the file is next opened. Accordingly, the state of the file is set at the time the file is opened, and changes implemented in parallel by other clients will not be applied until the file is opened again.
[0155] In some embodiments, file system 906 may utilize storage layer 130 to maintain the file's "original" data (e.g., a consistent version of the file) while modifications are made in the cloned workspace. As used herein, maintaining the file's "original" data and / or a consistent version of the file refers to maintaining the file data in a state corresponding to the time the file was opened and / or maintaining a log of file modifications from which the state of the file data in its original, unmodified state can be reconstructed.
[0156] Fig. 9F shows an embodiment of a system 900F including storage layer 130 configured to implement a close-to-open file consistency model. File system 906 (and / or other storage clients 106) may employ storage layer 130 to efficiently implement close-to-open file consistency. Storage layer 130 may be configured to: a) clone files in response to file open requests from file system clients 926A-N, resulting in a "primary" or "consistent" version of the file and a "working" version of the file; b) perform storage operations with respect to the working version of the file; and c) merge the working version of the file into the primary version of the file in response to a file close. The storage layer 130 may be configured to clone the file data in one or more range clone operations as disclosed herein (e.g., using the range clone embodiments of Fig. 3A-E, Fig. 4A-E, Fig. 5A-B and / or the like). The storage layer 130 may be further configured to merge the working version of the file and the primary or consistent version of the file using one or more range merging and / or folding operations, as disclosed herein. The working version of the file may represent the state of the file at the time the file was opened by a particular storage client 926A-N. The storage client 926A-N may have exclusive access to the working version of the file, and as such, the working version of the file may be isolated from the file modifications made by other clients 926A-N.The storage layer 130 may be configured to maintain the original, unmodified file data with respect to the file's "primary" or "consistent" logical interface, which may include maintaining associations between the file data and the consistent logical interface while performing storage operations with respect to the file's working logical interface. Conflicts between file modifications made by different storage clients 926A-N may be resolved according to a conflict resolution or merge strategy, such as last write (e.g., the most recent write overwrites previous writes); copying on conflicts (e.g., creating separate versions of the file); priority based on the client 926A-N, application, process, and / or the like; and so on.
[0157] In the embodiment of Fig. 9F includes the translation module 134 at state 953A Fig. between the LIDs of a file (file LIDs 950A) and data of file 952A on the storage medium 140 at memory addresses P0-P3. Fig. may be generated using the forward mapping 160 disclosed herein and / or one or more intermediate mapping layers as described in connection with the Fig. 5A-B disclosed to be implemented.
[0158] In state 953B, storage layer 130 may be configured to clone the file in response to a file open request from a storage client (storage client 926B). The request may be obtained via interface 131 as an explicit request, a request parameter (e.g., fadvise, ioctrl, etc.), and / or the like. The clone operation may include one or more range clone operations, which, as disclosed herein, may include allocating a new set of "cloned" file LIDs 950B corresponding to the working version file and associating the set of cloned identifiers 950B with the same file data 952A as the LIDs 950A of the primary version of the file (the original or consistent set of logical identifiers 950A).The range clone operation may further include storing a persistent note 366 on the storage medium 140 to associate the file data 952A with both the LIDs of the primary file 950A and the LIDs of the working version of the file 950B, as disclosed above.
[0159] In some embodiments, storage layer 130 and / or file system 906 may be configured to direct file operations performed by storage client 926B to the working version of the file (the working set of LIDs 950B). Accordingly, modifications made by storage client 926B may be made with respect to the cloned file LIDs 950B. Such modifications may not affect the state of the original, primary version of the file LIDs 950A. Therefore, storage client 926B may modify the working version of the file with respect to LIDs 950B without changing the LIDs 950A of the original, primary version of the file.
[0160] In state 953C, the storage client 926B has performed a storage operation (through the storage layer 130) to modify data of the file stored at memory address P3; the modified data may be appended to the storage log at memory address P64. In response, the translation module 134 Fig. update to bind the LIDs of the cloned working version of the file 950B to the modified file data 952B at the memory address P64. Another LID(s) that was(were) not modified by the storage client 926B may still be bound to the original, unmodified file data 952A. The storage layer 130 is configured to Fig. between the identifiers 950A of the primary version of the file and the unmodified file data 952A at memory addresses P0-3.
[0161] Another storage client 926N may issue a request to open the file before storage client 926B has closed the file. In response, and as shown in state 953D, storage layer 130 may create another clone of the primary file (cloning the primary file identifiers 950A). The cloned LIDs (FIDs 950C) may correspond to the original state of the file without the modifications made by storage client 926B to the cloned identifier range 950B. Accordingly, the cloned LIDs 950C may be mapped 951C to the original, unmodified file data 952A at memory addresses P0-3. Storage client 926N may perform storage operations with respect to the new cloned file identifier range 950C in parallel with storage client 926B.Changes made by clients 926B and 926N may be isolated within their respective LID ranges 950B and 950C and as such may not be applied to the primary version of the file (LIDs 950A and / or each other).
[0162] State 953E illustrates the result of the storage client 926B closing the file. In response to a request from the storage client 926B to close the file, the storage layer 130 may be configured to merge the contents of the corresponding range (FIDs 950B) into the primary version of the file (LIDs 950A) in one or more range merge operations. However, the changes may not be merged into the version of the file in use by the storage client 926N (FIDs 950C); the storage client 926N may not have access to the modifications until the client 926N reopens the file. Incorporating the modifications may include one or more range merge operations, as disclosed herein. The range merge operations may be configured to merge the modifications made to the cloned LID range 950B into the LID range 950A of the primary version of the file.In the embodiment of . Fig. 9F, the range merge operation includes updating the Fig. the primary file LIDs 950A to reference the modified file data 952B at memory address P64. The data not modified by the client 924B may remain bound to the original, unmodified file data 952A at P0-3.
[0163] As disclosed herein, in some embodiments, the modified file data 952B may include persistent metadata configured to associate the modified file data 952B at memory address P64 with one or more of the LIDs 950B (rather than with the LIDs 950A associated with the primary version of the file). The range merge operation may therefore further include appending a persistent annotation 366 to the storage medium 140 configured to associate one or more of the ranges of LIDs 950A with the modified file data 952B at memory address P64. The data at memory address P64 may be rewritten with updated persistent metadata in one or more background operations. Following the file close operation (and corresponding range merge operations), the translation module 134 may be configured to release the LIDs of range 950B.
[0164] The client 926N may modify the file with respect to the cloned file identifiers 950C. As in state 953F of Fig. 9G, storage client 926N may perform one or more operations that contradict the modifications implemented by client 926B. The modifications may occur before client 950B has closed the file (before the modifications by client 926B have been applied to the LIDs 950A of the primary version of the file as in state 953E). As such, LIDs 950A are mapped 951A to the original, unmodified file data 952A, one or more of the range 950B identifiers associated with storage client 926B are mapped to the modified file data 952B, and one or more of the range 950C identifiers associated with storage client 926N are mapped to the conflicting file data 952C. The LIDs 950B and 950C, which correspond to unmodified data, may still reference the original, unmodified file data 952A.
[0165] Clients 926B and 926C may eventually close their respective files, which may include merging the modifications made to the respective LID ranges 950B and 950C into the range 950A of the primary version of the file. Storage layer 130 may be configured to resolve conflicts between ranges 950B and 950C according to a merge strategy 944. In some embodiments, merge strategy 944 may be based on the order in which storage clients 926B and 926C closed the files; the modifications of the most recently closed file may overwrite previously applied modifications (e.g., the modifications may be serialized). As illustrated in state 953G, storage client 950B may issue the file close request before storage client 950C.After the client 950B closes the file, the storage layer 130 may merge modifications made to the area 950B into the area 950A of the primary version of the file (as shown in state 953E of FIG. Fig. 9F). Closing the file by client 926C may result in overwriting some of the modifications made by storage client 950B (the modified data 952B) with data 952C, as illustrated in state 953G of Fig. 9G. The data at P3 and P64 may be marked for removal from storage medium 140 because it is no longer referenced by the primary file or a current working version of the file. As disclosed above, storage layer 130 may be configured to implement other merge strategies, such as a priority-based merge strategy 944. A priority-based merge strategy may resolve conflicts based on the relative priorities of storage clients 926B and / or 926C. In state 953H, storage client 926C may close the file after storage client 926B; however, the modifications of storage client 926B may be retained due to merge strategy 944 indicating that the modifications of storage client 926B have a higher priority than conflicting modifications of storage client 926C.Accordingly, the LIDs 950A of the primary version of the file may continue to reference the modified file data 952B of the storage client 926B, and the conflicting file data of the storage client 926C (data 952C at P96) may be marked for automatic garbage collection along with the outdated file data 952A at P3. In other embodiments, the merge strategy 944 may include a copy-on-conflict strategy resulting in the creation of two primary versions of the file. In such embodiments, and as illustrated in state 953I, the storage layer 130 may be configured to incorporate the modifications of the storage client 926B into the primary file (using the primary file LIDs 950A) and may incorporate the conflicting modifications of the storage client 926C into a new version of the file (file identifiers 950D).
[0166] Although particular embodiments of a merge strategy 944 are described herein, the disclosure is not limited in this regard and could implement and / or incorporate any suitable merge strategy 944. The merge strategy 944 may be implemented within the storage layer 130 and / or the file system 906. In some embodiments, the merge strategy 944 of the storage layer 130 and / or the file system 906 may be configured through the interface 131 of the storage layer 130. The merge strategy 944 may apply to all file operations performed by the storage layer 130. Alternatively or additionally, the merge strategy 944 may be determined on a per-file and / or per-conflict basis through, among other things, file system API calls, fadvise, ioctrl, and / or the like, as disclosed above.
[0167] The storage layer 130 may be further configured to implement efficient atomic storage operations. Fig. 10A is a block diagram of one embodiment of a system 1000A that includes a storage layer 130 configured to implement atomic storage operations. As used herein, an atomic storage operation refers to a storage operation that either completes completely as a whole or is rolled back. Accordingly, atomic storage operations cannot be partially completed; the storage layer 130 may be configured to invalidate and / or remove data from incomplete atomic storage operations. Implementing atomic storage operations, and particularly atomic storage operations that involve multiple steps and / or refer to multiple different identifier ranges or I / O vectors, can impose high overhead costs. For example, some database systems implement atomic storage operations using multiple sets of redundant write operations.
[0168] The storage layer 130 may include a transaction module 1036 configured to implement memory transactions. The transaction module 1036 may include an atomic memory module 1035 to employ range cloning, range relocation, and / or other operations disclosed herein to increase the efficiency of atomic memory operations. In some embodiments, the interface 131 provides APIs and / or interfaces for performing vector atomic memory operations. A vector may be defined as a data structure, such as: struct iovect { uint64 iov_base; / / Base address of a memory subregion for input or output uint32 iov_len; / / Size of the memory referenced by iov_base uint64 dest_lid; / / Logical identifier of the destination}
[0169] The parameter iov_base can reference a memory or buffer location containing data of the vector, iov_len can refer to a length or size of the data to the logical destination identifier(s) for the vector (e.g., the logical base identifier, with the length of the range implied by and / or derived from the input buffer iov_len).
[0170] A vector memory request to write data to one or more vectors can therefore be defined as follows: vector write ( int fileids, const struct iovect *iov, uint32 iov_cnt, uint32 flag)
[0171] The above vector write operation can be configured to collect data from each of the vector data structures referenced by the *iov pointer and / or specified by the vector count parameter (iov_cnt) and write the data to the logical destination identifier(s) specified in the respective iovect structures (e.g., dest_lid). The flag parameter can specify whether the vector write operation should be implemented as an atomic vector operation.
[0172] As illustrated above, a vector memory request may involve performing the same operation on each of a plurality of vectors (e.g., implicitly performing a write operation that refers to one or more different vectors). In some embodiments, a vector memory request may specify different I / O operations for each sub-vector. Accordingly, each iovect data structure may include a respective operation indicator. In some embodiments, the iovect structure may be extended as follows: struct iovect { uint64 iov_base; / / Base address of a memory sub-region for input or output uint32 iov_len; / / Size of memory referenced by iov_base uint32 iov_flag; / / Vector operation flag uint64 dest_lid; / / Logical identifier of the destination}
[0173] The iov_flag parameter can specify the memory operation to be performed on the vector. The iov_flag can specify any suitable memory operation, including, but not limited to, a write operation, a read operation, an atomic write operation, a trim or discard request, an erase request, a format request, a pattern write request (e.g., a request to write a specified pattern), a null write request, an atomic write operation with a verification request, an allocation request, or the like. The vector memory request interface described above can be extended to accept vector structures: vector_request( int fileids, const struct iovect *iov, uint32 iov_cnt, uint32 flag)
[0174] The flag parameter can specify whether the vector operations of vector_request should be performed atomically. Further embodiments of atomic memory operations are disclosed in U.S. patent application Ser. No. 13 / 725,728, entitled "Systems, Methods, and Interfaces for Vector Input / Output Operations," filed December 21, 2012, to Ashish Batwara et al., which is incorporated herein by reference.
[0175] The transaction module 1036 may include an atomic storage module 1035 configured to implement atomic storage operations within the storage layer 130. The atomic storage module 1035 may be configured to implement storage operations of an atomic storage request with respect to a different set of identifiers than the target or destination identifiers of the request. Upon completion of the atomic storage operations, the atomic storage module 1035 may be configured to move the data to the respective target or destination identifier(s) of the atomic storage request, as disclosed herein.
[0176] In some embodiments, the atomic storage module 1035 implements atomic storage operations directed to a first set of identifiers with respect to a second, different set of identifiers. The second set of identifiers may be considered ephemeral, temporary, working, or in-process identifiers. The second set of identifiers may not be directly accessible to the storage clients 106. The second set of identifiers may correspond to a particular subset of the logical address space 132, a particular virtual address space (e.g., a VAS 532), a separate namespace, and / or the like. Upon completion of the storage operations of the atomic storage request, the atomic storage module 1035 may implement a range move operation configured to associate data of the atomic storage request with the first set of identifiers.The data may be separated from the second set of identifiers. As above, the second set of identifiers may be distinguishable from LIDs of the logical address space 132 and / or VIDs of a VAS 532. In the event of an error condition, the reconstruction module 1037 may identify data bound to such identifiers related to failed transactions (e.g., incomplete atomic memory operations). The identified data may be invalidated during metadata reconstruction operations and / or corresponding entries may be omitted from the memory metadata 135 (e.g., omitted from the forward map 160, the VAS forward map 560, the reference map 460, and / or the like).
[0177] In some embodiments, the atomic memory module 1035 implements atomic memory operations in a separate address space, such as the transaction address space 1032 of Fig. 10A. Although Fig. 10A describes the use of a transactional address space 1032, the disclosure is not limited in this regard and could be adapted to use any suitable address space and / or namespace, including, but not limited to, a portion of the logical address space 132 (e.g., a range, an extension, and / or a set of LIDs), a portion of a VAS 532, a reference map 460, an intermediate address space, and / or the like. The identifiers of the transactional address space 1032 (transaction identifiers) may not be directly accessible to the storage clients 106.
[0178] The atomic storage module 1035 may perform atomic storage operations with respect to the transactional address space 1032 and, upon completion of the atomic storage operations, may perform an atomic range move operation configured to move data of the atomic storage operations from the transactional address space 1032 to the logical address space 132 (or to another destination or target namespace, such as a particular VAS 532). The atomic range move operation may include updating bindings in the forward map 160, writing metadata to the storage medium 140 (e.g., appending a persistent annotation 366 to the log), and / or the like, as disclosed herein.
[0179] In the embodiment of Fig. 10A, a memory client 106 issues an atomic memory request that refers to vectors 1040A and 1040B in logical address space 132. As in Fig. As illustrated in Figure 10A, vectors 1040A and 1040B may correspond to existing entries within forward map 160. Before the atomic store operation is implemented (at state 1015A), LIDs 10-13 of vector 1040A may be bound to memory addresses P1-P4, and LIDs 36-38 of vector 1040B may be bound to memory addresses P6-8. In other embodiments, the atomic store request may refer to LIDs that are unassigned and / or not yet bound to memory addresses and, as such, do not have corresponding entries within forward map 160 (and / or other mapping layers).
[0180] In response to the atomic memory request, the atomic memory module 1035 may access a second set of identifiers within the transactional address space 1032, including through use of a redirection module 1034. The redirection module 1034 may be configured to map the second set of identifiers within the transactional address space 1032. The transaction identifiers may be used to implement portions of the atomic memory request (e.g., to track in-process portions of the atomic memory operations). The redirection module 1034 may be further configured to link the second set of identifiers to the first set of LIDs (e.g., the target LIDs of the atomic memory request) through use of, among other things, the memory metadata 135, entries in the forward index 160, and / or other index metadata, and / or the like.In some embodiments, the atomic memory module 1035 may be further configured to perform range clone operation(s) configured to bind the second set of identifiers to the same memory addresses as the first set of identifiers (vectors 1040A and 1040B), as disclosed herein.
[0181] As illustrated in state 1015B, the redirection module 1034 may allocate a second set of identifiers comprising vectors 1042A and 1042B containing transaction identifiers Z0-3 and Z6-8. Bindings between transaction identifiers and storage locations may be maintained in the storage metadata, among other things, by using an intermediate mapping layer such as the transaction map 1060. The transaction map 1060 may include mappings between transaction identifiers and LIDs of the logical address space 132 (and / or VIDs of a VAS 532). In the embodiment of Fig. 10A, the transaction map 1060 includes links 1064A between the transaction identifiers of vector 1042A and corresponding LIDs of vector 1040A (LIDs 10-13). The transaction map 1060 further includes links 1064B between the transaction identifiers of vector 1042B and LIDs 36-38 of vector 1040B. The transaction map 1060 may further include bindings between transaction identifiers and storage locations. State 1015B shows range clone operation(s) in transaction map 1060 that includes bindings 1062A between transaction identifiers 1042A and LID locations P1-4 in vector 1040A and bindings 1062B between transaction identifiers 1042B and LID locations P6-8 in vector 1040B.
[0182] The atomic memory module 1035 may implement atomic memory operations of the atomic memory request within the transaction address space 1032, which may include redirecting memory operations from the first set of LIDs (the vectors 1040A and / or 1040B) to the second set of identifiers (the transaction identifiers of 1042A and 1042B). Redirecting the memory operations may include translating references to LIDs in the atomic memory operation to the second set of transaction identifiers by using, among other things, the transaction map 1060. For example, a memory operation referring to LID 10 may be translated to the transaction identifier Z0 based on the Fig. the transaction map 1060. Memory operations configured to allocate logical capacity may be redirected to (and maintained therein) the transaction address space 1032. For example, a request to extend vector 1040A to include LIDs 14-20 may include: a) allocating the LIDs in the logical address space 1032, b) allocating corresponding transaction identifiers in the transaction address space 1032, and c) concatenating the allocated transaction identifiers and LIDs in the transaction map 1060. A request to TRIM LIDs may include marking the corresponding identifiers in the transaction map 1060 as invalid. The corresponding LIDs in the logical address space 132 may be TRIMmed in response to the range shift operation performed after the completion of the atomic memory operations, as disclosed in more detail herein.
[0183] As illustrated in state 1015C, the storage operations of the atomic store request may include appending data to the storage medium 140 at storage locations P9-13 and P100-102. The corresponding storage operations may be redirected to the transaction address space 1032, as disclosed herein. Accordingly, the data of the atomic store request may be associated with the transaction identifiers Z0-3 and Z6-8, which may include: a) binding the storage locations P9-13 and P100-102 to the transaction identifiers Z0-3 and Z6-8 in the transaction map 1060; and b) storing the data at P9-13 and P100-102 with persistent metadata 114 configured to associate the data with the respective transaction identifiers Z0-3 and Z6-8.
[0184] Other memory operations may be performed concurrently with and / or interleaved within the atomic vector operations. Accordingly, the data of the atomic memory request need not be stored at contiguous memory locations in the memory address space 144 of the storage medium 140. Data of the atomic memory request may be distinguished from other data not associated with the atomic memory request based on, among other things, the bindings between the data at memory locations P9-10 and P100-102 and the transaction identifiers Z0-3 and Z6-8.
[0185] As further illustrated at state 1015C, the original, unmodified state of the LIDs in vectors 1040A and 1040B may be unchanged while the atomic storage operation(s) are in progress; the data at storage locations P1-4 and P6-8 may remain on storage medium 140, and the bindings between the original, unmodified data and LIDs 10-13 and 36-38 may be maintained in forward index 160. Therefore, the original, unmodified data corresponding to vectors 1040A and 1040B may be maintained in a consistent state while the atomic storage operation(s) are being performed and may be preserved regardless of errors in the atomic storage operation(s).
[0186] Completion of the atomic store operation may include merging the contents of the transactional address space 1032 into the logical address space 132. As illustrated in state 1015D, after completing the atomic store operations in the transactional address space (as shown in state 1015C), the atomic store module 1035 may perform a range shift operation to modify the logical interface of the data at P9-13 and P100-102 to bind the data to the first set of LIDs (the destination LIDs of vectors 1040A and 1040B). The range shift operation may include updating the forward map 160 to associate LIDs 10-13 of vector 1040A with memory locations P9-13 and to associate LIDs 36-38 of vector 1040B with memory locations P100-102.The range move operation may further include storing a persistent annotation 366 on storage medium 140 to bind memory addresses P9-P13 to LIDs 10-13 and P100-102 to LIDs 36-38, as disclosed herein. The range move operation may be implemented in other ways, including the reference entry embodiments of FIG. Fig. 4A-E and / or the intermediate embodiments of Fig. 5A-B, but not limited thereto. The range move operation may also include rewriting the data in a context-dependent format consistent with the updated logical interface, which may include rewriting the data with persistent metadata 114 configured to associate the data with the first set of LIDs (e.g., LIDs 10-13 and / or 36-38 of the logical address space 132, respectively). The data may be rewritten in one or more background operations, as disclosed herein.
[0187] The atomic storage module 1035 may be configured to acknowledge completion of the atomic storage request in response to the completion of the range move operation. Accordingly, the completion may be acknowledged in response to storing the persistent note 366 and / or placing the persistent note 366 in a write buffer (and / or a power-failure domain of the storage device 141). Completion of the atomic storage request may further include releasing the transaction identifiers used to implement the atomic storage request.
[0188] In some embodiments, the range shift operation may further comprise modifying LID mappings within the logical address space 132. As disclosed above, an atomic memory operation may comprise a request to allocate LIDs within the logical address space 132. Implementing such an operation may comprise: a) allocating the requested LIDs (or alternative LIDs) within the logical address space 132, b) allocating corresponding transaction identifiers in the transaction address space 132, and c) linking the LIDs to the transaction identifiers, as disclosed herein. The range shift operation may comprise removing the transaction identifiers and / or moving data associated with the corresponding transaction identifiers to the allocated LIDs, as disclosed herein.Implementing an operation to TRIM one or more LIDs may include marking the corresponding transaction identifier(s) in the transaction map 1060 as invalid. The corresponding scope shift operation may include TRIMming the LIDs mapped to the transaction identifiers by, among other things, removing the LIDs from the forward index 160 and / or storing persistent metadata indicating that the LIDs have been removed. In the embodiment of FIG. Fig. 10A, an atomic storage request including a request to TRIM LID 12 may include invalidating the transaction identifier Z2 in the transaction map 1060. Invalidating the transaction identifier Z2 may include maintaining an entry in the transaction map 1060 representing the transaction identifier Z2 and marking the entry as invalid, deleted, TRIMmed, or the like. Completing the atomic storage request may include: a) invalidating the LID mapped to the transaction identifier Z2 (e.g., invalidating the entry in the forward map 160 corresponding to LID 12 and / or other storage metadata 135); and / or b) configuring the persistent annotation 366 to TRIM the LID 12 (e.g., indicate that the data at the storage location(s) bound to the LID 12 does not need to be retained on the storage medium 140).
[0189] In some embodiments, the storage layer 130 may further include a reconstruction module 1037 configured to reconstruct the storage metadata 135 from the contents of the storage log. The reconstruction module 1037 may reconstruct the storage metadata 135 in response to a failure condition resulting in loss and / or corruption of the storage metadata 135. The reconstruction module 1037 may be configured to traverse physical locations in the storage address space 144 in a log order (e.g., from newest to oldest or vice versa). The reconstruction module 1037 may access the persistent metadata 114 of the storage log to, among other things, reconstruct the LID for storage address associations of the forward map 160.
[0190] In response to a failure of the atomic memory operation of Fig. 10A, the reconstruction module 1037 may be configured to reconstruct the vectors 1040A and 1040B based on the contents of P1-4 and P6-8, respectively. The reconstruction module 1037 may determine, based on the association of the data with the identifiers Z0-3 and Z6-9 of the in-process address space 1032, that the data stored at P9-13 and / or P100-102 relates to an incomplete memory request. The reconstruction module 1037 may therefore omit the entries from the forward index 160 and invalidate data at the corresponding memory locations. Alternatively, the reconstruction module 1037 may reconstruct corresponding entries in the transaction map 1060 such that the partially atomic memory request can be completed (e.g., continue from the point of failure rather than starting the atomic memory operation from the beginning).
[0191] The reconstruction module 1037 may be further configured to identify data related to a completed atomic memory request. After a successful completion of the atomic memory request from Fig. 10A, when reconstructing the storage metadata 135 based on the persistent annotation 366 stored at the storage location 109, the reconstruction module 1037 may determine that the physical storage locations P9-13 and P100-102 correspond to LIDs of the logical address space 132 (and are the result of a successful atomic storage request). As disclosed above, the persistent annotation 366 may include persistent metadata configured to associate the data at P9-13 and P100-102 with the LIDs of the vectors 1040A and 1040B. Accordingly, the reconstruction module 1037 may be further configured to reconstruct the associations of the state 1015C using the persistent annotation 366, regardless of whether the corresponding data has been rewritten in an updated context-dependent format.
[0192] Fig. 10B shows further embodiments 1000B of atomic storage operations implemented using, among others, the storage layer 130 and aggregation layer 530 disclosed herein. At state 1017A, an atomic storage request may be obtained that relates to a first set of VIDs. As illustrated in state 1017A, the target VIDs of vector 4096-4159 may not correspond to any existing VID mapping(s) and / or data. Accordingly, the VAS forward map 560 may not include entries corresponding to the VIDs 4096-4159 and / or entries in the intermediate map 1070.
[0193] As illustrated in state 1017B, in response to the atomic storage request, the atomic storage module 1035 may be configured to allocate VIDs corresponding to the atomic storage request (entries 4096, 64). If the requested entries are not available (e.g., already allocated to another client and / or existing data), the atomic storage request may fail. Alternatively, the atomic storage module 1035 may allocate a different set of VIDs to implement the atomic storage request, which may be returned to the storage client 106 upon completion of the atomic storage request.
[0194] The redirection module 1034 may be configured to assign temporary in-process identifiers 9872Z,64 corresponding to the VIDs 4096,64. As with the transaction identifiers disclosed herein, the in-process identifiers 9872Z,64 may correspond to a different namespace than the target VIDs (e.g., a transaction address space 1032), a particular subset of the VAS 560, a separate VAS 560, and / or the like. The redirection module 1034 may be configured to link the VIDs 4096,64 to the in-process identifiers 9872Z,64 by using, among other things, metadata from respective entries within the VAS forward map 560 (and / or other storage metadata 135) and / or by using a transaction map 1060 (or an in-process index). Binding the transaction identifiers 9872Z,64 to the intermediate identifiers 1032,64 may include attaching a persistent annotation 1066A to the storage medium 140.The persistent annotation 1066A may include persistent metadata configured to associate the in-process identifiers with the intermediate identifiers 1032,64.
[0195] In state 1017C, the atomic storage module 1035 may be configured to implement the storage operations corresponding to the atomic storage request with respect to the in-process identifiers, which may include redirecting (and / or translating) VIDs of the atomic storage request to corresponding in-process identifiers. The storage operations may include appending data to the storage medium 140 using, among other things, the log storage module 137. The in-process identifiers 9872Z,64 may be bound to the attached data by the intermediate mapping layer 1070 (identifiers 1032,64) and / or the persistent annotation 1066A. The data attached to the storage log may include persistent metadata 114 configured to bind the data of the atomic storage operation to respective intermediate identifiers 1032,64.The persistent metadata 114 may further include a flag or other indicator configured to identify the data as part of an atomic storage request.
[0196] Completing the atomic memory request may include a range move operation configured to modify the logical interface of the attached data to bind the data to the target VIDs of the atomic memory request (VIDs 4096,64). Upon completion of the memory operations of the atomic memory request, the atomic memory module 1035 may implement the range move operation by, among other things: a) updating the VAS forward map 560 to associate the target VIDs 4096,64 with the intermediate entry 1032,64; and b) removing the mapping between the in-process identifiers 9872Z,64 and the intermediate entry 1032,64.
[0197] The range move operation may further include releasing the in-process identifiers 9872Z,64 (e.g., removing the entries and / or corresponding metadata from an in-process index 1060, as disclosed above). As illustrated in state 1017D, the range move operation may further include attaching another persistent note 1066B to the storage medium 140 configured to identify the modified logical interface of the attached data. The persistent note 1066B may bind the intermediate entry 1032,64 to the target VID range 4096,64. The persistent note 1066B may further indicate that the atomic storage operation completed successfully.The atomic storage module 1035 may be configured to acknowledge completion of the atomic storage request in response to storing the persistent note 1066B on the storage medium 140 (and / or scheduling the persistent note 1066B for storage in, e.g., a power-resilient domain).
[0198] As disclosed above, the range relocation operations implemented by atomic memory module 1035 may include modifications to the target namespace (e.g., logical address space 132). In one embodiment, an atomic memory request may include, for example, a request to TRIM an I / O vector. Implementing the atomic memory request may include storing a persistent annotation 366 within the memory log configured to TRIM a corresponding in-process or transaction identifier(s), as disclosed above. The range relocation operation may include implementing the TRIM operation within the target namespace. The TRIM operation may include configuring the persistent annotation 1066B to TRIM the target I / O vector and / or moving the stored TRIM instruction to the target I / O vector, as disclosed herein.
[0199] The persistent annotation 1066B may be used by the reconstruction module 1037 to restore the memory metadata 135 in the event of an error condition, as disclosed above. The reconstruction module 1037 may restore the bindings of the intermediate map 1070 and / or the VAS forward map 560. In state 1017D, the reconstruction module 1037 may reconstruct the associations between 1032,64 and the corresponding memory addresses by using the persistent metadata 114 stored with the corresponding data segments within the protocol (e.g., within respective packet headers). The reconstruction module 1037 may be further configured to associate the intermediate addresses 1032,64 with the VIDs 4096,64 by using the persistent annotation 1066B.
[0200] As illustrated in state 1017E, the reconstruction module 1037 may identify data of a failed atomic store operation in response to determining that no persistent indicia 1066B exists on the storage medium 140 indicating completion of the atomic store request. The reconstruction module 1037 may determine that the appended data is part of an incomplete, failed atomic store request in response to identifying data bound to the intermediate identifiers 1032,64 that are a) bound to in-process identifiers 9872Z,64 and / or b) marked as atomic (in the respective persistent metadata 114).In response, the reconstruction module 1037 may: a) remove and / or omit the entry 1032,64 from the intermediate map 1070, b) remove and / or omit the entries 4096,64 and / or 9872Z,64 from the VAS forward map 560, and / or c) invalidate the corresponding data on the storage medium 140.
[0201] Fig. 11 is a flow diagram of one embodiment of a method 1100 for managing a logical interface of data stored in a context-dependent format on a non-transitory storage medium.
[0202] Step 1120 may include modifying a logical interface of data stored in a context-dependent format on a non-transitory storage medium. The logical interface may be modified at step 1120 in response to performing an operation on the data, which may include, but is not limited to, a clone operation, a deduplication operation, a move operation, or the like. The request may originate from a storage client 106, the storage layer 130 (e.g., the deduplication module 374), or the like.
[0203] Modifying the logical interface may include modifying the LID(s) associated with the data, which may include, but is not limited to, referencing the data using one or more additional LIDs (e.g., cloning, deduplicating, etc.), changing the LID(s) associated with the data (e.g., moving), or the like. The modified logical interface may be inconsistent with the context-dependent format of the data on storage medium 140, as described above.
[0204] Step 1120 may further include storing a persistent annotation on the storage medium 140 that identifies the modification to the logical interface. The persistent annotation may be used to make the logical operation persistent and crash-safe so that the modified logical interface (e.g., the storage metadata 135) of the data can be reconstructed from the contents of the storage medium 140 (if necessary). Step 1120 may further include acknowledging that the logical interface has been modified (e.g., returning from an API call, returning an explicit acknowledgment, or the like). The acknowledgment (and access by the modified logical interface at step 1130) occurs before the context-dependent format of the data on the storage medium 140 is updated.Accordingly, the logical operation may not wait until the data is rewritten and / or relocated; as disclosed herein, updating the context-dependent format of the data may be delayed and / or implemented in a process that is outside the "critical path" of method 1100 and / or the path for servicing other storage operations and / or requests.
[0205] Step 1130 may include providing access to the data in the inconsistent context-dependent format through the modified logical interface of step 1120. As described above, updating the context-dependent format of the data to be consistent with the modified context-dependent interface may involve rewriting and / or relocating the data on the non-volatile storage media, which may impose additional latency on the operation of step 1120 and / or other storage operations related to the modified logical interface. Therefore, the storage layer 130 may be configured to provide access to the data in the inconsistent context-dependent format while (or before) updating the context-dependent format of the data.Providing the data at step 1130 may include referencing and / or linking one or more reference entries corresponding to the data (via one or more indirect entries), as described above.
[0206] Step 1140 may include updating the context-dependent format of the data on storage medium 140 to be consistent with the modified logical interface of step 1120. Step 1140 may include rewriting and / or relocating the data to another media storage location on storage medium 140. As described above, step 1140 may be implemented using an operation that is outside the critical path of step 1120 and / or other storage requests performed by storage layer 130; step 1140 may be implemented by another, autonomous module, such as media management module 370, deduplication module 374, or the like. Accordingly, the context-dependent format of the data may be updated independently of servicing other storage operations and / or requests.As such, step 1140 may include delaying an immediate update of the context-dependent format of the data and updating the context-dependent format of the data in one or more "background" operations, such as a media management operation. Alternatively or additionally, updating the context-dependent format of the data may occur in response to (e.g., in conjunction with) other storage operations. A subsequent request to modify the data may, for example, cause the data to be rewritten elsewhere and in the updated context-dependent format.
[0207] Step 1140 may further include updating the storage metadata 135 when the context-dependent format of the data is updated. As the data is rewritten and / or relocated in the updated context-dependent format, the storage layer 130 may update the storage metadata 135 (e.g., the forward map 160) accordingly. The updates may include removing one or more links to reference entries in a reference map 460 and / or replacing indirect entries with local entries, as described above. Step 1140 may further include invalidating and / or removing a persistent annotation from the storage medium 140 in response to updating the context-dependent format of the data and / or persistently storing the storage metadata 135, as disclosed above.
[0208] Fig. 12 is a flowchart of another embodiment of a method 1200 for managing a logical interface of data stored in a context-dependent format on a non-transitory storage medium. The method 1200 may be implemented by one or more modules and / or components of the storage layer 130, as disclosed herein.
[0209] Step 1220 includes selecting a storage section for recovery, such as an erase block or a logical erase block. As described above, the selection of step 1220 may be based on a number of different factors, such as a lack of available storage capacity, detecting that a percentage of data marked as invalid in a particular logical erase block reaches a threshold, consolidation of valid data, error detection rate reaching a threshold, improving data distribution, data refresh, or the like. Alternatively or additionally, the selection criteria of step 1220 may include whether the storage section comprises data in a context-dependent format that is inconsistent with a corresponding logical interface thereof, as described above.
[0210] As disclosed above, restoring (or reclaiming) a storage section may include erasing the storage section and relocating valid data therein (if any) to other locations on the non-volatile storage media. Step 1230 may include determining whether the context-dependent format of the data should be updated in a cleanup operation (e.g., is inconsistent with the logical interface of the data). Step 1230 may include accessing storage metadata 135, such as the forward map 160, the reference map 460, and / or the intermediate address space, as described above, to determine whether the persistent metadata (e.g., the logical interface metadata) of the data is consistent with the storage metadata 135 of the data. If the persistent metadata is inconsistent with the storage metadata 135 (e.g.,associate the data with different LIDs, as described above), flow continues at step 1240; otherwise, flow continues at step 1250.
[0211] Step 1240 may include updating the context-dependent format of the data to be consistent with the logical interface of the data. Step 1240 may include modifying the logical interface metadata to reference a different set of LIDs (and / or reference entries), as described above.
[0212] Step 1250 includes relocating the data to another storage location in a log format that preserves an ordered sequence of storage operations performed on the non-volatile storage media, as described above. Accordingly, the relocated data (in the updated context-dependent format) can be identified as the valid and current version of the data in reconstructing the storage metadata 135 (if necessary). Step 1250 may further include updating the storage metadata 135 to bind the logical interface of the data to the new media storage locations of the data, remove indirect and / or reference entries to the data in the inconsistent context-dependent format, and so forth, as disclosed herein.
[0213] Fig. 13 is a flow diagram of another embodiment of a method 1300 for managing logical interfaces of data stored in a context-dependent format. Step 1315 may include identifying duplicate data on one or more storage devices 120. Step 1315 may be performed by a deduplication module 374 operating within storage layer 130. Alternatively, step 1320 may be performed by storage layer 130 when performing storage operations.
[0214] Step 1315 may include determining and / or verifying that the storage medium 140 includes duplicate data (or already includes data from a write and / or modification request). Accordingly, step 1320 may occur within the path of a storage operation (e.g., when or before duplicate data is written to the storage medium 140) and / or may occur outside the path of servicing storage operations (e.g., identifying duplicate data already stored on the storage medium 140). Step 1320 may include creating and / or maintaining data signatures in the storage metadata 135 and using the signatures to identify duplicate data.
[0215] In response to identifying the duplicate data at step 1315, the storage layer 130 (or another module, such as the deduplication module 374) may modify a logical interface of a copy of the data so that a single copy can be referenced by two (or more) sets of LIDs. The modification to the logical interface at step 1320 may include updating the storage metadata 135 and / or storing a persistent note on the non-volatile storage media 135, as described above. Step 1320 may further include invalidating and / or removing other copies of the data on the non-volatile storage media, as described above.
[0216] The context-dependent format of the data on storage medium 140 may be inconsistent with the modified logical interface. Therefore, steps 1330 and 1340 may include providing access to the data in the inconsistent context-dependent format through the modified logical interface and updating the context-dependent format of the data on storage medium 140, as described above.
[0217] Fig. 14 is a flow diagram of one embodiment of a range merge operation implemented by the storage layer 130 disclosed herein. Step 1410 may include cloning a set of LIDs in a logical address space 132. Cloning the LIDs may include referencing the same set of data on the storage medium 140 (e.g., the same storage locations and / or memory addresses) by two or more different sets of LIDs. The two or more sets may include a working set of LIDs and an original consistency set of LIDs. The working set of LIDs may be used to perform file modification operations, and the original consistency set of LIDs may be configured to maintain an original, unmodified state of the data.
[0218] As disclosed above, the data cloned at step 1410 may be referenced by a set of LIDs that may be tied to storage locations of the data on the storage medium 140. Step 1410 may include allocating one or more sets of LIDs within the logical address space 132 and / or within a separate address space. The one or more other sets of LIDs may include a logical capacity equivalent to the logical capacity of the original set of LIDs (e.g., contain the same number of LIDs and / or correspond to the same amount of storage capacity). Step 1410 may further include associating and / or binding the logical identifiers of the one or more other sets of LIDs with the same data referenced by the original set of LIDs.Accordingly, step 1410 may include modifying the logical interface to the data to associate the data with two or more different sets of LIDs. In some embodiments, step 1410 includes allocating one or more sets of LIDs within the logical address space 132 and binding the LIDs to the same set of memory addresses. Alternatively or additionally, step 1410 may include creating one or more reference entries within a reference map 460 to indirectly link the LIDs of the two or more different sets of LIDs to the memory addresses through one or more reference entries, as described in connection with FIG. Fig. 4A-E. Alternatively, step 1410 may be implemented using one or more intermediate mapping layers (e.g., as described in connection with Fig. 5AB). Step 1410 may further include linking the two or more sets of LIDs by, among other things, metadata 984 and / or 994 associated with the LIDs. The metadata 984 and / or 994 may be configured to indicate that the LID sets represent clones of the same storage unit (e.g., versions of the same file). The metadata 984 and / or 994 may be further configured to specify and / or reference a merging strategy for the two or more sets of LIDs, as disclosed above.
[0219] Step 1410 may further include storing a persistent annotation 366 on the storage medium 140 configured to make the clone operation of step 1410 persistent and crash-safe. The persistent annotation 366 may be configured to indicate the modified logical interface of the data (e.g., to associate the data with the two or more sets of LIDs), indicate a merge strategy of the clone operation, and the like.
[0220] Step 1420 may include performing storage operations within one or more of various LID ranges of step 1410. The storage operations may be performed in response to requests received through interface 131 from one or more storage clients 106. The storage operations may include appending data to storage medium 140. The storage operations may therefore include modifying the associations and / or bindings between LIDs in one or more of the LID sets and storage locations on storage medium 140. Modifying the associations and / or bindings may further include directly mapping the LIDs in one or more of the LID sets to the appended data, directly and / or through one or more indirect references and / or mapping layers.
[0221] Step 1430 may include merging the LID sets, as disclosed above. Merging the LID sets may include incorporating modifications made in one of the LID regions into one or more of the LID sets, as disclosed above. Step 1430 may further include resolving one or more merge conflicts in accordance with a merge strategy. In some embodiments, merging includes deleting (e.g., invalidating) one or more of the LID sets, which may include removing entries from the forward map 160, removing shared references to storage locations from a reference count data structure, removing reference entries from a reference map 460, removing references in an intermediate mapping layer, and / or the like. Step 1430 may further include modifying a logical interface of the merged data, as disclosed above.The modified logical interface may update the LIDs used to reference the data originally stored with respect to one or more of the LID sets. The modified logical interface may be inconsistent with the context-dependent format of the data on the storage medium 140. Therefore, step 1430 may include appending one or more persistent annotations 366 on the storage medium 140 to associate merged data with an updated logical interface of the data (e.g., associating data originally stored in association with LIDs in the second set with LIDs in the first set). Step 1430 may further include providing access to the data in the inconsistent context-dependent format and / or updating the context-dependent format of the data in one or more background operations, as disclosed above.
[0222] Fig. 15 is a flow diagram of another embodiment of a method 1500 for range merge operations. Step 1520 may include receiving a request to create a logical copy of an LID range. The request may be received from a storage client 106 via an interface 131 and / or may be part of a higher-level API provided by the storage layer 130. The request may include an "operational mode" of the clone, which may include, but is not limited to, how the clones should be synchronized, if at all; how a merge should proceed (merge strategy); whether the logical copy should be designated as ephemeral; and so on.
[0223] Step 1530 may include allocating LIDs in the logical address space 132 to service the request. The allocation of step 1530 may further include reserving physical memory space to accommodate changes to the cloned LID range. The reservation of physical memory space may be based on the operating mode of the clone. For example, if all changes are to be synchronized between the clone and the original address space, a small portion (if any) of physical memory space may be reserved. Alternatively, the storage layer 130 may reserve additional physical memory capacity for logical copy operations that have a copy-on-conflict merge strategy. Step 1530 may further include allocating the clone in a designated portion or segment of the logical address space 132 (e.g., a region dedicated for use with logical copy and / or clone operations).Accordingly, step 1530 may include assigning a second, different set of LIDs to clone a first set of LIDs.
[0224] Step 1540 may include updating the logical interface of data corresponding to the clone to reference both the original LIDs bound to the data and the cloned LIDs assigned at step 1530. Step 1540 may include storing a persistent note 366 on the storage medium 140, as disclosed above.
[0225] Step 1550 includes receiving a memory request and determining whether the memory request refers to an LID in the first and / or second sets (in the cloned LID range). If so, flow continues at step 1560; otherwise, flow remains at step 1550.
[0226] Step 1560 may include determining what operations (if any) should be performed on the other associated LID ranges (e.g., synchronizing mapping operations, etc.). The determination of step 1560 may include accessing metadata 984 and / or 994, which may include and / or reference the clone's synchronization strategy.
[0227] Step 1570 may include performing the operations (if any) identified at step 1560 along with the requested memory operation. If one or more of the synchronization operations cannot be performed (e.g., additional logical address space 132 cannot be allocated for one or more of the clones), the underlying memory operation may fail.
[0228] Fig. 16 is a flow diagram of another embodiment of a method 1600 for implementing range cloning and / or range merging operations. Step 1610 may include cloning a range of LIDs, as disclosed above. Step 1610 may include cloning a set of LIDs associated with data stored on the storage medium 140 at respective storage addresses. Therefore, step 1610 may include associating two or more different sets of LIDs with the same set of storage locations (e.g., the same data). Step 1610 may further include storing one or more persistent annotations 366 on the storage medium 140 and / or rewriting the data in an updated context-dependent format, as disclosed above. Step 1610 may link the two or more sets of LIDs by, among other things, including metadata 984 and / or 994.The metadata 984 and / or 994 may include and / or reference a clone synchronization strategy, merge strategy, and / or the like, as disclosed above.
[0229] Step 1620 may include performing storage operations with respect to one or more of the two or more cloned LID ranges. Step 1620 may include synchronizing mapping operations between the cloned ranges. The storage operations of step 1620 may include appending data to the storage medium 140 and / or associating the appended data with LIDs of one or more of the different LID ranges.
[0230] Step 1630 includes receiving a request to merge the two or more LID ranges from step 1610. The merge request may be received through interface 131 and / or may be part of another higher-level operation, such as an atomic store operation or the like.
[0231] Step 1640 may include identifying merge conflicts between the two or more sets of LIDs (if any). Identifying merge conflicts may include identifying LIDs that have been modified in more than one of the two or more cloned LID ranges. Again, Fig. Referring to Figure 9C, step 1640 may include identifying a merge conflict in state 941D in response to determining that LIDs 072-073 in range 914 have been modified, and the corresponding LIDs 972-973 in range 924 have also been modified. As such, step 1640 may include comparing modifications within the LID clones to identify instances where conflicting modifications would be mapped to the same LID in the merge operation.
[0232] Step 1650 may include resolving merge conflicts identified at step 1640. Step 1650 may include determining an applicable merge strategy, which, as disclosed above, may determine how to resolve merge conflicts. The merge strategy may specify which version of an LID is included in the merged LID space and / or whether conflicts are resolved by maintaining separate copies of the LID spaces. Step 1650 may further include merging the LID spaces in accordance with the resolved merge conflicts, as disclosed above.
[0233] Fig. 17 is a flow diagram of one embodiment of a method 1700 for implementing atomic memory operations. Step 1710 may include accessing a second set of identifiers corresponding to a first set of identifiers of an atomic memory request. Step 1710 may be performed in response to an atomic memory request related to the first set of identifiers (e.g., target LIDs, VIDs, or the like). The first set of identifiers may correspond to existing data stored on storage medium 140. Alternatively, the atomic memory request may include a request to allocate some (or all) of the first set of identifiers in logical address space 132 or VAS 532. The atomic memory request may correspond to a plurality of memory operations related to different, disjoint vectors of LIDs and / or VIDs.Accordingly, the first set of identifiers may comprise a plurality of disjoint I / O vectors.
[0234] In some embodiments, step 1710 includes allocating identifiers corresponding to the first set of identifiers in a separate address space, such as the transaction address space 1032, the intermediate index 1070, the VAS 532, or the like. Alternatively, step 1710 may include allocating identifiers within a particular range or sub-region of the logical address space 132. Step 1710 may include allocating a corresponding set of identifiers having the same scope and / or logical capacity as the first set of identifiers. The second set of identifiers may be accessed or allocated using, among other things, the redirection module 1034, as disclosed herein.Step 1710 may further include linking the first and second sets of identifiers, including by using a transaction map 1060 as disclosed herein.
[0235] Step 1710 may further include implementing a range clone operation to bind the second set of identifiers to data of the first set of identifiers (if any). The range clone operation may be implemented using any of the embodiments disclosed herein, including the range clone embodiments of Fig. 3A-E, the reference entry embodiments of Fig. 4A-E and / or the intermediate imaging layer embodiments of Fig. 5A-B.
[0236] Step 1720 may include implementing memory operations of the atomic memory request with respect to the second set of identifiers accessed at step 1710. Step 1720 may include redirecting memory operations from the first set of identifiers to the second set of identifiers (e.g., translating between the first and second sets of identifiers as disclosed herein). The memory operations of step 1720 may include storing data on the storage medium 140 using, among other things, the log storage module 137. The memory operations of step 1720 may include, but are not limited to, a) operations to allocate memory resources (e.g., operations to allocate logical and / or physical memory resources), b) operations to deallocate memory resources (e.g.,TRIM, persistent TRIM, and / or the like), c) writing data to the storage log, d) modifying existing data stored on the storage medium 140, e) overwriting data stored on the storage medium 140, and / or the like. The log storage module 136 may be configured to implement the storage operations elsewhere within the memory address space 144, such that operations configured to modify, overwrite, and / or replace existing data stored on the storage medium 140 are appended to the storage log, while the data to be modified, overwritten, and / or replaced by the appended data remains unchanged on the storage medium 140.
[0237] Data written to the storage medium 140 at step 1720 may include persistent metadata 114 configured to indicate the logical interface of the data. The persistent metadata 114 may be configured to bind the data to the second set of identifiers accessed at step 1710. Alternatively or additionally, the persistent metadata 114 may be configured to bind the appended data to identifiers of an intermediate address space that are bound to the second set of identifiers by, among other things, a persistent annotation 366 stored in the storage log. In some embodiments, the persistent metadata 114 may be further configured to indicate that the data is part of an atomic storage operation. Alternatively or additionally, the data may be stored using the persistent metadata 114 (e.g.,by association between the data and the second set of identifiers) as part of an atomic storage operation.
[0238] Step 1730 may include completing the atomic storage request. Completing the atomic storage request may include a range move operation configured to move the data from the storage operations implemented in step 1720 with respect to the second set of identifiers to the first set of destination identifiers. The range move operation may be completed in a single, atomic write operation. The single atomic write operation may include an operation to store persistent metadata on the storage medium 140 (e.g., a persistent indicia 366 and / or 1066B). The persistent metadata may be configured to modify the logical interface of the data written in step 1720 to bind the data to the first set of identifiers (e.g., the destination LIDs or VIDs of the atomic storage request).The persistent metadata may be configured to modify the logical interface of a plurality of different storage vectors (e.g., a plurality of different, disjointed sets of identifiers). Step 1730 may further include updating storage metadata 135 to reference the attached data by the first set of identifiers, which may include modifying one or more mappings in the forward index 160, the reference map 460, the intermediate mapping layer 1070, and / or the like.
[0239] Step 1730 may further include rewriting the data stored at 1720 in a context-dependent format configured to associate the data with the first set of logical identifiers. The data may be rewritten in one or more background operations. The storage layer 130 and / or the aggregation layer 530 may provide access to the data by the first set of identifiers before the data is rewritten.
[0240] Step 1730 may further include confirming completion of the atomic storage request. Completion may be confirmed in response to storing the persistent metadata (persistent note 366 and / or 1066B) on storage medium 140 and / or determining that the persistent metadata is stored with reasonable security on storage medium 140.
[0241] Fig. 18 is a flow diagram of another embodiment of a method 1800 for atomic memory operations. Step 1810 may include receiving an atomic memory request. The atomic memory request may be received at the memory layer 130 via, among other things, the memory interface 131. Alternatively, the atomic memory request may be received via the interface 531 of the aggregation layer 530. The atomic memory request may include a plurality of atomic memory operations to be performed within respective I / O vectors, each of which may correspond to a different, respective target set of LIDs and / or VIDs.
[0242] Step 1820 may include implementing the memory operations of the atomic memory request with respect to a set of transactional or in-process identifiers. The transaction identifiers may correspond to a particular subset of the logical address space 132; a VAS 532 (or a subset within a particular VAS 532); a separate namespace such as the transactional address space 1032 disclosed above; and / or the like. Step 1820 may include allocating and / or identifying transaction identifiers corresponding to the target identifiers of the atomic memory request; the transaction identifiers may include a plurality of identifier ranges and / or extensions corresponding to the I / O vectors of a vectored atomic memory request.In some embodiments, the transaction identifiers may be assigned and / or identified in different ranges and / or extensions than the target identifiers. For example, the transaction identifiers used to implement an atomic storage operation corresponding to two non-contiguous ranges of LIDs 1024-2048 and 6144-7186 may be implemented within a single range of ephemeral identifiers 10240-12288 or a plurality of smaller ranges and / or extensions of transaction identifiers. The transaction identifiers may be linked to the target identifiers using, among other things, the storage metadata 135. In some embodiments, the transaction identifiers are linked to the target identifiers in a transaction map 1060.The transaction map 1060 may be further configured to bind the transaction identifiers to storage locations corresponding to the target LIDs and / or VIDs.
[0243] Step 1820 may further include implementing memory operations of the atomic memory request with respect to the transaction identifiers. One or more of the atomic memory operations may be based on the availability of resources within a target or destination namespace, such as the logical address space 132, the VAS 532, or the like. In some embodiments, the atomic memory request may include, for example, a request to allocate a particular set of LIDs. Step 1820 may therefore include preliminarily reserving logical capacity within a target namespace using, among other things, one or more persistent notes 366, as disclosed above. Step 1820 may further include allocating and / or reserving corresponding transaction identifiers, as disclosed above.In response to a failure of the allocation operation, the atomic storage module 1035 may: a) miss the atomic storage request, or b) allocate and / or reserve a different set of target LIDs within the target namespace, which may be returned upon completion of the atomic storage operation.
[0244] The storage operations of step 1820 may further include appending data to the storage log associated with persistent metadata 114 configured to identify the data as part of an atomic storage operation. The persistent metadata may include the transaction identifiers of step 1810. Alternatively or additionally, the persistent metadata may include an atomic storage flag (or other data value) configured to indicate that the data is part of an incomplete atomic storage operation.
[0245] Step 1830 may include completing the atomic store request. Step 1830 may include closing the atomic store request implemented at step 1820. Closing the atomic store request may include performing a range shift operation to bind the data of the store operations of step 1820 to the target identifiers, which may include updating the storage metadata 135 to map the target identifiers to the data stored in the store operations of step 1820. The range shift operation may further include implementing one or more TRIM operations within the target namespace, as disclosed above.Completion of the atomic storage request may further include storing persistent metadata on storage medium 140 configured to: a) bind atomic storage request data to the target identifiers and / or b) indicate that the atomic storage request has been completed. The persistent metadata may be appended to the storage log in a single storage operation (e.g., in a persistent note 366 and / or 1066B). Step 1830 may further include acknowledging completion of the atomic storage request. Completion may be acknowledged in response to completing the range move operations and / or storing the corresponding persistent metadata.
[0246] Fig.19 is a flow diagram of another embodiment of a method 1900 for atomic storage operations. Step 1910 may include accessing the storage log on storage medium 140. Step 1910 may be performed by reconstruction module 1037 to reconstruct storage metadata 135 after an error condition. Reconstruction module 1037 may be configured to access the storage log in accordance with the log order of the storage log. Reconstruction module 1037 may be configured to identify the most recently appended point in the storage log and traverse the storage log in a reverse log order (e.g., from the head of the log to the tail).
[0247] Step 1920 may include identifying data of an incomplete atomic memory request. Step 1920 may include identifying data bound to transaction or in-process identifiers and / or identifiers of a transaction and / or in-process address space 1032. Step 1920 may include accessing persistent metadata 114 stored with data segments in the memory log. Step 1920 may further include determining that other persistent metadata within the memory log fails to modify the identified data bindings (e.g., fails to associate the data segments with identifiers in a different address space, such as the logical address space 132 and / or the VAS 532).
[0248] Step 1930 may include omitting the identified data segments from the storage metadata 135, which may include invalidating the storage location(s) comprising the data and omitting entries corresponding to the data segments from the forward map 160, intermediate mapping layer 1070, and / or the like. Therefore, step 1930 may include rolling back a failed atomic storage operation so that data from the partially completed atomic storage operation does not affect the target identifiers and / or namespace.
[0249] This disclosure has been made with respect to various embodiments. However, those skilled in the art will recognize that changes and modifications may be made to the embodiments without departing from the scope of the present disclosure. For example, various operations, as well as components for performing operations, may be implemented in alternative ways depending on the particular application or in consideration of any number of cost functions associated with the operation of the system (e.g., one or more of the steps may be deleted, modified, or combined with other steps). Therefore, this disclosure is to be considered illustrative and not in a limiting sense, and all such modifications are intended to be included within the scope of this disclosure.Likewise, benefits, other advantages, and solutions to problems have been described above with respect to various embodiments. However, benefits, advantages, solutions to problems, and any elements that may cause any benefit, advantage, or solution to occur or be enhanced are not intended to be considered a critical, necessary, or essential feature or element. As used herein, the terms "comprises," "comprising," and all other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that includes a list of elements not only includes those elements, but may also include other elements not expressly listed or inherent in such a process, method, system, article, or apparatus.Also, as used herein, the terms “coupled,” “coupling,” and any other variations thereof are intended to cover a physical connection, an electrical connection, a magnetic connection, an optical connection, a communications connection, a functional connection, and / or any other connection.
[0250] Additionally, as will be appreciated by those of ordinary skill in the art, the principles of the present disclosure may be reflected in a computer program product on a machine-readable storage medium having machine-readable program code means embodied in the storage medium. Any tangible, non-transitory storage medium may be employed, including magnetic storage devices (hard disks, floppy disks, and the like), optical storage devices (CD-ROMs, DVDs, Blu-ray Discs, and the like), flash memory, and / or the like. These computer program instructions may be loaded onto a general-purpose computer, special-purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, executing on the computer or other programmable data processing apparatus, produce means for implementing the specified functions.These computer program instructions may also be stored in a machine-readable memory that can instruct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the machine-readable memory produce a work of art, including an implementation of means for implementing the specified function. The computer program instructions may also be loaded into a computer or other programmable data processing device to cause a series of operations to be performed on the computer or other programmable device to produce a computer-implemented process, such that the instructions, when executed on the computer or other programmable device, provide steps for implementing the specified functions.
[0251] While the principles of this disclosure have been demonstrated in various embodiments, many modifications of the structure, arrangements, proportions, elements, materials, and components may be employed, particularly adapted to a particular environment and operating requirements, without departing from the principles and scope of this disclosure. These and other changes or modifications are intended to be included within the scope of the present disclosure.
Claims
[1] A method for implementing an atomic memory request from a memory client (106), the memory request referring to target identifiers (10-13, 36-38) in an address space, comprising: Linking, in an intermediate mapping layer (1060), transaction identifiers (Z0-3, Z6-8) to target identifiers of the atomic storage request, wherein the intermediate mapping layer further contains bindings between transaction identifiers and storage locations (P1-4, P6-8, P9-13, P100-102), Performing storage operations of the atomic storage request, wherein the storage operations are redirected to the transaction identifiers, and wherein the storage operations include appending data to the storage medium (140) with persistent metadata (114) configured to associate the appended data with the transaction identifiers (Z0-3, Z6-8); and Moving data of the redirected memory operations from the transaction identifiers to the target identifiers in response to completion of the memory operations of the atomic memory request. [2] The method of claim 1, wherein the persistent metadata (114) further comprises a flag or other indicator configured to identify the data as part of the atomic storage request. [3] The method of claim 1, wherein moving the attached data comprises storing metadata on the storage medium (140) configured to associate the attached data with the target identifiers (10-13, 36-38). [4] The method of claim 1, wherein the storage operations comprise storing data on a non-volatile storage medium having metadata (114) configured to bind the stored data to one or more intermediate identifiers associated with the respective transaction identifiers (Z0-3, Z6-8). [5] The method of claim 4, wherein moving the stored data comprises storing a persistent indicia (114) on the non-transitory storage medium to bind the one or more intermediate identifiers to respective target identifiers. [6] The method of claim 1, further comprising acknowledging completion of the atomic memory request in response to moving the data of the redirected memory operations. [7] Apparatus for implementing an atomic memory request from a memory client, the memory request relating to a first set (1040A, 1040B) of logical identifiers of an address space, comprising: a redirection module (1034) configured to map, in an intermediate mapping layer (1060), a second set (1042A, 1042B) of identifiers to the first set of logical identifiers of a storage request, wherein the intermediate mapping layer further includes bindings between the second set of identifiers and storage locations, a log storage module (136) configured to perform storage operations of the storage request on a storage device within the second set of identifiers, wherein the storage operations include appending data to the storage medium with persistent metadata (114) configured to associate the appended data with the second set of identifiers (Z0-3, Z6-8); and an atomic memory module (1035) configured to shift the memory operations of the memory request to the first set of logical identifiers. [8] The apparatus of claim 7, wherein the atomic storage module is configured to postpone the storage operations by storing persistent metadata (114) on the storage device, and wherein the persistent metadata is configured to bind data segments stored on the storage device in association with the second set (1042A, 1042B) of identifiers to the first set (1040A, 1040B) of identifiers. [9] The apparatus of claim 8, wherein the atomic storage module is configured to confirm completion of the storage request in response to a determination that the persistent metadata (114) is stored on the storage device. [10] The apparatus of claim 8, wherein the first set (1040A, 1040B) of identifiers comprises a plurality of disjoint sets of logical identifiers, and wherein the persistent metadata comprises a single persistent annotation configured to bind the data segments to respective logical identifiers within the plurality of disjoint sets of logical identifiers. [11] The apparatus of claim 7, wherein the redirection module (1034) is configured to allocate the second set (1042A, 1042B) of identifiers within an address space different from an address space of the first set (1040A, 1040B) of identifiers. [12] The apparatus of claim 7, wherein the redirection module (1034) is configured to allocate the second set of identifiers within a designated portion of an address space of the first set of identifiers. [13] The apparatus of claim 7, wherein the second set (1042A, 1042B) of identifiers corresponds to a transaction address space, the apparatus further comprising a reconstruction module (1037) configured to invalidate data segments associated with identifiers of the transaction address space. [14] The apparatus of claim 7, wherein the atomic storage module is configured to invalidate one of the identifiers within the first set (1040A, 1040B) of identifiers in response to shifting a storage operation configured to invalidate data of one of the identifiers in the second set (1042A, 1042B) of identifiers. [15] The apparatus of claim 7, wherein the log storage module (136) is configured to append data segments to an ordered log of data segments on the storage device, and wherein the log storage module is configured to append a data segment that is not related to the storage request between two or more data segments of the storage request within the ordered log of data segments. [16] System comprising: Means for receiving an atomic memory request relating to a set of logical destination identifiers (10-13, 36-38) of an address space; Means for linking a set of transaction identifiers (Z0-3, Z6-8) to the destination set of logical identifiers in an intermediate mapping layer, the intermediate mapping layer further maintaining bindings between the transaction identifiers and storage locations; means for attaching a plurality of data packets relating to the atomic storage request to a sequential storage log on a storage device, the data packets comprising persistent metadata (114) configured to associate the data packets with respective transaction identifiers; and Means for attaching a persistent annotation to the storage log, the persistent annotation configured to associate the plurality of data packets comprising persistent metadata configured to associate the data packets with respective transaction identifiers, with respective logical destination identifiers. [17] The system of claim 16, further comprising means for mapping logical identifiers of the set of destination identifiers to respective transaction identifiers. [18] The system of claim 16, further comprising means for binding one or more of the transaction identifiers to a storage location associated with a logical identifier linked to the respective one or more transaction identifiers (Z0-3, Z6-8). [19] The system of claim 16, further comprising means for providing access to the data packets comprising the persistent metadata configured to associate the data packets with respective transaction identifiers (Z0-3, Z6-8) using respective logical destination identifiers in response to appending the persistent annotation. [20] The system of claim 16, wherein the logical destination identifiers correspond to one logical address space, and wherein the transaction identifiers (Z0-3, Z6-8) correspond to another transaction address space (1032), the system further comprising: Means for reconstructing an index of mappings between logical identifiers of the logical address space and data packets on the storage device by using persistent metadata of the data packets; and Means for invalidating data packets associated with transaction identifiers corresponding to the other transaction address space.
Citation Information
Patent Citations
Cable connector assembly with improved indication effect
US20160149353A1
Systems and methods for adaptive error-correction coding
US9116823B2
Systems, methods, and interfaces for vector input / output operations
US20130166855A1
