Method and device for performing atomic write operations with comments

DE112014003349B4Inactive Publication Date: 2024-11-21SANDISK TECHNOLOGIES LLC
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
DE112014003349
Authority / Receiving Office
DE · DE
Patent Type
Patents
Current Assignee / Owner
Priority Date
2014-03-31
Filing Date
2014-08-13
Publication Date
2024-11-21
Estimated Expiration
Not applicable · inactive patent

Smart Images

  • Figure 00000000_0000_ABST
    Figure 00000000_0000_ABST
Patent Text Reader

Abstract

Procedure (500) comprising the following: Received (510) a request to perform an atomic write operation on a record, and in response to the request (202), writing (520) the data set over several non-volatile storage units (130, 130A, 130B), including storing metadata (150) in a dedicated location (314A) within at least one of the several storage units, wherein the metadata is usable to determine whether the write is complete; where the request is received by an application (122, 122A) which has been allocated an address range (312A) of the multiple memory units, with the dedicated space being outside the address range.
Need to check novelty before this filing date? Find Prior Art

Description

STATE OF THE ART Technical field

[0001] This disclosure relates generally to the storage of data on a physical recording medium and in particular to the atomic storage of data. Description of the state of the art

[0002] In some cases, data writing to a storage device may be interrupted, preventing the data from being written to the storage device completely. This type of failure can occur, for example, as a result of a power outage to the storage device, a mechanical failure of the storage device, an operating system crash, a broken connection coupling the storage device to a computer system, a loss of network connectivity to a network-attached storage (NAS) or storage area network (SAN), and so on.

[0003] Various attempts have been made to mitigate storage failures. For example, some storage devices use battery-backed cache memory to store data while it is being written to the device. In the event of a power outage, the cache data can be used to complete the write operation once power is restored. As another example, certain file systems maintain a log to track changes made to a storage device. In the event of a failure, the log can be reviewed to restore a previous valid state of the drive. Examples of such file systems include the New Technology File System (NTFS), the Journaled File System (JFS), and the Third Extended File System (ext3).Some applications (such as distributed applications in cloud computing environments) may retain data until it is verified that the data has been successfully saved to memory.

[0004] US 2012 / 0030408 ​​A1 concerns a virtual memory layer where logical addresses are linked to physical addresses through memory metadata. US 2013 / 0166855 A1 concerns systems and methods for vector input / output operations. SUMMARY

[0005] A method and devices with the features of the independent claims are provided; dependent claims relate to preferred embodiments. The present disclosure describes embodiments in which data is written atomically across multiple storage devices. In various embodiments, a comment is stored in connection with the execution of an atomic write across multiple storage devices. The comments can be stored in a dedicated location within at least one of the storage devices and contain metadata that can be used to determine whether the atomic write was completed successfully. In one embodiment, for example, the comment can be used to identify that a portion of the data was not successfully written to one of the storage devices after a hot-swap of the storage device.In some embodiments, the comments can also be used to perform an atomic write rollback to restore the memory devices to a previous valid state.

[0006] In one embodiment, a method is disclosed that includes receiving a request to perform an atomic write operation on a data record. The method further includes, in response to the request, writing the data record to multiple storage units, including storing metadata in a dedicated location within at least one of the multiple storage units. The metadata can be used to determine whether the write operation was successful.

[0007] In another embodiment, a device is disclosed that includes a vector storage module and a rollback module. The vector storage module is configured to perform a group of storage operations for a data set across multiple storage devices, including storing metadata independently of the data set. In such an embodiment, the metadata specifies the group of storage operations. The rollback module is configured to perform the rollback of the group of storage operations based on the stored metadata.

[0008] In yet another embodiment, a device is disclosed that includes a memory module and an error detection module. The memory module is configured to attempt an atomic write of a data record to an address range associated with a group of memory devices. The memory module is also configured to store metadata in a memory block with an address outside the address range, such that the metadata indicates the write of the data record. The error detection module is configured to determine, based on the metadata, whether the atomic write was successful.

[0009] In yet another embodiment, a device is disclosed comprising a first means and a second means. The first means serves to store data using a log structure, wherein the first means comprises multiple memory blocks. The second means serves to atomically write data to a subset of the multiple memory blocks and to write metadata to a memory block that is not within the subset, wherein the metadata is usable to perform the rollback of the data write.

[0010] In another embodiment, a method is disclosed. The method includes an application that issues a request for an atomic write operation. In such an embodiment, the atomic write operation involves writing a data record across multiple storage units and writing metadata to a dedicated location within at least one of the multiple storage units. The method further includes the application determining whether the atomic write operation has been completed based on a read attempt of the metadata. BRIEF DESCRIPTION OF THE DRAWINGS Fig. Figure 1 is a block diagram illustrating an embodiment of a computing system configured to write data atomically to multiple storage devices. Fig. Figure 2 is a block diagram illustrating an embodiment of a driver for the storage device that is executable to perform atomic write operations. The Fig. 3A-3C are block diagrams illustrating embodiments of logical and physical address spaces. The Fig. 4A and Fig. 4B are block diagrams illustrating embodiments of packages that can be used to facilitate atomic storage. Fig. Figure 5 is a flowchart illustrating one embodiment of a process. The Fig. 6A and Fig. 6B are block diagrams illustrating embodiments of devices that incorporate modules. Fig. Figure 7A is a block diagram illustrating another embodiment of a device having a storage medium and an atomic writing medium. Fig. 7B is a flowchart illustrating one embodiment of an algorithm implemented by an atomic writing system. Fig. Figure 8 is a block diagram illustrating an embodiment of a storage system that includes the computing system.

[0011] The disclosure contains references to "an embodiment." The appearance of the phrase "in an embodiment" does not necessarily refer to the same embodiment. Particular features, structures, or characteristics may be combined in any suitable manner consistent with this disclosure.

[0012] This disclosure also includes and references the accompanying drawings. In the drawings, similar symbols typically identify similar components, unless the context dictates otherwise. The illustrative embodiments described in the detailed description, drawings, and claims are not intended to be limiting. Other embodiments may be used, and other modifications may be made to these exemplary embodiments, without departing from the scope of the disclosure.

[0013] Various units, circuits, or other components in this disclosure may be described or claimed to be "configured" to perform a task or tasks. In such contexts, "configured to" is used to connote structure by indicating that the units / circuits / components possess structure (for example, circuits) that performs this task or tasks during operation. It can therefore be said of the unit / circuit / component that it is configured to perform the task even if the specified unit / circuit / component is not currently operational (for example, not powered on). The units / circuits / components used with the phrase "configured to" possess hardware, for example, circuits, storage program instructions, etc., that are executable to carry out the operation.Additionally, "configured to" may refer to a generic structure (for example, generic circuits) that is handled by software and / or firmware (for example, an FPGA or a general-purpose processor running software) to operate in a manner capable of performing the task(s) at hand. The statement that a unit / circuit / component is "configured" to perform one or more tasks is expressly not intended to refer to 35 USC § 112, sixth paragraph, for that unit / circuit / component.

[0014] Various units, circuits, or other components may be described using terms like "first," "second," etc. These terms are used as labels for the nouns they precede and do not imply any type of order (for example, spatial, temporal, logical, etc.). For a storage device that has multiple memory blocks, for instance, the terms "first" and "second" memory block may be used to refer to any two memory blocks (for example, including memory blocks contained within other memory blocks). In other words, "first" and "second" memory blocks are not limited to the initial two memory blocks of the storage device. DETAILED DESCRIPTION

[0015] Various embodiments are described here with reference to atomic data writing. The term "atomic" has its usual and accepted meaning according to the prior art and refers to the execution of an operation in such a way that if the operation fails before completion, the operation has no effect. Consequently, atomic writing is writing in which a write operation is performed in such a way that if it fails before completion (for example, if a power outage occurs before all data has been written), it does not change the state of the storage device (that is, it does not affect data already stored on the storage device). The term "atomic" can also refer to a group of operations such that the failure of one of the operations causes the group to have no effect.As discussed below, a data set can be written across multiple storage devices (called a "vector write") by performing a group of writes (for example, a separate write operation of a portion of the data set to each storage device). Consequently, if such a write is performed atomically, if the writes fail (for example, if one of the storage devices experiences a power outage), none of the group of writes will have any effect. In some cases, one or more actions can be performed to remove any effect of an atomic write that failed. These recovery actions can be generally called a "rollback." For example, if data is written atomically across multiple devices, writes to specific devices may have completed successfully, while others have failed.In such a case, the data from successful write operations can be marked as invalid. (Alternatively, in some embodiments described below, the option may be given to allow partially written data to remain on the storage device even though the atomic write has failed.)

[0016] The revelation initially describes, with reference to Fig. 1. A computing system configured to write atomically across multiple storage devices. A driver usable to facilitate atomic writes is defined with reference to Fig. 2 described. To facilitate this description, reference is made to the Fig. Sections 3A to 3C describe the logical and physical address spaces associated with the storage devices. Embodiments of a packet usable in atomic write operations are described with reference to the Fig. 4A and Fig. 4B described. Embodiments in which atomic writing operations are performed are described in further detail with reference to the Fig. Sections 5 to 7B are described. An exemplary storage system that incorporates such a computing system is finally described with reference to Fig. 8 described.

[0017] With reference to Fig. Figure 1 shows a circuit diagram of the computing system 100. The computing system 100 can be any suitable type of computing device, such as a server, laptop, desktop, mobile device, etc. In some embodiments, the computing system 100 can have multiple computing devices that work together. In one embodiment, for example, the computing system 100 can consist of several servers connected to a data center configured to store data for multiple clients, such as the storage system described below in connection with Fig. 8 is discussed. In the illustrated embodiment, the computing system 100 comprises a processor 110, a read / write memory (RAM) 120, and one or more memory devices 130 coupled via an interconnect 140. As shown, the RAM 120 can contain program instructions for one or more applications 122 and an operating system (OS) 124 executable by the processor unit 110. The OS 124 can include a driver 126 for memory devices 130, which in turn can include a controller 132 and one or more memory banks 134.

[0018] In various embodiments, the driver 126 is described as having diverse functionalities. These functionalities can be implemented as software, hardware, or a combination thereof. Furthermore, such functionality can be implemented by software outside of the OS 124, for example, as an application 122 in one embodiment. In another embodiment, this functionality can be implemented by software stored within a memory of the controller 132 and executed by a processor of the controller 132. In yet another embodiment, the controller 132 can have dedicated circuits for implementing the functionality of the driver 126. In summary, the illustration of the driver 126 as implemented in the software within the OS 124 should not be considered restrictive but rather as an illustration of an exemplary embodiment.

[0019] Storage devices 130 are representative of any physical medium on which data can be recorded. As used here, the term "recorded" refers comprehensively to the process of an electronic computing device storing, writing, or otherwise transferring one or more data values ​​onto a physical recording medium for subsequent use. Consequently, "physical recording medium" is used here to refer to any medium on which data can be recorded by an electronic computing device. Furthermore, the term "memory" is used here as a synonym for "physical recording medium." In view of this comprehensive definition, the terms memory (referring to RAM 120) and memory (referring to a storage device 130) in Fig. 1 and elsewhere in this revelation refer to volatile and / or non-volatile media. Such media may also be called “storage” here, and sections of such media may be called “blocks,” “cells,” “storage blocks,” etc. Collectively, a group of these blocks may be called a “storage array,” etc. The term “solid-state storage array,” as used here, refers to a storage array that includes non-volatile solid media, such as the various forms of media described below.

[0020] References in this disclosure to “accessing” data in a storage device 130 refer to any type of transaction, including writing data to a storage device 130 and / or reading data from a storage device 130, as well as TRIM operations, maintenance accesses, exploration accesses, load and save operations under storage semantics, and the like. Furthermore, and in view of the broad definitions of “storage” referenced above, these accesses may apply to a storage device that has non-volatile memory and / or volatile components.

[0021] In some embodiments, the storage devices 130 can be implemented such that they comprise non-volatile memory. Consequently, in such an embodiment, memory banks 134 can comprise non-volatile storage devices, such as hard disks (for example, Integrated Drive Electronics (IDE) drives, Small Computer System Interface (SCSI) drives, Serial Attached SCSI (SAS) drives, Serial AT Attachment (SATA) drives, etc.), magnetic tape drives, writable optical drives (for example, CD drives, DVD drives, Blu-ray drives, etc.), etc.

[0022] In some embodiments, the memory devices 130 can be implemented such that they comprise non-volatile solid-state memory. Consequently, in such an embodiment, memory banks 134 can comprise any suitable type of solid-state memory media, including, but not limited to, NAND flash memory, NOR flash memory, nano-RAM (“NRAM”), magnetoresistive RAM (“MRAM”), phase-change RAM (“PRAM”), racetrack memory, memristor memory, wiring-based nanocrystal memory, silicon oxide-based sub-10 nanometer process memory, graphene memory, silicon oxide nitride oxide silicon (“SONOS”), resistive read / write memory (“RRAM”), programmable metallization cell (“PMC”), conductor bridge RAM (“CBRAM”), etc. In some embodiments, memory banks 134 can comprise several different types of solid-state memory media.

[0023] In other embodiments, the memory devices 130 can be implemented to include volatile memory. Memory banks 134 can therefore correspond to any suitable volatile memory, including, but not limited to, RAM, dynamic RAM (DRAM), static RAM (SRAM), and synchronous dynamic RAM (SDRAM). Although shown independently of the processor unit 110, in some embodiments the memory devices 130 can correspond to memory within the processor unit 110, such as one or more cache levels (for example, L1, L2, L3, etc.) within the processor unit 110.

[0024] In summary, various functionalities are described here in connection with storage devices 130. Such functionalities can apply to any suitable storage form, including both non-volatile and volatile forms. Although particular embodiments of the driver 126 are described here within the context of non-volatile solid-state storage arrays, the driver 126 can therefore apply to other recording media, such as volatile storage and other types of non-volatile storage, especially those that have a reclaim process.

[0025] In one embodiment, the controller 132 is configured to manage the operation of a storage device 130. Consequently, the controller 132 can facilitate the execution of write operations to specified addresses (for example, “physical addresses,” as discussed below), including selecting the appropriate banks 134 and accessing data within the corresponding cells within those banks. The controller 132 can facilitate the execution of write operations, including the programming of specific cells. The controller 132 can also perform preparatory operations to allow subsequent write operations to a storage device 130, such as, in one embodiment, erasing blocks from cells for subsequent reuse. (The cycle of programming and erasing a block of cells can be called a “PE cycle.”)In some embodiments, the controller 132 implements separate read and write data pipelines to perform read and write operations in parallel. In one embodiment, the controller 132 is also configured to communicate with the driver 126 (discussed below) via the interconnect 140. In some embodiments, for example, the controller 132 communicates information for read and write operations via Direct Memory Access (DMA) transactions coordinated by a DMA controller. Consequently, the controller 132 can support any suitable interconnect type, such as Peripheral Component Interconnect (PCI), PCI Express (PCI-e), Serial Advanced Technology Attachment (Serial ATA or SATA), Parallel ATA (PATA), Small Computer System Interface (SCSI), IEEE 1394 (FireWire), Fiber Channel, Universal Serial Bus (USB), etc.In some embodiments, the controller 132 can also perform other operations, such as error checking, data compression, encryption and decryption, packaging and depackaging, etc.

[0026] In various embodiments, a storage device 130 is organized like a log structure storage device. As used here, the term "log structure" refers to an arrangement of data on a storage medium by using an attachment point to determine where data is stored; the attachment point is incremented sequentially through an "address space" as data is stored. A log structure storage device is simply a storage device organized using a log structure. The use of a log structure also implies that metadata is stored in conjunction with the data to allow the storage device 130 to revert to a previous state (that is, a "log checkpoint").Such a recovery can be performed, for example, to facilitate recovery after a crash due to a power outage, to restore the last known valid state in case of data corruption, and so on. As used here, the term "address space" refers to a range of addresses (that is, a collection of one or more addresses) that can be used to specify data within a storage device. As described below, a log structure storage can have both logical and physical address spaces.The term "logical address space" refers to an address space as perceived by higher-level processes (for example, by applications 122 and the OS 124), although this address space may not be representative of how data is actually organized on the physical medium of a storage device 130 or the actual number of physical address spaces that are actually used, reserved, or allocated to a higher-level process. In contrast, the term "physical address space" refers to the address space used by lower-level processes and may indicate how data is organized on the physical medium of a storage device 130 and the actual number of physical address spaces used by a higher-level process. Implementations of logical and physical address spaces are described in more detail in conjunction with the respective sections. Fig. 3A and Fig. 3B discussed. One embodiment of a log structure is discussed in connection with Fig. 3C discussed.

[0027] In various embodiments, the use of a log structure can allow multiple instances of a data record to exist in a storage device 130 while the data is written to, modified, and rewritten to that memory. As part of tracking data in a physical address space, older instances of stored data (that is, those instances that are not the current instance) can be marked as invalid. For example, in one embodiment, if a value is to be updated, the value can be written to a memory block specified by the current append point (rather than the location where the value was previously stored). In response to the successful execution of the write, any previously stored instances of that value can be marked as invalid.As used here, the term "invalid" refers to data that the system can no longer store (for example, because a newer copy of the data exists). Similarly, the term "declare invalid" refers to marking data as invalid (for example, saving a record in a data structure). Declaring invalid can also refer to writing an instance of data to a storage device 130 if a previous instance of data exists on the storage device 130 (whereby this latter instance is rendered obsolete by writing the new instance).

[0028] In one embodiment, OS 124 implements a file system to facilitate the retrieval of data by applications 122. As used here, the term "application" (or "process") generally refers to a collection of program instructions that can be executed by one or more processors. Examples of applications might include OS 124, a file system manager, a database management system (DBMS), a user-side library, a driver (for example, the driver 126, filter drivers, a redundant array of independent drives (RAID) driver, etc.). In the illustrated embodiment, the applications 122 are user-level applications. The term "user-level" is used to describe applications that are not responsible for managing a computer system. Examples of user-level applications might include word processing applications, web browsers, etc.User-level applications are contrasted with core-level (or system-level) applications, which manage the operation of a computer system. Examples of core-level applications include an operating system, a driver, a hypervisor, and so on. In one embodiment, OS 124 may restrict user-level applications' access to certain functionalities (such as files, addresses, hardware, etc.) of Computer System 100; however, OS 124 may not restrict such functionalities for core-level applications.

[0029] In some embodiments, application data of Application 122 can be organized into one or more partitions called "virtual storage units." As used here, the term "virtual storage unit" or "VSU" refers to an allocated set of memory blocks (for example, logic blocks) that can be used by one or more applications to store application data. In some embodiments, VSUs can be implemented such that they provide more logic address space for Application 122 than the actual physical address space. The total number of logical blocks allocated to VSUs can therefore exceed the number of available memory blocks on the physical medium. To enable this allocation system to function, VSUs can be managed such that a VSU using less of its allocation shares physical media space with a VSU using more of its allocation.In some embodiments, VSUs can also be used to facilitate the caching of data from secondary drives. As used here, the term "storage unit" generally refers to either a storage device or a virtual storage unit within a storage device.

[0030] In some instances, an application 122 can access data within storage devices 130 by specifying a corresponding filename to the OS 124 via an application programming interface (API) request (in other cases, an application 122 can access data directly by specifying an address to read from or write to). In response to receiving the request, the OS 124 can access various file system information corresponding to directories and files (for example, within a set of inodes, the time allocation tables, etc.) to determine one or more addresses where data for the file is stored.

[0031] In one embodiment, the driver 126 is executable to allow applications 122 and the OS 124 to interact with storage devices 130. The driver 126 can therefore receive requests to perform read and write operations at specified logical block addresses and can issue corresponding commands to the controller 132 to execute these operations. In some embodiments, the driver 126 manages garbage collection for storage devices 130 to reclaim memory blocks containing invalid data. As used here, "reclaiming" a memory block refers to preparing the memory block for reuse (that is, such that the memory block can store new data). In the case of flash media, reclaiming may involve copying valid data from the memory block and erasing the block.In some embodiments, the driver 126 also maps logical addresses (for example, LBAs) to corresponding physical addresses to facilitate the execution of read and write operations (in other embodiments, the mapping of logical addresses to physical addresses can be performed elsewhere, such as on the controller 132). The driver 126 can therefore also implement one or more translation mappings (such as the mappings 222, which are described below in conjunction with...). Fig. 2 are discussed) manage, including adding and removing translations from the mappings while handling data on storage devices 130.

[0032] In various embodiments, the driver 126 can support the execution of atomic writes. As discussed above, an atomic write can be advantageous for conventional write operations because of its ability to be unaffected in the event of a failure. Consequently, in one embodiment, the driver 126 supports an atomic write to one of the storage devices 130. As described below, in some embodiments, to facilitate such a write operation, data can be encapsulated within one or more packets that have a header containing different forms of metadata. In one embodiment, this metadata can include one or more bits that identify that the data is stored as part of an atomic write.The driver 126 can further access this metadata to determine whether the write was successful and, if justified, to perform a rollback of the atomic write. In various embodiments, the driver 126 also supports atomic writing across multiple storage devices 130, virtual storage devices, or a combination thereof. (While particular embodiments may be described here in terms of atomic writing across multiple storage devices, such a description may consequently also apply to virtual storage units, including multiple virtual storage units located within a single storage device.) In such an embodiment, a data record can be divided into sections that are distributed across the storage units. For example, a data record spanning a range of addresses can be divided based on addresses.A group of write operations can then be performed such that each saves a section to a respective storage device 130. In the illustrated embodiment, to facilitate atomic writing, the driver 126 further stores a comment 150 as part of the atomic write operation to determine whether each of the write operations performed to the storage devices 130 was successfully saved. If one of the write operations fails (for example, if a power failure occurs in one of the storage devices 130 midway through), the driver 126 (or an application 122) can consequently use the comment 150 to identify that the vector write was not fully completed and perform a rollback of the write.

[0033] In one embodiment, comment 150 consists of metadata stored in a dedicated location within at least one of the storage devices 130. This location therefore does not contain any of the application data for which atomic writing is performed. (In other embodiments, however, comment 150 may be stored in a non-dedicated location.) As described below in connection with Fig. As described in 3A, in some embodiments this location may be inaccessible to applications other than driver 126, such as applications 122. In one embodiment, for example, driver 126 (or OS 124) may reject a request for data from application 122 if it specifies the address of this location. In another embodiment, application 122 may have a limited view of the entire address space, so that the application is unaware of any addresses outside this limited view. In some embodiments, this location may be static; for example, driver 126 may always write comment 150 to the same logical block address (LBA) or to the same LBAs. However, in other embodiments, the location of comment 150 may be dynamic.In one embodiment, the location of comment 150 can be independent of the data being stored; for example, comment 150 can be located at a logical block address separate from the addresses where the data is stored. In one embodiment, comment 150 is stored at the beginning of the atomic write, so that if a failure occurs at any point, there is a higher probability that at least comment 150 was successfully written. Comment 150 can therefore be available to facilitate failure identification and any rollback. In another embodiment, comment 150 can be stored at the end of the atomic write. The mere presence of comment 150 can therefore indicate that the atomic write was completed successfully.

[0034] In some embodiments, the driver 126 supports several forms of atomic write rollback. Consequently, in one embodiment, the driver 126 supports a full rollback, in which all data for an atomic write (for example, the requested data to be written as well as any recorded metadata about the write) is invalidated if the write operation fails. Alternatively, in some embodiments, the driver 126 supports a partial rollback, in which a portion of the written data may be allowed to remain on storage devices 130, while other portions are declared invalid. As described below in connection with Fig. As described in Section 2, in such an embodiment, the driver 126 can form an interface with an application 122 to provide the application 122 with information about a write failure and to provide the application 122 with different options for rolling back the atomic write. In some embodiments, the application 122 can select whether to perform a full rollback, a partial rollback, or no rollback, depending on the information provided by the driver 126.

[0035] In various embodiments, implementing an atomic write by recording metadata, such as a comment 150, can be advantageous because it can reduce the overhead of performing an atomic write across multiple storage devices. For example, an atomic write across multiple storage devices 130 might involve the driver 126 performing a handshake with each storage device 130, sending a request and waiting for an acknowledgment from each device. The driver 126 could further maintain a log of sent requests and received acknowledgments to determine that an atomic write has failed if no acknowledgment is received from a storage device 130.However, maintaining a log and performing communication handshakes in this manner can result in the maintenance of a significant amount of additional metadata and significantly increase the amount of bus traffic between the driver 126 and storage devices 130. Therefore, in various embodiments where metadata recording is used, the driver 126 can implement atomic writing by recording metadata, such as a comment 150, without performing handshake operations, communication acknowledgments, or maintaining a log.

[0036] With reference to Fig. Figure 2 shows a block diagram of the driver 126. In the illustrated embodiment, the driver 126 has a distribution layer 210, storage layers 220A and 220B, and input / output (I / O) layers 230A and 230B. In other embodiments, the driver 126 may be configured differently than shown.

[0037] In one embodiment, the distribution layer 210 manages data storage across multiple storage units (for example, storage units 130 in the illustrated embodiment), including the execution of atomic write operations. The distribution layer 210 can therefore receive read and write requests from applications 122 and / or from the OS 124, determine the correct one or more storage devices 130, and issue appropriate instructions to storage layers 220, which are discussed below. In one embodiment, a given request can, for example, specify an address range. The distribution layer 210 can then use a mapping (similar to the mappings 222 described below) to identify the storage devices 130 associated with addresses in that range.In some embodiments, the distribution layer 210 also interfaces with applications 122 to perform atomic write operations (for example, via an API). In the illustrated embodiment, the distribution layer 210 can therefore receive requests 202 for atomic writes, which provide indications of write states 204, and receive rollback instructions 206.

[0038] Atomic write requests 202, in one embodiment, are requests from an application 122 to perform atomic write operations. In some embodiments, the atomic write requests 202 may include an instance (that is, a copy) of the data to be written, one or more addresses (for example, LBAs), and / or an indication that the request concerns an atomic write (as opposed to a non-atomic write). In response to an atomic write request 202, the distribution layer 210 may issue one or more corresponding instructions to lower layers 220 and 230. In the case of a distributed write (that is, a write across multiple storage devices 130), the distribution layer 210 may issue instructions to storage layers 220 to write data, as well as one or more instructions to store a comment 150 on a storage device 130, such as storage device 130A.In some embodiments, the distribution layer 210 can also store multiple redundant instances of a comment 150 for a given write on multiple storage devices 130 (as by the dot block in . Fig. 2 specified), the comment 150 can, for example, be stored on each of storage devices 130, a subset of storage devices 130, etc. As in connection with Fig. As described in Section 4B, comment 150 in one embodiment has a sequence number and address range associated with atomic writing (for example, the LBA range specified in the atomic write request 202). In such an embodiment, the sequence number indicates the order in which write operations occurred. In one embodiment, for example, the distribution layer 210 can maintain a counter that is set each time it receives a (atomic or non-atomic) write request. In some embodiments, the distribution layer 210 can append the sequence number to each data segment stored in storage devices 130.In such an embodiment, if the first and second sections are written to storage devices 130A and 130B respectively, each section can therefore have a respective instance of the sequence number (in addition to the instance of the sequence number contained in comment 150).

[0039] In some embodiments, atomic write requests 202 may also include data provided by an application 122 for storage in a comment 150. In one embodiment, for example, an application 122 may maintain various forms of metadata, which may be specifically for performing atomic writes or for any other purpose. In such an embodiment, the application 122 may provide this metadata to the driver 126 so that it may be stored in a comment 150. An application 122 may therefore be able to define and manage which metadata is stored within a comment 150 in some embodiments. Furthermore, in some embodiments, an application 122 (as opposed to the driver 126) may use this information to determine whether an atomic write has failed.

[0040] In some embodiments, the distribution layer 210 can determine whether an atomic write was successful based on metadata in the comment 150, as described above. (In another embodiment, however, this determination can be performed by the application 122 based on information provided by the driver 126, for example, about the write status 204, which is discussed below.) In one embodiment, the distribution layer 210 performs this determination for each atomic write request 202 that it receives. Alternatively, in another embodiment, the distribution layer 210 performs this determination after receiving a notification that a system failure has occurred that may have caused a write failure.The distribution layer 210 can, for example, receive a notification from the OS 124 that the computer system 100 is rebooting or has rebooted without performing proper shutdown procedures, that the connection between the computer system 100 and a storage device 130 has been interrupted, etc. In one embodiment, the distribution layer 210 can initially attempt to read the comment 150 (or comments 150) associated with this write to determine whether an atomic write was successful. If the comment 150 is missing or corrupted, the distribution layer 210 can determine that the write was not successful. However, if the comment 150 is successfully read, in one embodiment, the distribution layer 210 can read the address range specified in the comment to identify where the written data was stored.In one embodiment, distribution layer 210 can further attempt to read the sequence numbers stored with the data and compare them with the sequence number stored in comment 150. If these numbers do not match, distribution layer 210 can conclude that the atomic write was not successful. Conversely, if the sequence numbers do match, distribution layer 210 can conclude that the write was successful. (Other techniques for determining whether a write has failed are discussed below in connection with...) Fig. (4B discussed.) In any case, the distribution layer 210 can provide an indication of its conclusion regarding the application 122 that issued the request 202 via a write status 204.

[0041] In one embodiment, write states 204 indicate whether the write operations associated with atomic write requests 202 were completed successfully. In some embodiments, state 204 may only have a bolee value indicating failure or success. However, in several embodiments, write states 204 may contain additional information that applications 122 can use. For example, in one embodiment, a write state 204 may indicate that a write operation failed and also identify the addresses (e.g., LBAs) to which data was not successfully written (or conversely, to which data was successfully written).In one embodiment, a write status 204 can further identify the storage device or storage devices 130 associated with the failure, as well as a potential cause of the failure (for example, an inability to communicate with one or more storage devices 130). Based on this information, in some embodiments, an application 122 can make a reasoned determination as to how it wishes the driver 126 to handle the failure and provide a corresponding instruction, such as a rollback instruction 206.

[0042] In some embodiments, write states 204, instead of explicitly indicating whether an atomic write has been completed, can contain information that application 122 can use to infer whether an atomic write has failed. In one embodiment, for example, the driver 126 can perform an atomic write by initially writing the data and then subsequently writing a comment 150. Furthermore, the driver 126 can support an API call from application 122 that allows application 122 to query whether writes have been made to specific addresses (for example, LBAs) (that is, whether valid data exists at those locations). In such an embodiment, application 122 can issue an atomic write request 202 and then query, via the API call, whether a write has been made to the address of comment 150.In this example, a write status 204 can correspond to a response issued by the distribution layer 210 to the query. If the write status 204 indicates that no write was performed to the address, application 122 can conclude that comment 150 was not saved and that the atomic write therefore failed. In another embodiment, application 122 can only attempt to read comment 150 and receive a write status 204 indicating a read failure. Consequently, application 122 can conclude, based on the attempted read, that the atomic write was unsuccessful. In other embodiments, write states 204 can contain other information that application 122 can use, such as the content of comment 150, checksum values, timestamps, etc.

[0043] In one embodiment, rollback instructions 206 specify whether the driver 126 should perform a rollback of an atomic write, which may include failed atomic writes as well as successful (i.e., completed) atomic writes. (In another embodiment, the distribution layer 210 can perform a rollback automatically upon detecting a failure, that is, without receiving an explicit instruction to do so.) As discussed above, in some embodiments, the driver 126 can support multiple rollback types, such as a full rollback, a partial rollback, etc. Consequently, in such an embodiment, a rollback instruction 206 can specify the type of rollback to be performed. In some embodiments, a rollback instruction 206 can also include an instance of the data that was written incorrectly, so that the driver 126 can successfully complete the write operation.More specifically, in such an embodiment, data written atomically can be temporarily stored elsewhere in the computing system 100 (for example, RAM 120 or processor unit 110) until it can be verified whether the data has been successfully written to a storage device 130. (In one embodiment, this temporary storage capability can be a function of the application 122; in other embodiments, it can be performed by the OS 124, the driver 126, etc.) If a write failure occurs, the missing data (for example, as identified by the addresses specified in a write state 204) can be included in a rollback instruction 206 to complete the write operation by attempting to write the missing data to the storage devices 130.In one embodiment, when the driver 126 performs a full or partial rollback to remove data written during a failed atomic write, the driver 126 may issue one or more TRIM commands to invalidate the data, which, as described above, may involve retaining the requested data as well as any metadata recorded during the write operation. In other embodiments, the driver 126 may employ other techniques to remove the data written during a failed atomic write, such as deleting the data, overwriting the data, etc.

[0044] In one embodiment, memory layers 220 manage operations with respect to a given storage device 130. In another embodiment, memory layers 220 translate logical addresses into corresponding physical addresses, as described in more detail in connection with the Fig. Described in Sections 3A to 3C. In the illustrated embodiment, each storage layer 220 maintains a separate mapping 222 for its device 130. (In other embodiments, the layers 220 may share a mapping, a mapping may be carried by any other block, etc.) In various embodiments, mappings 222 perform translations of logical addresses to physical addresses within a storage device 130. Consequently, as data is moved and invalidated, it may reside at different physical addresses on a storage device 130 over time. However, by using a mapping 222, an application may be able to access a most recent data record by specifying the same logical address (for example, the same logical block address (LBA)), even though two or more versions of the data may reside at different physical addresses.Mappings 222 can be implemented by using any suitable data structure. In one embodiment, mappings 222 are therefore binary tree data structures. In other embodiments, a mapping 222 can be an array, a linked list, a hash table, etc. In some embodiments, a mapping 222 can be implemented by using multiple data structures. In various embodiments, the memory layers 220 can manage grooming for their respective storage devices 130 (that is, the identification and reclaiming of memory blocks so that they can subsequently be programmed to store data, which is also called garbage collection).Memory layers 220 can use any number of different criteria to select memory blocks for reclaim, such as the amount of invalid data, the temperature of data (i.e., how frequently the data is accessed), the amount of erroneous data captured when data is read from those blocks, etc.

[0045] I / O layers 230 provide an interface between the driver 126 and memory controllers 132 on storage devices 130. Consequently, I / O layers 230 can handle subordinate interface operations, such as issuing program, erase, and read commands to the controllers 132. I / O layers 230 can also track various forms of drive state for storage device 130. In some embodiments, I / O layers 230 can communicate with a DMA controller to configure one or more channels for DMA transactions via the interconnect 140. Although shown as independent blocks, in one embodiment, I / O layers 230 can form a single block.

[0046] With reference to Fig. Figure 3A depicts a block diagram of atomic memory with respect to a logical address space 302 and a physical address space 304. In one embodiment, the logical address space 302 represents the organization of data as perceived by higher-level processes, such as the applications 122 and the OS 124. In another embodiment, the physical address space 304 represents the organization of data on the physical media.

[0047] In one embodiment, the logical address space 302 is subdivided into logical addresses corresponding to logical blocks 310A to 310D (also called sectors). In some embodiments, the logical addresses are logical block addresses (LBAs) (in other embodiments, the logical addresses may correspond to some form of logical identifier). In one embodiment, logical sectors / blocks 310 represent the smallest block of data associated with a given logical address. As just one example, a logical block 310 may be approximately 512 bytes in size (while logical erase blocks and logical pages, discussed below, may be approximately 40 MB and 8 kB, respectively). As yet another example, a logical block 310 may be approximately 4 kB in size.

[0048] In one embodiment, the physical address space 304 is subdivided into physical addresses that correspond to the arrangement of data on the physical recording media. This is explained in more detail below in connection with… Fig. As discussed in section 3B, in one embodiment the contents of logical blocks 310 can be stored as packets 360 within logical delete blocks 320. As discussed in connection with Fig. As discussed in 3C, the physical address space 304 can be organized as a log structure in various embodiments, in which write operations can be performed on only one or more attachment points.

[0049] In some embodiments, the driver 126 may have a logical address space 302 that is significantly larger than the physical address space 304 of the storage device 130, such that the logical address space 302 is a "sparsely populated address space." (For the purposes of this disclosure, a sparsely populated address space is any logical address space that is at least 10% larger than the physical address space of the underlying storage devices.) In one embodiment, for example, the driver 126 may have a 48-bit sparsely populated address space compared to a 32-bit space for physical addresses. In such an embodiment, applications 122 may consume considerably less than the entire address range of the logical address space 302, so that substantial unused portions of the logical address space 302 may exist between blocks of stored data.

[0050] In some embodiments, the driver can allocate sections of the logical address space 302 to applications 122 for data storage. In the illustrated embodiment, for example, applications 122A and 122B were each assigned logical address ranges 312A and 312B, respectively. (In other embodiments, however, the applications 122 can share an allocated range 312.) Consequently, application 122A can read and write data by addressing logical blocks 310 within range 312A. In one embodiment, for example, application 122A can request an atomic write to an address range containing logical blocks 310A1 and 310A2. These logical blocks 310 can then be stored as packets 360A and 360B via storage devices 130A and 130B.Similarly, the application 122B can access data (such as data in logical block 310B) by addressing logical blocks 310 within area 312B. In some embodiments, areas 312 can correspond to separate partitions (also called virtual storage units (VSUs), as discussed above) on storage devices 130.

[0051] In various embodiments, the location of packets 360 written during an atomic write can depend on the mapping of logical addresses to corresponding physical addresses (which in turn may depend on the specific arrangement of partitions and the specific arrangement of storage devices 130 in the computing system 100). In the illustrated embodiment, an atomic write of logical blocks 310A1 and 310A2 to packet 360A and packet 360B, respectively, located in storage devices 130A and 130B, is therefore performed, because the logical addresses of logical blocks 310A1 and 310A2 are mapped to the physical addresses in both storage devices 130A and 130B.In other cases, the logical addresses of logical blocks 310A1 and 310A2 may be assigned differently, so that the data is written to only one storage device 130 (or only to a subset of storage devices 130). In one embodiment, an application 122A may therefore issue a first request for an atomic write, resulting in data being stored in both storage devices 130A and 130B, and may issue a subsequent second request, resulting in data being stored only on storage device 130A.

[0052] As discussed above, in various embodiments, the driver 126 can store one or more comments 150, which can be used to detect atomic write failure and atomic write rollback. In the illustrated embodiment, a comment 150 is stored within a logical block 310, shown for illustrative purposes as a dedicated block 314. In such an embodiment, dedicated blocks 314 can be located within a reserved logical address space 316. In one embodiment, the dedicated space 316 is a range of one or more logical block addresses reserved for storing dedicated blocks 314 (that is, a space 316 does not contain logical blocks 310 with application data from user-level applications, such as the applications 122 discussed above).In some embodiments, an area 316 may be located outside of areas 312 and may be inaccessible to applications 122 (as discussed above, this may be because the driver 126 rejects any data request specifying an address in a reserved area 316, an application 122 may be unaware of any addresses in a reserved area 316, etc.). In such an embodiment, an application 122 may still be able to read the content of comment 150 via an API call to the driver 126; however, the application 122 may not be able to directly access the content via a write request specifying the address (or addresses) of comment 150.Although depicted as a reserved logical address space, in other embodiments, address space 316 can be a physical address space containing one or more physical addresses of storage devices 130. In such an embodiment, address space 316 can extend across multiple storage devices 130 or it can be on only one of the storage devices 130. In still other embodiments, address space 316 can be unreserved; that is, a comment 150 can be stored in a memory block that is mixed with memory blocks containing user data (i.e., data of a user-level application). Space 316 can also contain addresses that are continuous or non-continuous. In the illustrated embodiment, applications 122A and 122B are each associated with one of space 316A and 316B, respectively.In such an embodiment, each area 316 can store comments 150 for write operations associated with its respective application 122. For example, dedicated block 314A can store one comment 150 for the most recent atomic write requested by application 122A; dedicated block 314B can store one comment 150 for the most recent atomic write requested by application 122B, so that the driver 126 is able to perform a rollback for both atomic writes. In some embodiments, an area 316 can be capable of storing multiple dedicated blocks 314 for a set of the most recent atomic writes. For example, area 316A can store comments 150 for the last four requested atomic writes. In other embodiments, applications 122 can share a reserved area 316.In some embodiments, a region 312 may be located together with its corresponding region 316; for example, region 316A may be adjacent to region 312A. In other embodiments, the regions 316 may be separate from their respective regions 312. In such embodiments, the regions 316 may be located together; for example, the regions 316 may form an adjacent range of logical addresses. In some embodiments, the regions 316 may be static, while in other embodiments, the regions 316 may be dynamic.

[0053] In the illustrated embodiment, the dedicated blocks 314 are stored as comment packets 362. In some embodiments, packets 362 can be protected so that they are not unintentionally modified or changed or deleted by applications 122, a garbage collection process of the driver 126, etc. To achieve this, in one embodiment, comment packets 362 can be stored in a special reserved area on storage devices 130, for example, a special logical delete block 320 reserved for storing metadata maintained by the driver 126. Access to the special reserved area on the storage devices 130 can be controlled such that an application 122 can only access metadata for its atomic storage operation. One embodiment of a comment packet 362 is described below in conjunction with Fig. 4B discussed.

[0054] With reference to Fig. Figure 3B shows a block diagram of memory blocks within the memory device 130. In the illustrative embodiment, a memory bank 134 is organized into logical erase blocks (LEBs) 320, which have several physical erase blocks (PEBs) 330 located in separate dies 306. A logical erase block 320 is further subdivided into several logical pages 340, which in turn have several physical pages 350. The physical pages 350 have several packets 360, which may be grouped into error correction code (ECC) pieces 370.

[0055] As used here, the term "erase block" refers to either a logical erase block or a physical erase block. In one embodiment, a physical erase block 330 represents the smallest memory block with a given memory bank 134 that can be erased at a given time (for example, due to the wiring of cells on the die). In another embodiment, the logical erase blocks 320 represent the smallest block that can be erased by the controller 132 in response to receiving an erase command. In such an embodiment, when the controller 123 receives an erase command specifying a particular logical erase block 320, the controller 132 can erase every physical erase block 330 within the logical erase block 320 simultaneously.It is noted that physical erase blocks 330 within a given logical erase block 320 (for example, blocks 330A and 330B) can be considered adjacent in the physical address space 304, even if they are located in the separate die 306. The term "adjacent" can therefore apply not only to data stored within the same physical medium, but also to data stored within separate media.

[0056] In one embodiment, a physical page 350 represents the smallest memory block within a given memory bank 134 that can be written to at a given time. In another embodiment, a logical page 340 is the smallest writable memory block supported by the controller 132. (In one embodiment, the controller 132 may have a buffer memory configured to hold data up to one logical page; after the buffer memory is full, the controller 132 may write the contents of the buffer memory concurrently to one logical page.) In some instances, subdividing a logical page 304 across multiple dies 306 may result in faster access times for a record when multiple dies 306 are accessed concurrently.

[0057] In one embodiment, a packet 360 represents the smallest memory block within a given memory bank 134 that can be read at a given time. In another embodiment, an ECC piece 370 is the smallest memory block that can be read by the controller 132. In some embodiments, packets 360 can be slightly larger than logic blocks 310 because they contain the contents of a logic block 310 (or in some cases, several logic blocks 310) as well as a packet header.

[0058] In some embodiments, the driver 126 can associate metadata with one or more of the memory blocks 320 to 370. As used here, the term “metadata” refers to system data that can be used to facilitate the operation of the solid-state storage device 130; metadata is to be contrasted, for example, with data generated by applications (that is, “application data”) or forms of data that are regarded by an operating system as “user data”.In one embodiment, a logical erase block 320 may, for example, contain metadata that specifies unrestricted user statistics (for example, the number of program erase cycles performed on this block 320), health statistics (for example, a value indicating how often corrupted data has been read from this block 320), security or access control parameters, sequence information (for example, a sequence indicator), a persistent metadata flag (indicating, for example, inclusion in an atomic memory operation), a transaction monitor, or the like. As in connection with... Fig. As described in 4A, a package 360 ​​can also contain various forms of metadata. In various embodiments, the driver 126 can use this information together with other forms of metadata to manage the operation of the storage device 130. The driver 126 can use this information, for example, to facilitate the execution of read and write operations, to restore storage devices 130 to a previous state (including, for example, restoring various data structures used by the driver and / or replaying a sequence of storage operations performed on the storage device 130), etc.

[0059] With reference to Fig. Figure 3C shows a block diagram of a log structure 380 within the physical address space 304. As shown, in various embodiments, data is stored sequentially at an appendage point 382 (also called a "head") that begins at an initial logical page 340, for example, page 340A. As additional data is stored, the appendage point 382 continues to subsequent logical pages 340 in the log structure 380, for example, pages 340B, 340C, etc. (As used here, the term "adjacent" can be used to describe memory blocks that are written one after the other in sequential order. As shown, for example, logical page 340A is adjacent to logical page 340B, and LEB 320A is adjacent to LEB 320B.)Finally, after storing sufficient data, append point 382 reaches the "last" page 340 in the storage device 130, and at this point, append point 382 loops back to the initial logical page 340. The log structure 380 is therefore represented as a loop / cycle. As more data is stored, the number of available logical pages 340 (shown as unshaded pages 340) decreases, and the number of used logical pages 340 (shown as shaded pages 340) increases. As discussed above, in order to reuse these logical pages 340 (that is, to make them available to receive further write operations), the driver 126, in one embodiment, performs erase operations on logical erase blocks 320.In one embodiment, an end 384 is maintained to identify the oldest logical page 340 still in use within the structure 380 (pages other than the one at the end are considered younger than the end). When the logical delete block 320 containing the oldest logical page 340 is finally deleted, the end 384 is moved forward to the next oldest logical page 340 in use at the end of the log structure 380.

[0060] In general, data that is modified less frequently than other data in storage device 130 migrate to end 384 (so that data can be described as having a "colder temperature," or simply as "cold data"). On the other hand, data that is modified more frequently (that can be described as having a "hotter temperature," or "hot" data) is typically located closer to head 382. Valid data located in LEB 320A is therefore likely to be "colder" than data in LEB 320B.

[0061] In embodiments where multiple storage devices 130 are used, each storage device 130 can be implemented as a separate log structure. Consequently, in one embodiment, each storage layer 220 can track an attachment point for its respective storage device 130. Therefore, when a write operation is performed across multiple storage devices 130, each data segment can be written to one of the attachment points of the respective storage devices 130. In other words, a data segment can be written to attachment point 382 for storage device 130A, a data segment can be written to attachment point 382 for storage device 130B, and so on. In other embodiments, however, storage devices 130 can be organized in a non-log structure format.

[0062] With reference to Fig. Figure 4A shows a block diagram of a packet 360. In the illustrated embodiment, the packet 360 has a head 410 and a payload 420. As described above, a packet 360 can, in various embodiments, contain various forms of metadata usable by the driver 126. In the illustrated embodiment, this metadata includes a logical block address (LBA) 412 for the data in the payload 420, a set of atomic bits 414, and an error correction code (ECC) 416. The packet head 410 can also contain additional forms of metadata (as specified by the Miscellaneous block 418), such as a packet size, links to other packets, etc. In other embodiments, blocks 412 to 420 can be configured differently than shown.

[0063] In one embodiment, atomic bits 414 can be used to determine whether an atomic write to a single storage device 130 has failed. In one embodiment, atomic bits 414 can indicate the beginning and end of packets 360 in a set of packets 360 that is stored during an atomic write. Consequently, if a beginning or end packet 360 is missing or corrupted, the driver 126 can conclude that the write was not completed successfully. In various embodiments, atomic bits 414 are separated from the comment 150, which can be used to determine whether an atomic write across multiple storage devices 130 has failed, as discussed above and described below.

[0064] With reference to Fig. Figure 4B shows a block diagram of an annotation package 362. In the illustrated embodiment, the annotation package 362 has a head 410 and a payload 420 containing an annotation 150. As discussed above, in some embodiments, an annotation 150 has an LBA range 422 to which the atomic write was performed and a sequence number 424 corresponding to the time of occurrence of the atomic write. A sequence number 424 can therefore include a timestamp (indicating, for example, the time of the data, a duration while the computing system 100 was in operation, etc.), a value indicating the number of writes performed, etc. In some embodiments, each atomic write across multiple memory units is assigned a separate sequence number 424.In other embodiments, each individual write to a storage unit, forming an atomic write, is assigned a corresponding sequence number 424, so that, for example, an atomic write across four storage devices is assigned four corresponding sequence numbers for the four individual write operations that constitute the atomic write. In some embodiments, a sequence number 424 can be assigned to each output message to facilitate atomic writing. For example, an atomic write may involve executing several individual write operations to each storage device 130, furthermore involving the output of multiple messages per individual write operation; consequently, in such an implementation, a sequence number 424 can be assigned to each message.Comment 150 may also contain additional information (as specified by block Miscellaneous 426), such that, in addition to specifying the storage devices 130 used during atomic writing, it may contain metadata other than that used to facilitate atomic writing (that is, metadata used for a different purpose), etc. This information may also include metadata provided by an application 122, as discussed above. Comment 150 in comment package 362 may also be implemented differently than shown. Consequently, in other embodiments, blocks 422 to 426 may be arranged differently; more or fewer blocks may be present.

[0065] The driver 126 can use various techniques to determine, based on metadata in a comment 150, whether an atomic write has failed. In one embodiment, an instance of the sequence number is stored with each set of packets 360 written to a storage device 130. The sequence number can, for example, be written to a payload 420 for the last packet 360 in each set. In such an embodiment, the driver 126 can use the LBA area 422 to identify where these sequence numbers are stored. The driver 126 can then compare these numbers with the sequence number 424. If they do not match, the driver 126 can conclude that the write has failed. However, if they match, the driver 126 can conclude that the write was successful. In another embodiment, the driver 126 can compare the LBAs in area 422 with LBAs 412 in packets 360.If they do not match, the driver 126 can conclude that the write operation was not successful. In yet another embodiment, the driver 126 can use the LBA area 422 to identify the relevant packets 360 on various storage devices 130. The driver 126 can then use atomic bits 414 to determine whether any of the individual write operations performed on storage devices 130 have failed. Consequently, if atomic bits 414 indicate a failure of any of the write operations, the driver 126 can conclude that the atomic write operation across the multiple storage devices 130 has failed. In various embodiments, if a rollback is performed, the driver 126 can issue TRIM instructions for each of the addresses specified in the address space 422 (or a subset of the addresses in the case of a partial rollback) to invalidate the data at those locations.

[0066] With reference to Fig. Figure 5 shows a block diagram of a method 500. The method 500 is an embodiment of a method that can be executed by a device, such as a computer system 100 or a storage device 130. In one embodiment, the device can therefore execute program instructions from a driver, such as driver 126, to execute the method 500. In some embodiments, executing the method 500 can restore the ability of a system to recover from a crash, such as a power failure, operating system failure, storage device failure, etc.

[0067] In step 510, a request (for example, request 202) is received to perform an atomic write operation on a data record. In some embodiments, the request is received by an application (for example, application 122A) that has been assigned an address range of the storage units (for example, storage units 130, virtual storage units within one or more storage devices 130, or a combination thereof). As discussed above, this address range can be a logical or physical address range, adjacent or non-adjacent, and it can be limited to a single storage unit or extend across multiple storage units. The address range can be accessible to the application for storing data (for example, a logical block 310A). In one embodiment, the request can specify one or more addresses of the range where data is to be stored.

[0068] In step 520, in response to the request, the data record is written across multiple storage units (for example, storage units 130A and 130B), including the storage of metadata (for example, comment 150) in a dedicated location within at least one of the storage units. In some embodiments, step 520 involves storing a first instance of the metadata in a dedicated location within the first of the multiple storage units (for example, comment 150 on storage unit 130A), and storing a second instance of the metadata in a dedicated location within the second of the multiple storage units (for example, a copy of the same comment 150 on storage unit 130B). In various embodiments, the metadata can be used to determine whether the write operation was successful.Metadata can be used explicitly or implicitly to determine write success. Explicit use of metadata might involve comparing timestamps or sequence identifiers across different instances of the metadata to ensure they belong to the same atomic write transaction. Implicit use of metadata might involve a simple check to see if the metadata exists for each storage unit. If the metadata for a particular storage unit is missing, this can provide a clear implicit indication that the atomic write transaction was unsuccessful.

[0069] In one embodiment, the dedicated location (for example, a reserved area 316) is located outside the address range (for example, an area 312). In some embodiments, step 520 includes assigning sequence numbers to writes performed to units, so that the sequence numbers indicate the order in which the writes occurred. In one embodiment, the metadata specifies an address range (for example, LBA area 422) where the record was written and a sequence number (for example, sequence number 424) corresponding to the write of the record. In some embodiments, step 520 includes storing a respective instance of a sequence number on each of the storage units to which the record is written and determining whether the write was successful by checking if the instances of the sequence number match.In some embodiments, the stored metadata includes metadata provided by an application that made the request. In one embodiment, step 520 involves reading the metadata from each of the multiple storage units and determining, based on the metadata reading, that the atomic write operation failed. In one embodiment, the reading and determination can be performed by the application performing the atomic write; however, in another embodiment, the reading and determination can be performed by the driver. In some embodiments, step 520 involves indicating (for example, via write status 204) that the successful completion of the write operation failed to an application that made the request and receiving a request (for example, rollback instruction 206) from the application to perform the rollback of the atomic write operation.In some embodiments, the specification involves identifying one or more logical block addresses to which the data record was not successfully written. In one embodiment, step 520 involves rolling back the atomic write operation, including invalidating data written to the specified address range (for example, LBA range 422).

[0070] It should be noted that a similar method in one embodiment is also considered for an application that interacts with a computing system that executes method 500, for example, a computing system that executes application 122.

[0071] With reference to Fig. Figure 6A shows a block diagram of a 600 device that incorporates modules. As used here, the term "module" refers to circuits configured to perform operations, or to memory containing program instructions that can be executed by one or more processors to perform operations. A module can therefore be implemented as a hardware circuit in a variety of ways. The hardware circuit may, for example, include user-defined, very high-level integration (VLSI) circuits or gate arrays, commercially available semiconductors such as logic devices, transistors, or other discrete components. A module may also be implemented in programmable hardware devices such as field-programmable gate arrays, programmable array logic, programmable logic devices, or the like.A module can also be any suitable form of non-volatile, computer-readable media that stores program instructions which are executed to perform specified operations. The device 600 can include modules to implement any of the functionalities described herein. In the illustrated embodiment, for example, the device 600 includes a vector memory module 610 and a rollback module 620.

[0072] In one embodiment, the vector memory module 610 is configured to perform a group of save operations for a data record on multiple storage devices, including saving metadata (for example, a comment 150), independently of the data record. In such an embodiment, the metadata specifies the group of save operations. In some embodiments, the vector memory module 610 is configured to store the metadata (for example, the comment package 362) within a memory block in at least one of the storage devices. The memory block may have an address (for example, an LBA associated with a range 316) that is not accessible to an application associated with the data record. In one embodiment, the vector memory module 610 is configured to indicate whether the group of save operations was executed successfully (for example, via a write status 204).In some embodiments, the vector memory module 610 is configured to perform a storage operation for a section of the data set, which involves encapsulating the section within a packet (for example, packet 360) with a packet header (for example, head 410). The packet header may include one or more bits (for example, atomic bits 414) that can be used to determine whether the storage operation was successful. In such an embodiment, the one or more bits are separate from the stored metadata.

[0073] In one embodiment, the rollback module 620 is configured to perform the rollback of the group of memory operations based on the stored metadata. In another embodiment, the rollback module 620 is configured to perform the rollback in response to receiving a request (for example, a rollback instruction 206). In some embodiments, the request may be a request to perform a partial rollback. In such an embodiment, the rollback module 620 is configured to perform the rollback of a subset of the group of memory operations, but not all of the group of memory operations, in response to the request. In several embodiments, the rollback module 620 is configured to perform the rollback of the group of memory operations by issuing one or more TRIM instructions to the multiple memory devices.

[0074] In some embodiments, the vector memory module 610 and / or the rollback module 620 are located within a controller, such as the controller 132. In another embodiment, the modules 610 and / or 620 may be located within a memory, such as the memory 120. In summary, the modules of the device 600 may be implemented in any suitable way to perform the functionality described herein. The device 600 may also correspond to any suitable structure that has the functionality of the modules 610 and 620. In one embodiment, the device 600 is a computing system that includes (or is coupled to) a memory, such as a memory device 130. In another embodiment, the device 600 is a card that includes a controller (such as a controller 132) and one or more memory elements (such as memory banks 134).In yet another embodiment, the device 600 is a computing system that includes a storage system which stores modules 610 and 620.

[0075] With reference to Fig. Figure 6B shows a block diagram of a device 650 comprising modules. The device 650 can have modules to implement any of the functionalities described herein and can conform to any suitable structure, such as the one discussed above with reference to the device 600. In the illustrated embodiment, the device 650 comprises a memory module 660 and a fault detection module 670.

[0076] In one embodiment, the memory module 660 is configured to attempt an atomic write of a data record to an address range associated with a group of memory devices. In such an embodiment, the memory module 660 is further configured to store metadata (for example, a comment 150) in a memory block (for example, in the dedicated block 314) that has an address outside the address range (for example, range 312A), the metadata specifying the write of the data record.In some embodiments, the memory module 660 is configured to write a respective section of the data set to each storage device in the group of storage devices (for example, packets 360A and 360B to storage devices 130A and 130B, respectively), along with a respective instance of a sequence number (for example, in payload 420 of a packet 360) indicating the order in which atomic writes were attempted. In some embodiments, the memory module 660 is configured to maintain a respective memory block (for example, dedicated blocks 314A and 314B) for each of several applications (for example, applications 122A and 122B) such that each memory block contains metadata corresponding to the latest atomic write for that application.In some embodiments, the memory module 660 is configured to manage multiple memory blocks for an application (for example, multiple dedicated blocks 314A in area 316A for application 122A) such that the multiple memory blocks contain metadata corresponding to several of the most recently executed atomic memory operations for the application. In one embodiment, the address range is a logical block address range (for example, area 312A) of a sparsely populated address space (for example, logical address space 302). In another embodiment, the external address is a logical block address that does not abut the logical block address range.

[0077] In one embodiment, the error detection module 670 is configured to determine whether the atomic write was successful based on the metadata. In other embodiments, the error detection module 670 is configured to determine whether the atomic write was successful by comparing the instances of the sequence number with an instance of the sequence number in the metadata (for example, sequence number 424).

[0078] With reference to Fig. Figure 7A shows a block diagram of a device 700 comprising a storage medium 710 and an atomic writing medium 720. The device 700 can correspond to any suitable structure that has the functionalities of the storage medium 710 and the atomic writing medium 720. The device 700 can, for example, be any suitable type of computing device, such as a server, laptop, desktop, mobile device, etc. In some embodiments, the device 700 can comprise several computing devices that operate together. In another embodiment, the device 700 is a card comprising a controller (such as a controller 132) and one or more memory elements (such as memory banks 134).

[0079] In various embodiments, the storage means 710 can implement any of the functionalities described herein with respect to the storage device 130. In one embodiment, the storage means 710 therefore serves to store data using a log structure. The storage means 710 can correspond to any suitable structure, such as the one discussed above with reference to the storage device 130 (for example, one or more memory banks 134, the computing system 100, the storage system 200, etc.). The phrase "means for storing data using a log structure" therefore refers to any of the structures listed above, as well as their corresponding equivalents.

[0080] In various embodiments, the atomic write device 720 can implement any of the functionalities described herein in relation to the driver 126. According to one embodiment, the atomic write device 720 serves to atomically write data to a subset (for example, area 312A) of the multiple memory blocks, and to write metadata to a memory block (for example, the dedicated block 314A) that is not within the subset. In such an embodiment, the metadata can be used for rolling back the data write. In some embodiments, the atomic write devices 720 can also implement functionalities other than those described in connection with the driver 126.

[0081] The atomic writing device 720 can correspond to any suitable structure. In one embodiment, the atomic writing device 720 is a hardware circuit configured to perform operations (for example, the controller 132). The hardware circuit can, for example, comprise user-defined very high-level integration (VLSI) circuits or gate arrays, commercially available semiconductors such as logic devices, transistors, or other separate components. The device 720 can also be implemented in programmable hardware devices such as field-programmable gate arrays, programmable array logic, programmable logic devices, or the like. In another embodiment, the device 720 comprises a memory in which program instructions are stored (for example, the RAM 120) that can be executed by one or more processors (for example, the processor unit 110) to implement an algorithm.In one embodiment, the agent 720 implements the algorithm associated with . Fig. 7B was discussed. In some embodiments, means 720 corresponds to modules 610, 620, 660 and / or 670. The phrase “means for atomically writing data to a subset of the multiple memory blocks and writing metadata to a memory block that is not within the subset” therefore applies to any of the structures listed above, as well as their corresponding equivalents.

[0082] With reference to Fig. Figure 7B is a block diagram illustrating algorithm 730. In one embodiment, algorithm 730 is implemented by the atomic writing device 720. In the illustrated embodiment, algorithm 730 includes step 732, which involves writing a data set to multiple storage devices and a comment to at least one of the storage devices, as discussed above. Algorithm 730 further includes, in step 734, using the comment to verify the successful completion of the data set writing process. As discussed above, in some embodiments, step 734 may involve comparing sequence numbers, logical addresses, atomic bits at positions identified by the comment, and so on.

[0083] With reference to Fig.Figure 8 is a block diagram showing a storage system 800 including a computing system 100. As discussed above, the computing system 100 can have one or more applications that operate on data stored in the storage device 130. In the illustrated embodiment, the computing system 100 runs a storage server application 810 to enable client systems 820A and 820B to access and store data on one or more storage devices 130 over the network 830. In one embodiment, for example, the storage system 800 can be associated within an enterprise environment where the server application 810 distributes enterprise data from the storage device 130 to clients 820. In some embodiments, the clients 820 can run other server applications, such as web servers, mail servers, virtual private network (VPN) servers, etc.to distribute data to other computing systems. In some configurations, the Storage Server Application 810 can, for example, implement various (NAS) protocols, such as File Transfer Protocol (FTP), Network File System (NFS), Server Message (SMB), Apple File Protocol (AFP), etc. In some configurations, the Computing System 100 can be one of several Computing Systems 100 configured to implement a Storage Area Network (SAN).

[0084] This disclosure was made with reference to various exemplary embodiments. However, the person skilled in the art recognizes that changes and modifications to the exemplary embodiments can be made without departing from the scope of this disclosure. Various operating steps and components for carrying out operating steps can, for example, be implemented in alternative ways depending on the specific application or considering any number of cost functions associated with the operation of the system (for example, one or more steps can be deleted, modified, or combined with other steps). This disclosure is therefore to be regarded as illustrative and not limiting, and all such modifications are intended to be within its scope. Likewise, advantages, other benefits, and solutions to problems associated with various embodiments have been described.Advantages, benefits, solutions to problems, and any elements that may cause or enhance the occurrence of any advantage, benefit, or solution shall not be construed as a critical, necessary, or essential feature or element. As used here, the term "include" and any other variation thereof is intended to cover non-exclusive inclusion, such as that a process, procedure, article, or device comprising a list of elements may not only have those elements but may also have other elements not expressly listed or inherent in such process, procedure, system, article, or device.As used here, the terms “coupled”, “couple” and any variation thereof are intended to cover a physical connection, an electrical connection, a magnetic connection, an optical connection, a communication connection, a functional connection and / or any other connection.

[0085] Additionally, and as the average person skilled in the art understands, concepts of the present disclosure can be reproduced in a computer program product on a machine-readable storage medium that has machine-readable program code means embodied in the storage medium. Any specific non-volatile machine-readable storage medium can be used, including magnetic storage devices (hard disks, floppy disks, and the like), optical storage devices (CD-ROMs, DVDs, Blu-ray discs, and the like), flash memory, and / or the like. These computer program instructions can be loaded onto a general-purpose computer, special-purpose computer, or other programmable data processing device to create an engine such that the instructions running on the computer or other programmable data processing device provide means for implementing the specified functions.These computer program instructions can also be stored in machine-readable memory, which can instruct a computer or other programmable data processing device to operate in a specific manner, such that the instructions stored in the machine-readable memory constitute a manufactured item, including the means of implementation that executes the specified function. The computer program instructions can also be loaded onto a computer or other programmable data processing device to cause the execution of a series of operational steps on the computer or other programmable device to produce a computer-implemented process, such that the instructions running on the computer or other programmable device provide steps for implementing the specified functions.

[0086] Although the concepts of this disclosure have been shown in various embodiments, many changes to the structure, features, proportions, elements, materials, and components, which are particularly suitable for a specific environment and operating requirements, can be used without deviating from the concepts and scope of this disclosure. These and other changes and modifications are intended to remain within the scope of this disclosure.

Claims

[1] Procedure (500) comprising: Receiving (510) a request to perform an atomic write operation on a record, and in response to the request (202), writing (520) the data set across a plurality of non-volatile storage devices (130, 130A, 130B), including storing metadata (150) in a dedicated location (314A) within at least one of the plurality of storage devices, the metadata being usable to determine whether the writing is complete; wherein the request is received from an application (122, 122A) that has been assigned an address range (312A) of the plurality of memory units, the dedicated space being outside the address range. [2] The method of claim 1, further comprising: Assigning at least one sequence number to write operations performed to the plurality of storage units (130, 130A, 130B), the sequence numbers indicating an order in which the write operations occurred, and where the metadata specifies an address range where the record is written and a sequence number corresponding to the writing of the record. [3] The method of claim 2, further comprising: Rollback of the atomic write operation, where the rollback comprises invalidating data written to the specified address range. [4] The method of claim 1, wherein writing comprises storing a respective instance of a sequence number on each of the plurality of storage devices (130, 130A, 130B) to which the data set is written, wherein the metadata comprises a different instance of the sequence number, and wherein the method further comprises determining whether the writing has been completed by checking whether the instances of the sequence numbers match. [5] The method of claim 1, further comprising: Reading the metadata from each of the plurality of storage units (130, 130A, 130B), and Based on reading the metadata, determine whether the atomic write operation failed. [6] The method of claim 1, further comprising: Indicating that the successful completion of the write has failed to an application, wherein the indicating comprises identifying one or more logical block addresses for which the record was not successfully written, and Receiving a request for rollback (206) of the atomic write operation from the application (122). [7] The method of claim 1, wherein the stored metadata comprises metadata provided by an application (122, 122A) that provided the request. [8] The method of claim 1, wherein storing the metadata comprises: Storing a first instance of the metadata in a dedicated location within a first of the plurality of storage devices, and Storing a second instance of the metadata at a dedicated location within a second of the plurality of storage devices, wherein the first and second instances are redundant. [9] Device (600) comprising: a vector storage module (610) configured to perform a group of storage operations for a data set on a plurality of non-volatile storage devices (130, 130A, 130B) in response to a request (202), including storing metadata in a dedicated location (314A) within at least one of the plurality of storage units independent of the data set, the metadata indicating the group of storage operations, the request being received from an application (122, 122A) that has been assigned an address range (312A) of the plurality of storage units, the dedicated location being outside the address range, and a rollback module (620) configured to roll back the group of storage operations based on the stored metadata. [10] The device of claim 9, wherein the vector storage module (610) is configured to store the metadata within a memory block in at least one of the plurality of storage devices, the memory block having an address inaccessible to an application associated with the data set. [11] The device of claim 9, wherein the vector storage module (610) is configured to indicate whether the group of storage operations was successfully executed, and wherein the rollback module (620) is configured to perform the rollback in response to receiving a request. [12] The device of claim 11, wherein the request is a request to perform a partial rollback, and wherein the rollback module (620) is configured to perform the rollback of a subset of the group of storage operations but not all of the group of storage operations in response to the request. [13] The apparatus of claim 9, wherein the rollback module (620) is configured to perform the rollback of the group of storage operations by issuing one or more TRIM commands to the plurality of storage devices. [14] The apparatus of claim 9, wherein the vector storage module (610) is configured to perform a storage operation for a portion of the data set comprising encapsulating the portion within a packet having a packet header, the packet header including one or more bits usable to determine whether the storage operation was successfully performed, and the one or more bits being separate from the stored metadata. [15] Device (650) comprising: a memory module (660) configured to attempt, in response to a request (202) from an application, an atomic write for a data set to an address range associated with a group of non-volatile memory devices (130, 130A, 130B), the address range (312A) being assigned to the application (122A), the memory module being configured to store metadata in a dedicated memory block (314A) of the group of memory devices having an address outside the address range, the metadata indicating the writing of the data set, and an error detection module (670) configured to determine whether the atomic write was completed successfully based on the metadata. [16] The device of claim 15, wherein the storage module (660) is configured to write a respective portion of the data set to each storage device in the group of storage devices along with a respective instance of a sequence number indicating an order in which the atomic write was attempted, and wherein the error detection module (670) is configured to determine whether the atomic write was successfully completed by comparing the instances of the sequence number with an instance of the sequence number in the metadata. [17] The device of claim 15, wherein the memory module (660) is configured to maintain a respective memory block for each of a plurality of applications, each memory block having metadata corresponding to a most recent atomic write for that application. [18] The device of claim 15, wherein the memory module (660) is configured to maintain a plurality of memory blocks for an application, the plurality of memory blocks comprising metadata corresponding to a plurality of the most recent atomic writes for that application. [19] The device of claim 15, wherein the address range is a logical block address range of a sparse address space, and wherein the external address is a logical block address that does not abut the logical block address range.

Citation Information

Patent Citations

  • Apparatus, system, and method for atomic storage operations

    US20120030408A1

  • Systems, methods, and interfaces for vector input / output operations

    US20130166855A1