METADATA JOURNAL FOR A STORAGE SYSTEM

Abstract

An example involves receiving a read request specifying a target scope and a target checkpoint, and executing an initial query to retrieve the first entry of a metadata table. The first entry is the most recent of those entries older than the target checkpoint that record write scopes overlapping the target scope. The example also includes determining, if the write scope in the first entry does not cover the entire target scope, a remaining portion of the target scope that excludes the write scope in the first entry. The example further includes executing a second query to retrieve a second entry of the metadata table, where the second entry is the most recent record in a second subset of entries older than the target checkpoint that record write scopes overlapping the remaining portion of the target scope.

Description

background

[0001] Computer devices can include components such as a processor, memory, a cache system, and a storage device. The storage device can be a hard disk drive, which uses a magnetic medium to store and retrieve blocks of data. Some storage systems can transfer data between different locations or devices. For example, some systems can transfer and store copies of important data for archiving and recovery purposes. Brief description of the drawings

[0002] Some embodiments are described with reference to the following illustrations. Fig. Figure 1 is a schematic diagram of an exemplary storage system according to some embodiments. Fig. Figure 2 is a representation of an exemplary data structure in accordance with some implementations. Fig. Figure 3 is an illustration of an example process in accordance with some implementations. Fig. 4A-4C are illustrations of example operations in accordance with some implementations. Fig. Figure 5 is an illustration of an example process in accordance with some implementations. Fig. Figure 6 is an illustration of an example of a process according to some implementations. Fig. Figure 7 is a schematic diagram of an example computer device according to some implementations. Fig. Figure 8 is a diagram of a machine-readable medium that stores instructions in accordance with some implementations. Fig. Figure 9 is an illustration of an example process in accordance with some implementations.

[0003] In the drawings, identical reference numbers denote similar, but not necessarily identical, elements. The illustrations are not necessarily to scale, and the size of some parts may be exaggerated to make the example shown clearer. Furthermore, the drawings contain examples and / or embodiments that correspond to the description; however, the description is not limited to the examples and / or embodiments shown in the drawings. Detailed description

[0004] In this disclosure, the use of the term "a," "an," or "the" includes the plural forms unless the context clearly indicates otherwise. Similarly, the terms "includes," "including," "comprises," "have," or "have," when used in this disclosure, specify the presence of the elements indicated but do not exclude the presence or addition of other elements.

[0005] In some examples, a computer system can permanently store data on one or more storage devices. For instance, a server might store a collection of data on a local storage array and also store a backup copy of the data collection on a remote backup device. In some examples, the backup copy might be stored in a different format than the data collection. For example, the backup copy might be a deduplicated representation of the data collection. The term "storage system" as used here can include a single storage device or an array of storage devices. A storage system can also include one or more storage controllers that manage access to the storage device(s). A "data unit" can refer to any portion of the data that can be separately identified within the storage system.In some cases, a data unit may refer to a chunk, a collection of chunks, or another part of data.

[0006] In some examples, a storage system can store data units in persistent memory. Persistent storage can be achieved using one or more persistent (e.g., non-volatile) storage devices, such as disk-based storage devices (e.g., hard disk drives (HDDs)), solid-state devices (SSDs), such as flash memory devices, or a combination thereof. As used herein, a "controller" can refer to a hardware processing circuit that may include any or a combination of a microprocessor, a core of a multi-core microprocessor, a microcontroller, a programmable integrated circuit, a programmable gate array, a digital signal processor, or other hardware processing circuitry.Alternatively, a "control unit" can refer to a combination of a hardware processing circuit and machine-readable instructions (software and / or firmware) that can be executed on the hardware processing circuit.

[0007] In some examples, a collection of data can be stored in a block-based storage system. A "block-based" storage system can refer to a system that stores data in the form of data blocks (also referred to here as a "block level"). In some examples, a block level can be a level at which a block-based storage device (such as a hard disk drive (HDD), a solid-state drive (SSD), or similar) or a virtual volume can store data. The block-based storage device can receive the data blocks that comprise a data collection as a stream of data blocks.

[0008] In some examples, a journal can provide continuous data protection (CDP) for a storage system. The journal can be implemented as a sequential log of entries that store copies or details of block-level write operations in the storage system. For example, each entry can record the data blocks written to a storage volume, as well as the storage address(es) to which the data blocks were written (e.g., an offset and length in the storage volume). As such, the journal can form a historical record of all data written to the storage system. In addition, the journal can record or display journal events (referred to here as "checkpoints") that represent various fixed points in time. If the stored data becomes corrupted (e.g.,(e.g., through a malware attack), the journal entries can be read and then used to reconstruct the stored data as it existed at a point in time represented by a checkpoint. In some examples, this data reconstruction might involve reading and processing a set of journal entries, starting with the oldest entry and ending with the desired checkpoint. However, if this set of journal entries is relatively large (e.g., if a relatively large number of data writes were recorded between journal initialization and the desired checkpoint), the data reconstruction can consume a significant amount of processing and network resources.

[0009] In accordance with some implementations of this disclosure, a metadata journal can record information about data written to a block volume. Additionally, a data journal can store copies of the data blocks written to the disk. The metadata journal can be implemented as a database table (or similar data structure) that can contain multiple metadata entries (i.e., records or rows). For example, each metadata entry can store a reference to the copy of the data written (stored in the data journal) when the data was written. Additionally, each metadata entry can record a disk address or range (e.g., an offset and length) to which the data was written. Furthermore, in some implementations, each metadata entry can record an identifier of the last checkpoint that preceded the creation of that entry.When data reconstruction is required, a database query is executed against a target address range and a target checkpoint. The query returns the entry for the most recent data write operation that overlaps with the target range and also precedes the target checkpoint. If the address recorded in the returned entry does not cover the entire target range, the target range is modified to remove the portion recorded in the returned entry, and the query is repeated using the modified target range. This query process can be repeated until the results cover the entire target range. In this way, the journal can be used to perform data reconstruction relatively quickly and efficiently, reducing the consumption of processing and network resources. Various aspects of the disclosed technique are discussed further below with reference to the [reference to the relevant document / document]. Fig. 1-9. FIGS. 1-2 - Example of a storage system

[0010] Fig. Figure 1 shows an example of a storage system 100 with a computer device 110 and a storage device 140 according to some implementations. The computer device 110 may include a storage machine 120 to generate and / or transmit write operations to the storage device 140. For example, the storage machine 120 may receive an input data stream (“input”) and then send a block-based write operation to the storage device 140. The input may specify a file system operation (e.g., adding a new file, deleting an existing directory, moving an existing file, etc.). The write operation may cause stored data 125 (e.g., data and / or metadata blocks) to be written to a specific address or location on a specific disk 150 of the storage device 140.

[0011] In some implementations, the journal engine 130 can create or update a journal 160 and store part or all of the journal 160 in the storage device 140. The journal 160 can contain a metadata journal 170 and a data journal 180. In some implementations, the data journal 180 can be a data structure for storing copies of the stored data 125 (e.g., the data blocks written to the disk 150). Furthermore, the metadata journal 170 can be a data structure for storing information (i.e., metadata) about the write operations on the storage device 140. In some implementations, the metadata journal 170 can also record or specify checkpoints to represent different points in time.

[0012] In some implementations, the metadata journal 170 can be a database table containing multiple entries. Fig. Figure 2 shows, for example, a metadata table 200, which generally corresponds to a sample implementation of the metadata journal 170. Each entry in the metadata table 200 can record one or more metadata fields for another write operation performed by the storage machine 120. For example, each entry can contain an entry identifier (“identifier”) 210, a journal address (“address”) 230, a storage location (“location”) 240, and a checkpoint identifier (“checkpoint”) 250. The journal address 230 can be a reference (e.g., a pointer) to a copy of the data stored in the data journal 180 (i.e., a copy of the data written during the write operation). The storage location 240 can specify the storage address range (e.g., offset and length) on the disk 150 into which the data was written. The checkpoint identifier 250 can specify the last checkpoint that preceded the creation of the entry.In some implementations, the entry may contain an entry identifier 210, which uniquely identifies this entry. In other implementations, however, the entry may not contain an entry identifier 210 and may instead be indexed or identified by the entry's position (e.g., a row number) in the metadata table 200. An example process for generating the metadata table 200 is described below with reference to [reference missing]. Fig. 3 and 4A-4C.

[0013] It should be noted that Fig. While Figure 2 shows an example of metadata table 200, implementations are not limited in this respect. In some implementations, for example, the location field 240 can be replaced by two fields: a start offset field (e.g., to specify a start offset from the beginning of the disk) and an end offset field (e.g., to specify an end offset from the beginning of the disk). Furthermore, metadata table 200 can contain additional fields or elements, such as a journal type field (which indicates, for example, whether metadata table 200 contains information about writes to disk 150 or about writes during a failover process), an element type field (which indicates, for example, whether the data write operation is compressed or uncompressed), a compressed length field (which indicates, for example, the compressed length for a compressed write operation), and so on.Furthermore, the metadata table 200 can also contain fewer fields, different fields, etc. Additionally, the functionality of the metadata table 200 can be implemented in other types of data structures (e.g., a flat file, an XML file, etc.). In some alternative implementations, the metadata table 200 can be configured to represent both the metadata journal 170 and a data journal 180 (in ). Fig. (shown in 1). In such alternative implementations, for example, the journal address field 230 can be replaced by a field that stores the data copy itself (i.e., instead of storing the data copy in a data journal 180).

[0014] Referring again to Fig. 1. In some implementations, when certain entries in journal 160 reach a maximum age, these entries can be removed from journal 160. Furthermore, the removed entries can be used to create a mirror volume 170. The mirror volume 170 can be a copy of storage volume 150 as it existed at the time the removed entries were added to journal 160. For example, the write operations recorded in the removed entries can be applied (e.g., executed) to the mirror volume 170 in the recorded order.

[0015] In some implementations, the journal engine 130 can use journal 160 to reconstruct storage volume 150 as it existed at a point in time represented by a checkpoint. For example, if storage volume 150 becomes corrupted or is lost (e.g., due to a device failure or malware attack), the journal engine 130 can execute a database query to retrieve journal data for a target address range and a target checkpoint. The query can select (and return metadata from) a single journal entry from metadata journal 170 that corresponds to the most recent data write operation in a storage area that overlaps with the target area and also precedes the target checkpoint. As used here, a storage area "overlaps" the target area if part of the target area is also contained within part of the storage area.Journal Engine 130 can read the location range (from the single journal entry selected by the query) and determine whether that location range covers the entire target range. As used here, a location range "covers" the entire target range if the location range includes at least every part of the target range (for example, the target range is a subrange of the location range, or the target range is identical to the location range). If Journal Engine 130 determines that the location range (from the single journal entry selected by the query) does not cover the entire target range, the target range can be modified to remove the location range, and the query can be rerun using the modified target range. This query operation can be repeated until the results cover the entire target range.In this way, the Journal Engine 130 and Journal 160 can be used to reconstruct a data volume at a control point relatively quickly and efficiently, thus enabling efficient allocation of processing and network resources. An example data reconstruction process using Journal 160 is described below with reference to the [reference to be added]. Fig. 5 and Fig. 6 described.

[0016] In some implementations, the Journal Engine 130 can be implemented as part of (or together with) a database engine (in Fig. (1 not shown). As used here, a “database engine” can refer to instructions on at least one machine-readable storage medium that can be executed by at least one processor to manage a structured database using a query language. For example, a database engine can process queries defined using the structured query language (SQL) to load and retrieve records from a database table.

[0017] In some implementations, the memory engine 120 and / or the journal engine 130 can be implemented using hardware (e.g., electronic circuits) or a combination of hardware and programming (e.g., with at least one processor and instructions executable by that at least one processor). In implementations that use executable instructions, such instructions can be stored in machine-readable storage media (e.g., storage device 140), in hardware (e.g., circuits), etc. The storage device 140 can include one or more non-transient storage media such as hard disk drives (HDDs), solid-state drives (SSDs), optical disks, etc., or a combination thereof. In some implementations, the storage device 140 can also include one or more block-based storage devices.

[0018] In some implementations, computer device 110 can be a physical computer device (e.g., a server, a machine, a desktop, etc.). Computer device 110 can, for example, include a controller, memory, and persistent storage (in Fig. (1 not shown). The controller can be implemented by hardware (e.g., electronic circuits) or a combination of hardware and programming (e.g., with at least one processor and instructions that can be executed by that at least one processor and stored on at least one machine-readable storage medium). The memory can be implemented in semiconductor memory such as random-access memory (RAM). The persistent memory can include one or more non-transient storage media such as hard disk drives (HDDs), solid-state drives (SSDs), optical disks, etc., or a combination thereof. In other implementations, the processing unit 110 can be one or more virtual processing units (e.g., virtual machines, containers, etc.).For example, the storage engine 120 can be implemented in a first virtual machine, and the journal log engine 130 can be implemented in a second virtual machine. Furthermore, in such an example, the first virtual machine can contain the disk 150 and the second virtual machine the journal 160.

[0019] It should be noted that Fig. While Figure 1 shows an example, implementations are not limited by this example. For instance, it is conceivable that the storage system 100 could comprise any number of computer devices 110 and / or storage devices 140. In another example, it is considered that the functionality of the storage engine 120 and / or the journal log engine 130 could be in a single engine or software, in another engine or software of the storage system 100, or in an external system or device (in Fig. (1 not shown), may be contained in separate virtual computing devices, in a single virtual computing device, or in any combination thereof. Furthermore, the storage system may include 100 additional devices and / or components, fewer components, different components, different arrangements, etc. Other combinations and / or variations are also possible. FIGS. 3 and 4A-4C - Example procedure for creating a journal

[0020] Fig. Figure 3 shows an example process 300 for creating a journal, in accordance with some implementations. In some examples, process 300 can be accessed from part or all of the memory system 100 (shown in Fig. 1) be performed. Process 300 can be implemented in hardware or a combination of hardware and programming (e.g., machine-readable instructions that can be executed by one or more processors). The machine-readable instructions can be stored on a non-transitory, computer-readable medium, such as an optical, semiconductor, or magnetic storage device. The machine-readable instructions can be executed by a single processor, multiple processors, a single processing machine, multiple processing machines, etc. For illustration, details of Process 300 are given below with reference to the Fig. Sections 4A-4C describe some example implementations. However, other implementations are also possible.

[0021] Process 300 can begin with decision block 310, which determines whether a write command has been received. If the result is positive ("YES"), process 300 can continue in block 320, including inserting a copy of the write data into a data journal. Block 330 can include creating a new entry in a metadata journal. Block 340 can include populating the entry with a data journal reference and a storage location.

[0022] Block 350 may include populating the entry with an identifier of the last checkpoint. Block 360 may include executing the write command to store data at the memory location. After block 360, or after a negative determination in decision block 310 ("NO"), process 200 may continue in decision block 370, including determining whether a journal timer has expired. If decision block 370 determines that the journal timer has not expired ("NO"), process 300 may return to decision block 310 (i.e., to determine again whether a write command was received). Otherwise, if decision block 370 determines that the journal timer has expired ("YES"), process 300 may continue in block 380, including creating a new checkpoint for the journal. Furthermore, block 390 may include resetting the journal timer.After block 390, process 300 can return to decision block 310.

[0023] In Fig. 4A, for example, is an example where an initial sequence 400 (i.e., including blocks 410, 412, 414) is processed by a controller (e.g., the processing machine(s) located in the Fig. Block 410 is executed by the computing device 110 shown in Figure 1. As shown, block 410 involves creating a checkpoint C-1 at an initial time. For example, the controller might detect the expiration of a journal timer and then create or define checkpoint C-1. In some implementations, the journal timer might be a cyclic timer that specifies (i.e., upon expiration) a desired time interval between checkpoints in the journal log (e.g., two seconds, five seconds, etc.). In other implementations, the creation of the checkpoint might be triggered by alternative or additional events. For example, a checkpoint might be created in response to a user command, a program command, a system interrupt, or a system event (e.g., an error or failure event), etc.

[0024] Block 412 involves writing data unit “A” 241 to the area “L7-L8” (e.g., from memory location “L7” to memory location “L8” on data carrier 150). In response to write operation 412, the control unit stores a copy 231 of data unit “A” at journal address “J1” of data journal 180. The control unit also creates entry “1” in metadata table 200 to record information about write operation 412. Specifically, the control unit populates entry “1” with the data identifier “A”, the journal address “J1”, the memory location “L7-L8”, and the checkpoint identifier “C-1” (i.e., the last checkpoint that preceded write operation 412).

[0025] Block 414 involves writing data unit “B” 242 to the area “L2-L3” in storage medium 150. In response to write operation 414, the control unit stores a copy 232 of data unit “B” in journal address “J2” of data journal 180. Furthermore, the control unit creates entry “2” in metadata table 200 to record information about write operation 414. Specifically, the control unit populates entry “2” with the data identifier “B”, the journal address “J2”, the storage location “L2-L3”, and the checkpoint identifier “C-1” (i.e., the last checkpoint that preceded write 414).

[0026] Fig. Figure 4B shows an example in which the control unit executes a second sequence 402 (i.e., including blocks 420, 422, 424) after completion of the first sequence 400 (in Fig. 4A). As shown, block 420 includes creating a checkpoint C-2 at a second time. Block 422 includes writing data unit “C” 243 to the area “L1-L2”. In response to write operation 422, the ECU stores a copy 233 of data unit “C” in journal address “J3” of data journal 180. In addition, the ECU creates entry “3” in metadata table 200 to record information about write operation 422 (i.e., data identifier “C”, journal address “J3”, storage location “L1-L2”, and checkpoint identifier “C-2”). It should be noted that, as in Fig. As shown in Figure 4B, write operation 422 causes the portion of data unit "B" 242 at memory location "L2" to be overwritten by a portion of data unit "C" 243. However, a remaining portion of data unit "B" 242 (i.e., the portion of data unit "B" 242 that was not overwritten by write operation 422) remains during the process described in Figure 4B. Fig. The period shown in 4B is stored at position “L3”.

[0027] Block 424 involves writing data unit “D” 244 to the area “L5-L6” in data carrier 150. In response to write operation 424, the control unit stores a copy 234 of data unit “D” in journal address “J4” of data journal 180. Furthermore, the control unit creates entry “4” in metadata table 200 to record information about write operation 424 (i.e., data identifier “D”, journal address “J4”, storage location “L5-L6”, and checkpoint identifier “C-2”).

[0028] In Fig. 4C is an example shown where the control unit, after completion of the second sequence 402 (in Fig. (as shown in Figure 4B) performs a third sequence 404 (i.e., including blocks 430, 432, 434, 435, 436, and 438). As shown, block 430 includes creating a checkpoint C-3 at a third time. Block 432 includes writing data unit "E" 245 to the area "L4-L5". In response to write operation 432, the ECU stores a copy 235 of data unit "E" at journal address "J5" of data journal 180. Additionally, the ECU creates entry "5" in metadata table 200 to record information about write operation 432 (i.e., data identifier "C", journal address "J3", memory location "L1-L2", and checkpoint identifier "C-3").

[0029] Block 434 involves writing data unit “F” 246 to the area “L2-L3” in storage medium 150. In response to write operation 434, the control unit stores a copy 236 of data unit “F” in journal address “J6” of data journal 180. Additionally, the control unit creates entry “6” in metadata table 200 to record information about write operation 434 (i.e., data identifier “D”, journal address “J4”, storage location “L5-L6”, and checkpoint identifier “C-2”).

[0030] Block 435 involves creating a checkpoint C-4 at a fourth time. Block 436 involves writing data unit "G" 247 to the area "L2-L3". In response to write operation 436, the control unit stores a copy 237 of data unit "G" in journal address "J7" of data journal 180. Additionally, the control unit creates entry "7" in metadata table 200 to record information about write operation 436 (i.e., data identifier "G", journal address "J7", storage location "L2-L3", and checkpoint identifier "C-4").

[0031] Block 438 involves writing data unit “H” 248 to the area “L3-L4” in data carrier 150. In response to write operation 438, the control unit stores a copy 238 of data unit “G” in journal address “J8” of data journal 180. Furthermore, the control unit creates entry “8” in metadata table 200 to record information regarding write operation 438 (i.e., data identifier “H”, journal address “J8”, storage location “L3-L4”, and checkpoint identifier “C-4”). It should be noted that, as in Fig. Figure 4C shows that the newer write operations (i.e., the more recent write operations) can overwrite some or all of the data written to storage disk 150 by older write operations (i.e., the less recent write operations).

[0032] In some implementations, older entries can be removed from metadata table 200 (e.g., upon reaching a maximum number or age), and the write operations recorded in the removed entries can be performed to create a mirror volume (e.g., the one in Fig. 1. Mirror volume 190 shown). The mirror volume can represent storage volume 150 as it existed at the time the removed entries were added to metadata table 200.

[0033] It should be noted that the Fig. Figures 4A-4C show the entries of metadata table 200 with a "Data Identifier" field to ensure clarity and presentation. However, it is conceivable that metadata table 200 does not contain the "Data Identifier" field in some or all implementations. FIGS. 5-6 - Example procedure for reading a journal

[0034] Fig. Figure 5 shows an example process 500 for reading a journal, in accordance with some implementations. In some examples, process 500 can be accessed from part or all of the memory system 100 (shown in Figure 5). Fig. 1) be performed. Process 500 can be implemented in hardware or a combination of hardware and programming (e.g., machine-readable instructions that can be executed by one or more processors). The machine-readable instructions can be stored on a non-transitory, computer-readable medium, such as an optical, semiconductor, or magnetic storage device. The machine-readable instructions can be executed by a single processor, multiple processors, a single processing machine, multiple processing machines, etc. For illustration, details of Process 500 are given below with reference to Fig. Section 6 describes an implementation example. However, other implementations are also possible.

[0035] Block 510 can involve receiving a read request for a destination area and a target checkpoint in a journal. Block 520 can involve generating a database query for the destination area and the target checkpoint. Block 530 can involve executing the query to retrieve the last entry in a metadata table that matches the query.

[0036] Decision block 540 can include determining whether the location range recorded in the input covers the entire destination location range. If the determination is negative ("NO"), process 500 can continue into block 550, including modifying the destination location range to exclude the portion covered by the range recorded in the input. After block 550, process 500 can return to block 520 (i.e., to generate a new query for the modified destination location range and the destination control point). In this way, blocks 520, 530, 540, and 550 can form a loop that is repeated for several iterations (i.e., until a positive determination is reached in decision block 540).

[0037] As in Fig. As shown in Figure 5, if the decision block 540 is positive ("YES"), process 500 can continue in block 560, including the generation of output data using all query results (i.e., the query results from one or more instances of query execution in block 530). After block 560, process 500 can be completed. In some implementations, process 500 can be executed to reconstruct a disk at a checkpoint relatively quickly and efficiently, thereby enabling efficient allocation of processing and network resources.

[0038] For example, a controller (e.g., the one in) receives Fig. 1 shown memory controller 110) the read instruction 610 for the destination area “L1-L4” and the destination control point “CP-4” (see Fig. 6) In other words, the read command 610 is intended to retrieve the data located at memory locations “L1”, “L2”, “L3” and “L4” (e.g. in the Fig. 1. Data carrier 150) shown at the time when the checkpoint “CP-4” was created.

[0039] In response to the read command 610, the controller generates an initial query 620. In the Fig. In the example shown, the first query 620 is configured to identify from the metadata table 200 ("FROM Metadata") a set of table entries whose checkpoint values ​​lie before the target checkpoint "CP-4" and whose storage areas overlap the target area [L1-L4] ("WHERE Checkpoint <CP-4 AND Location overlaps [L1-L4]“). Anders ausgedrückt, kann die Abfrage 620 einen oder mehrere übereinstimmende Einträge identifizieren, deren Prüfpunktwert kleiner (oder älter) als „CP-4“ ist, was bedeutet, dass jeder übereinstimmende Eintrag vor der Erstellung des Zielprüfpunkts „CP-4“ in die Metadatentabelle 200 eingegeben wurde. Ferner muss jeder übereinstimmende Eintrag einen Speicherortbereich aufzeichnen, der sich zumindest teilweise mit dem Zielortbereich „L1-L4“ überschneidet (d. h., wenn ein Teil des aufgezeichneten Speicherortbereichs auch in einem Teil des Zielortbereichs „L1-L4“ enthalten ist).Furthermore, the first query 620 is configured to sort the identified set of table entries in descending order of the "ID" field value and restrict the query output to a single table entry sorted first in descending order ("ORDER BY ID Descending LIMIT 1"). Additionally, the first query 620 is configured to retrieve the Data, Address, and City fields ("SELECT Data, Address, City") from the single table entry sorted first in descending order by the "ID" field value. In some implementations, the "ID" field of the table entries may indicate the order in which the table entries were created in the metadata table 200. When the identified set of table entries is sorted in descending order of the "ID" field value (i.e.,(from newest to oldest entry), the single table entry that is sorted first is the most recent entry of the identified set of table entries. In other words, the output of query 620 is the most recent entry in a subset of table entries, where each of the subset of table entries is older than the target control point "CP-4" and captures a locus that at least partially overlaps with the target locus [L1-L4].

[0040] As in Fig. As shown in Figure 6, the control unit executes the first query 620 to identify the matching entry "6" in the metadata table 200 and generate the query result 630 using the matching entry "6". Specifically, the query result 630 contains metadata recorded in the matching entry "6" of the metadata table 200, namely the data identifier "F", the journal address "J6", and the location range "L2-L3".

[0041] The control unit detects that the location range “L2-L3” (recorded in entry “6”) does not cover the entire destination range “L1-L4” (i.e., does not include at least every part of it). In response to this detection, the control unit 640 determines a first modified destination range “L1, L4” (i.e., by excluding the area portion “L2-L3” from the destination range “L1-L4”) and then generates a second query 650 using the first modified destination range “L1, L4”.

[0042] The control unit executes the second query 650 to identify the matching entry "5" in metadata table 200 and generate query result 660 using the matching entry "5". Specifically, query result 660 contains metadata recorded in the matching entry "5" of metadata table 200, namely the data identifier "E", the journal address "J5", and the location range "L4-L5". It should be noted that the second query 650 in Fig. While query 6 is represented as a single query configured to retrieve data from two disjoint areas (i.e., areas "L1" and "L4"), implementations are not restricted in this respect. For example, the second query 650 can be executed as two related queries (or subqueries) retrieving data from two disjoint areas: one query for the target area "L1" and another query for the target area "L4". The results of these two related queries can then be combined to produce query result 660.

[0043] The control unit detects that the location range “L4-L5” (recorded in entry “5”) does not cover the entire first modified destination location range “L1, L4”. In response to this detection, the control unit 670 determines a second modified destination location range “L1” (i.e., by excluding the part “L4-L5” from the first modified destination location range “L1, L4”) and then generates a third query 680 using the second modified destination location range “L1”.

[0044] The control unit executes the third query 680 to identify the matching entry "3" in metadata table 200 and generate query result 690 using the matching entry "3". Query result 660 contains, in particular, metadata stored in the matching entry "3" of metadata table 200, namely the data identifier "C", the journal address "J3", and the location range "L1-L2".

[0045] The control unit determines that the location range "L1-L2" (recorded in entry "3") covers the entire second modified target location range "L1". In response to this determination, the control unit 695 combines query results 630, 660, and 690 (i.e., the results of executed queries 620, 650, and 680) to produce output 697. In some implementations, output 697 may contain, for each location in the requested target location range "L1-L4", the data segment that was last written to that location. For example, output 697, as in Fig. Figure 6 shows the portion of data unit "C" at position "L1", data unit "F" at positions "L2-L3", and the portion of data unit "E" at position "L4". In some implementations, output 697 can be used to restore the state of the destination area as it existed at the time of the destination checkpoint.

[0046] In some implementations, if multiple queries (e.g., queries 620, 650, 680) are executed against metadata table 200, and if the combination of the results of these multiple queries does not cover all locations in the requested target location range "L1-L4", an additional query against a mirror disk (e.g., the one in Fig. The query on the mirror volume shown in 190) can be executed to obtain the data for the missing location. Furthermore, the result of the query on the mirror volume can be combined with the results of the previous queries on the metadata table 200, and these combined results can be used to generate output 697.

[0047] In some implementations, query 620 can include a set of conditions to specify a memory area that overlaps, at least partially, with the target memory area "L1-L4". For example, suppose that in the entries of metadata table 200, each memory location is recorded as a start offset (SO) and an end offset (EO). Further, suppose that in query 620, each target location range is specified as a start-destination location (T) and a target length (L). It is also assumed that each write operation recorded in metadata table 200 has a maximum length (M) (e.g., a maximum write length imposed by the storage system). In this example, query 620 can specify that a matching entry must satisfy each of the following conditions: [SO < (T+L)], [SO > (TM)], and [EO > T].Query 620 can be restricted so that it does not return any entries that begin more than the maximum length M before the start of the target area, thus limiting the search space. In this way, query 620 can be executed without consuming a relatively large amount of processing resources.

[0048] For the sake of clarity and illustration, in Fig. Six different elements (e.g., the metadata table 200, the queries 620, 650, 680, and the results 630, 660, 690) are represented with "data identifier" fields or values. However, it is conceivable that these elements do not contain the "data identifier" fields or values ​​in some or all implementations.

[0049] It should also be noted that in the Fig. In the example shown in Figure 6, the descending order of creation of the table entries is determined by sorting the entry identifiers (i.e., row numbers) in descending order. However, implementations are not limited in this respect. In some implementations, for example, queries 620, 650, and 680 can use a database index 255, which specifies, for each storage location on a disk, the entries for write operations to that storage location in the order of entry creation. The database index 255 can be a data structure generated by a database engine and stored before queries 620, 650, and 680 are executed (e.g., in the persistent memory of the database). Fig. 1 shown computing device 110). In some implementations, such use of the database index 255 can enable relatively faster execution of queries 620, 650, 680. FIG. 7 - Example of a computer setup

[0050] Fig. Figure 7 shows a schematic representation of an exemplary computing device 700. In some examples, the computing device 700 can generally be a part or all of the computing device 110 (shown in Figure 7). Fig. 1A), which may be separate from the storage device 140. As shown, the computing device 700 may contain a hardware processor 702 and a machine-readable storage medium 705 containing instructions 710-750. The machine-readable storage medium 705 may be a non-transient medium. The instructions 710-750 may be executed by the hardware processor 702 or by a processing machine contained in the hardware processor 702.

[0051] Instruction 710 can be executed to receive a read request for journal data, where the read request specifies a target memory area and a target checkpoint. For example, see Fig. 6, a controller (e.g., the one in Fig. 1 Storage Controller 110 shown) the read command 610 for the target storage area “L1-L4” and the target checkpoint “CP-4”.

[0052] Statement 720 can be executed to perform an initial query in response to receiving a read request, in order to retrieve the first entry of a metadata table containing a multitude of entries. The first entry is the most recent entry in an initial subset of entries, and each entry in this initial subset is older than the target checkpoint and records a write range that overlaps at least partially with the target range. For example, the controller executes a query 620 configured so that a matching entry must have a checkpoint value less than (or older than) "CP-4," indicating that the matching entry was written to metadata table 200 before the creation of the target checkpoint "CP-4."Furthermore, query 620 is configured such that a matching entry must record a location range that overlaps at least partially with the target location range "L1-L4". Additionally, query 620 is configured such that if multiple entries are older than checkpoint "CP-4" and also overlap the target location range "L1-L4", these multiple entries are sorted in metadata table 200 in descending order of their creation (i.e., from newest to oldest), and the query output is a single entry sorted first in descending order (i.e., the most recent of the multiple entries). Accordingly, as in... Fig. 6 shown, query 620 was executed to identify the matching entry “6” in the metadata table 200 and to generate query result 630 using the matching entry “6”.

[0053] Statement 730 can be executed to determine whether a write range recorded in the first entry covers the entire destination range. Statement 740 can be executed, in response to the finding that the write range recorded in the first entry does not cover the entire destination range, to determine a remaining portion of the destination range that excludes the write range recorded in the first entry. Example: In Fig. 6. The control unit determines that the location range “L2-L3” (recorded in the corresponding entry “6”) does not cover the entire destination location range “L1-L4” (i.e., it overlaps). In response to this determination, the control unit 640 determines an initial modified destination range “L1, L4” (i.e., by excluding the part “L2-L3” from the destination range “L1-L4”).

[0054] Statement 750 can be executed to perform a second query to retrieve a second entry from the metadata table, where the second entry is the most recent record in a second subset of entries that are older than the target control point and record write areas that overlap at least partially with the remaining portion of the target area. For example, the controller generates (see Fig. 6) A second query 650 using the first modified destination range “L1, L4”. The control unit executes the second query 650 to identify the matching entry “5” in the metadata table 200 and generates the query result 660 using the matching entry “5”.

[0055] Furthermore, the control unit determines that the location range "L4-L5" (recorded in entry "5") does not cover the entire first modified destination location range "L1, L4". In response to this determination, the control unit 670 determines a second modified destination location range "L1" (i.e., by excluding the portion "L4-L5" from the first modified destination location range "L1, L4") and then generates a third query 680 using the second modified destination location range "L1". The controller executes the third query 680 to identify the matching entry "3" in the metadata table 200 and generates query result 690 using the matching entry "3". Furthermore, the control unit 695 combines the query results 630, 660, and 690 (i.e.,The results of executed queries 620, 650, 680) are used to generate output 697 if it determines that the location range "L1-L2" (recorded in entry "3") covers the entire second modified destination location range "L1". In some implementations, output 697 can include, for each location in the requested destination range "L1-L4", the data segment that was last written to that location. In some implementations, output 697 can be used to restore the destination range to the state it was in at the time of the destination checkpoint. FIG. 8 - Example of a machine-readable storage medium

[0056] Fig. Figure 8 shows a machine-readable storage medium 800 with instructions 810-850 according to some implementations. Instructions 810-850 can be executed by a single processor, multiple processors, a single processing machine, multiple processing machines, etc., which may be separate from a storage device. The machine-readable medium 800 can be a non-transient storage medium, such as an optical, semiconductor, or magnetic storage medium. Instructions 810-850 can generally correspond to the examples given above with reference to instructions 710-750 (in Fig. 7 (shown) were described.

[0057] Statement 810 can be executed to receive a read request for journal data, where the read request specifies a destination range and a destination checkpoint. Statement 820 can be executed, in response to receiving the read request, to perform an initial query to retrieve the first entry of a metadata table containing a plurality of entries, where the first entry is the most recent entry in a first subset of entries, and where each entry in the first subset is older than the destination checkpoint and records a write range that at least partially overlaps the destination range.

[0058] Statement 830 can be executed to determine whether a write range recorded in the first entry covers the entire destination area. Statement 840 can be executed, in response to the finding that the write range recorded in the first entry does not cover the entire destination area, to determine a remaining portion of the destination area that excludes the write range recorded in the first entry. Statement 850 can be executed to perform a second query to retrieve a second entry from the metadata table, where the second entry is the most recent entry in a second subset of entries that are older than the destination checkpoint and record write ranges that overlap at least a portion of the remaining portion of the destination area. FIG. 9 - Example of reading a journal

[0059] Fig. Figure 9 shows an example process 900 for reading a journal, in accordance with some implementations. In some examples, process 900 can be accessed from part or all of the memory system 100 (shown in Fig. 1) be performed. Process 900 can be implemented in hardware or a combination of hardware and programming (e.g., machine-readable instructions that can be executed by one or more processors). The machine-readable instructions can be stored on a non-transitory, computer-readable medium, such as an optical, semiconductor, or magnetic storage device. The machine-readable instructions can be executed by a single processor, multiple processors, a single processing machine, multiple processing machines, etc.

[0060] Block 910 can include the receipt of a journal data read request by a controller, wherein the read request specifies a destination range and a destination checkpoint. In response to the receipt of the read request, block 920 can include the execution of an initial query by the controller to retrieve an initial entry from a metadata table containing a plurality of entries, wherein the initial entry is the most recent entry in an initial subset of entries, and wherein each of the initial subset of entries is older than the destination checkpoint and records a write range that at least partially overlaps the destination range.

[0061] Block 930 may involve the control unit determining whether a write area recorded in the first entry covers the entire destination area. In response to the finding that the write area recorded in the first entry does not cover the entire destination area, Block 940 may involve the control unit determining a remaining portion of the destination area that excludes the write area recorded in the first entry. Block 950 may involve the control unit executing a second query to retrieve a second entry from the table, where the second entry is the most recent record in a second subset of entries, and where each of the second subset of entries is older than the destination control point and records a write area that overlaps at least a portion of the remaining portion of the destination area.Blocks 910-950 can generally correspond to the examples given above with reference to instructions 710-750 (in . Fig. 7 (shown) were described. conclusion

[0062] In accordance with some implementations described here, a metadata journal can record information about data written to a block volume. Additionally, a data journal can store copies of the data blocks written to the disk. The metadata journal can be implemented as a database table that can contain multiple metadata entries. Each metadata entry can store a reference to the copy of the data written during the data write operation (stored in the data journal). Furthermore, each metadata entry can record a disk address or disk range to which the data was written. In some implementations, each metadata entry can record an identifier of the last checkpoint that preceded the creation of that entry. When the data needs to be reconstructed, a database query is executed against a target address range and a target checkpoint.The query returns the entry for the most recent data write operation that overlaps with the target range and also precedes the target checkpoint. If the address recorded in the returned entry does not cover the entire target range, the target range is modified to remove the portion recorded in the returned entry, and the query is repeated using the modified target range. This query process can be repeated until the results cover the entire target range. In this way, the journal can be used to reconstruct data relatively quickly and efficiently, thereby reducing the consumption of processing and network resources.

[0063] It should be noted that the Fig. As examples 1-9 show, the implementations are not limited in this respect. For example, the system can be 100, as in Fig.Figure 1 shows additional devices and / or components, fewer components, different components, different arrangements, etc. Another example is that the functionality of Journal Engine 130 described above may be included in any other engine or software of System 100. Other combinations and / or variations are also possible.

[0064] Data and instructions are stored in appropriate storage devices, which are implemented as one or more computer-readable or machine-readable storage media. Storage media include various forms of non-transient memory, including semiconductor memory such as dynamic or static random-access memory (DRAM or SRAM), erasable and programmable read-only memory (EPROM), electrically erasable and programmable read-only memory (EEPROM), and flash memory; magnetic disks such as hard disks, floppy disks, and removable disks; other magnetic media including tapes; optical media such as compact discs (CDs) or digital video discs (DVDs); or other types of storage devices.

[0065] It should be noted that the instructions discussed above can be provided on a single computer-readable or machine-readable storage medium, or alternatively, on multiple computer-readable or machine-readable storage media distributed throughout a large system, possibly with multiple nodes. Such computer-readable or machine-readable storage medium or media are considered part of an article (or article of manufacture). An article or article of manufacture can refer to any single component manufactured or to multiple components. The storage medium or media can be located either in the machine on which the machine-readable instructions are executed or at a remote location from which machine-readable instructions can be downloaded for execution over a network.

[0066] The foregoing description includes numerous details to provide an understanding of the subject matter disclosed herein. However, implementations may be practiced without some of these details. Other implementations may include modifications and deviations from the details described above. It is intended that the accompanying claims cover such modifications and variations.

Claims

[1] A computer system comprising the following: a processor; and a machine-readable storage medium containing instructions, the instructions of which can be executed by the processor to: to receive a read request for journal data, wherein the read request specifies a destination range and a destination checkpoint; In response to receiving the read request, perform an initial query to retrieve an initial entry of a metadata table containing a plurality of entries, wherein the initial entry is a most recent entry in an initial subset of entries, and wherein each of the initial subset of entries is older than the target checkpoint and records a write range that overlaps at least partially with the target range; to determine whether a write range recorded in the first entry covers the entire destination range; in response to a finding that the write area recorded in the first entry does not cover the entire destination area, to determine a remaining portion of the destination area that excludes the write area recorded in the first entry; and to perform a second query to retrieve a second entry of the metadata table, wherein the second entry is a most recent record in a second subset of entries, and wherein each of the second subset of entries is older than the target checkpoint and records a write area that overlaps at least part of the remaining part of the target area. [2] Computer system according to claim 1, comprising instructions which can be executed by the processor to: to determine whether a write area recorded in the second entry covers the entire remaining part of the destination area; In response to the finding that the write area recorded in the second entry covers the entire remaining part of the destination area, to generate an output using combined results from the first query and the second query. [3] Computer system according to claim 2, comprising instructions which can be executed by the processor to: to restore the target area to a state that existed at the time of the target checkpoint, based on the output. [4] Computer system according to claim 1, comprising instructions which can be executed by the processor to: Detect a write operation of a data unit to a data carrier; in response to the detection of the write operation: to store a copy of the data unit in a data journal; to create a new entry in the metadata table; and to populate the new entry with a reference to the copy of the data unit stored in the data journal. [5] Computer system according to claim 4, comprising instructions that can be executed by the processor to, in response to the detection of the write operation: to populate the new entry with the location of the data unit stored on the data carrier; and to assign the new entry an identifier of a checkpoint that was last performed before the write operation was detected. [6] Computer system according to claim 4, wherein each entry of the metadata table includes metadata for a different write operation on the data carrier. [7] Computer system according to claim 1, comprising instructions which can be executed by the processor to: to sort the first subset of entries in the metadata table in descending order of creation; to select a single entry from the sorted first subset of entries, which is the first sorted entry in descending order, where the selected single entry is the first entry retrieved by the first query. [8] Computer system according to claim 7, comprising instructions which can be executed by the processor to: to sort the first subset of entries using a database index, where the database index is a stored data structure that specifies, for each storage location of a plurality of storage locations, the entries for write operations to that storage location in the order of entry creation. [9] Computer system according to claim 1, wherein: In each entry of the metadata table, a write area is recorded as start offset (SO) and end offset (EO); In the first query, the destination area is specified by a start-destination location (T) and a destination length (L); Each write operation recorded in the metadata table has a maximum write length (M); and The first query indicates that for each entry in the first subset of entries: SO<(T+L) SO>(T−M), and EO>T. [10] A non-transitory machine-readable storage medium containing instructions that can be executed by a processor to: to receive a read request for journal data, wherein the read request specifies a destination range and a destination checkpoint; in response to receiving the read request, to perform an initial query to retrieve an initial entry of a metadata table containing a plurality of entries, wherein the initial entry is a most recent entry in an initial subset of entries, and wherein each of the initial subset of entries is older than the target checkpoint and records a write area that overlaps at least partially with the target area; to determine whether a write range recorded in the first entry covers the entire destination range; in response to a finding that the write area recorded in the first entry does not cover the entire destination area, to determine a remaining portion of the destination area that excludes the write area recorded in the first entry; and to perform a second query to retrieve a second entry of the metadata table, wherein the second entry is a most recent record in a second subset of entries, and wherein each of the second subset of entries is older than the target checkpoint and records a write area that overlaps at least part of the remaining part of the target area. [11] Non-transient, machine-readable medium according to claim 10, containing instructions that can be executed by the processor to: to determine whether a write area recorded in the second entry covers the entire remaining part of the destination area; In response to the finding that the write area recorded in the second entry covers the entire remaining part of the destination area, to generate an output using combined results from the first query and the second query. [12] Non-transitory, machine-readable medium according to claim 11, containing instructions that can be executed by the processor to: to restore the target area to a state that existed at the time of the target checkpoint, based on the output. [13] Non-transient, machine-readable medium according to claim 10, containing instructions that can be executed by the processor to: to detect a write operation of a data unit to a data carrier; in response to the detection of the writing process: to store a copy of the data unit in a data journal; to create a new entry in the metadata table; and to populate the new entry with a reference to the copy of the data unit stored in the data journal. [14] Non-transitory, machine-readable medium according to claim 13, containing instructions that can be executed by the processor to, in response to the detection of the write operation: to populate the new entry with the location of the data unit stored on the data carrier; and to mark the new entry with the identifier of a checkpoint that was last performed before the write operation was detected. [15] A procedure comprising the following: Receiving a read request for journal data by a controller, wherein the read request specifies a destination range and a destination checkpoint; In response to receiving the read request, the controller executes an initial query to retrieve an initial entry from a metadata table containing a plurality of entries, wherein the initial entry is the most recent entry in an initial subset of entries, and wherein each of the initial subset of entries is older than the target checkpoint and records a write range that overlaps at least partially with the target range; Determine, via the controller, whether a write range recorded in the first entry covers the entire destination range; In response to a finding that the write area recorded in the first entry does not cover the entire destination area, the controller determines a remaining portion of the destination area that excludes the write area recorded in the first entry; and Executing a second query by the controller to retrieve a second entry from the metadata table, where the second entry is a most recent record in a second subset of entries that are older than the target checkpoint and record write ranges that overlap at least some of the remaining portion of the target range. [16] The method of claim 15, comprising: Determine whether a write area recorded in the second entry covers the entire remaining part of the destination area; In response to a finding that the write area recorded in the second entry covers the entire remaining portion of the destination area, an output is generated using combined results from the first query and the second query. [17] The method of claim 16, comprising: Reset, using the output, the destination area to a state that existed at the time of the destination checkpoint. [18] The method of claim 15, comprising: Detecting a write operation of a data unit to a data carrier; in response to the detection of the writing process: Storing a copy of the data unit in a data journal; Creating a new entry in the metadata table; and Populating the new entry with a reference to the copy of the data unit stored in the data journal. [19] Method according to claim 18, comprising, in response to the detection of the writing process: Populating the new entry with the location of the data unit stored in the storage medium; and Populating the new entry with the identifier of a checkpoint that was last performed before the write operation was detected. [20] Method according to claim 18, wherein each entry of the metadata table includes metadata for a different write operation on the storage medium.

Images