Metadata processing method and equipment of storage engine, storage medium and program product

Abstract

The embodiment of the invention provides a metadata processing method and device for a storage engine, a storage medium and a program product, and the method comprises the steps: creating a snapshot for each unpersistent metadata block in a memory used by the storage engine, and persistently storing the snapshots in a metadata storage partition of a nonvolatile storage medium in sequence. Snapshot index information of the snapshots in the metadata storage partition is recorded in a memory; and if the unpersistent snapshot index information in the memory reaches the preset data volume, persistently storing the unpersistent snapshot index information in the metadata storage partition by a snapshot index information data block, and adding a pointer of the previous snapshot index information data block in the snapshot index information data block. According to the embodiment of the invention, by performing block persistence on the unpersistent metadata in the memory and performing block persistence on the unpersistent snapshot index information, the situation that the normal use of a storage engine by a user is influenced due to the fact that more resources are occupied by centralized persistence is avoided, and the metadata recovery efficiency can be improved.

Description

Technical Field

[0001] Embodiments of the present disclosure relate to the field of computer technologies, and in particular, to a method, device, storage medium, and program product for processing metadata of a storage engine. Background Art

[0002] When a user writes data in a storage engine, the metadata of the data is usually cached in the memory used by the storage engine, and then persistently stored in a non-volatile storage medium in a centralized manner, so as to be able to restore the persistently stored metadata to the memory after the device is restarted abnormally, and support the storage engine to continue to provide services.

[0003] However, in the prior art, the centralized persistent storage of metadata usually adopts timed full-volume persistent storage, which occupies more resources and is time-consuming, affecting the normal service of the storage engine. Especially for a storage engine based on ZNS (Zoned Namespaces), the amount of its metadata is larger and the types of metadata are more. The centralized persistent storage occupies more resources and takes longer time, which will seriously affect the normal service of the storage engine. Summary of the Invention

[0004] Embodiments of the present disclosure provide a method, device, storage medium, and program product for processing metadata of a storage engine, so as to avoid the centralized persistence of metadata occupying more resources and affecting the normal service of the storage engine.

[0005] In a first aspect, an embodiment of the present disclosure provides a method for processing metadata of a storage engine, including:

[0006] Create a snapshot for each unpersisted metadata block in the memory used by the storage engine, sequentially persistently store the snapshots in the metadata storage partition of the non-volatile storage medium, and record the snapshot index information of each snapshot in the metadata storage partition in the memory; where each metadata block includes the metadata of a data block managed by the storage engine;

[0007] If the unpersisted snapshot index information in the memory reaches a preset data volume, persistently store the unpersisted snapshot index information in a snapshot index information data block in the metadata storage partition, and add a pointer to the previous snapshot index information data block to the snapshot index information data block.

[0008] In a second aspect, an embodiment of the present disclosure provides a device for processing metadata of a storage engine, including:

[0009] A metadata persistence unit, configured to create a snapshot for each unpersisted metadata block in the memory used by the storage engine, persistently store each snapshot in sequence into a metadata storage partition of a non-volatile storage medium, and record the snapshot index information of each snapshot in the metadata storage partition in the memory; wherein each metadata block includes the metadata of a data block managed by the storage engine;

[0010] An indexing unit, configured to, if the unpersisted snapshot index information in the memory reaches a preset data volume, persistently store the unpersisted snapshot index information into the metadata storage partition as a snapshot index information data block, and add a pointer to the previous snapshot index information data block to the snapshot index information data block.

[0011] In a third aspect, an embodiment of the present disclosure provides an electronic device, including: a processor and a memory;

[0012] The memory stores computer-executable instructions;

[0013] The processor executes the computer-executable instructions stored in the memory, so that the at least one processor executes the metadata processing method of the storage engine as described in the first aspect and various possible designs of the first aspect above.

[0014] In a fourth aspect, an embodiment of the present disclosure provides a computer-readable storage medium, in which computer-executable instructions are stored, and when the processor executes the computer-executable instructions, the metadata processing method of the storage engine as described in the first aspect and various possible designs of the first aspect above is implemented.

[0015] In a fifth aspect, an embodiment of the present disclosure provides a computer program product, including a computer program, and when the computer program is executed by a processor, the metadata processing method of the storage engine as described in the first aspect and various possible designs of the first aspect above is implemented.

[0016] The metadata processing method, device, storage medium, and program product of the storage engine provided by the embodiments of the present disclosure create snapshots for each unpersisted metadata block in the memory used by the storage engine, persistently store the snapshots in sequence in the metadata storage partition of the non-volatile storage medium, and record the snapshot index information of each snapshot in the metadata storage partition in the memory; where each metadata block includes the metadata of a data block managed by the storage engine; if the unpersisted snapshot index information in the memory reaches the preset data volume, the unpersisted snapshot index information is persistently stored in the metadata storage partition as a snapshot index information data block, and a pointer to the previous snapshot index information data block is added to the snapshot index information data block. The embodiments of the present disclosure can avoid the concentration of persistent storage from occupying too many resources and affecting the normal use of the storage engine by the user through the block-by-block persistence of unpersisted metadata in the memory and the block-by-block persistence of unpersisted snapshot index information; and can quickly find each snapshot based on the snapshot index information data block to restore it to the memory when restoring metadata, improving the metadata recovery efficiency, and at the same time avoiding the metadata from occupying too much storage space, ensuring the amount of storage space available for the user. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the drawings in the following description are some embodiments of the present disclosure. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0018] Figure 1 It is a scenario example diagram of the metadata processing method of the storage engine provided by an embodiment of the present disclosure;

[0019] Figure 2 It is a schematic flowchart of the metadata processing method of the storage engine provided by an embodiment of the present disclosure;

[0020] Figure 3 It is a schematic diagram of the metadata storage partition provided by an embodiment of the present disclosure;

[0021] Figure 4 It is a schematic flowchart of the metadata processing method of the storage engine provided by another embodiment of the present disclosure;

[0022] Figure 5 It is a schematic diagram of the data storage partition provided by an embodiment of the present disclosure;

[0023] Figure 6 It is a structural block diagram of the metadata processing device of the storage engine provided by an embodiment of the present disclosure;

[0024] Figure 7 Schematic diagram of the hardware structure of an electronic device provided by an embodiment of the present disclosure. Detailed implementation manners

[0025] To make the objectives, technical solutions and advantages of the embodiments of the present disclosure clearer, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present disclosure. Apparently, the described embodiments are some but not all of the embodiments of the present disclosure. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present disclosure without making creative efforts shall fall within the protection scope of the present disclosure.

[0026] First, the technical terms of the present disclosure are explained:

[0027] Zoned Namespace (ZNS) is a storage technology mainly used for Solid State Disk (SSD) management, aiming to improve storage efficiency and performance, especially in scenarios that require sequential write operations.

[0028] Non-volatile storage media, such as NAND flash memory, do not lose data when powered off and have very fast read and write speeds. They are the current mainstream enterprise-level and consumer-level storage NVME (Logical Device Interface Specification) SSDs.

[0029] Segment is the mapping of the basic management unit Zone (physical storage partition) of ZNS in the storage engine layer, that is, a logical storage partition. Each storage partition mentioned in the following embodiments of the present disclosure may be a logical storage partition and can be mapped to a physical storage partition according to the mapping relationship. Among them, the storage partition used to store metadata can be called the metadata storage partition; the storage partition used to store user data can be called the data storage partition.

[0030] Chunk is a file (data block) managed by the storage engine. The data in a data block can be written into one or more data storage partitions. If a data block is written into multiple data storage partitions, each data storage partition includes a part of the data of the data block.

[0031] Sector is the basic storage unit for read and write operations of ZNS devices.

[0032] Checkpoint, snapshot, is an operation to persist metadata, ensuring the rapid recovery of metadata after restart.

[0033] A globally atomic incrementing digital generator is used to distinguish the order of occurrence of atomic operations based on size.

[0034] Garbage Collection (GC), based on the characteristics of NAND, for a data block to be rewritten, it needs to be erased first and then written. Before erasing, the valid data on the data block needs to be migrated to other data blocks, the valid data is compressed and merged, and then erased.

[0035] In the prior art, when a user writes data in a storage engine, the metadata of the data is usually cached in the memory used by the storage engine and will be persistently stored in a non-volatile storage medium centrally later, so as to be able to restore the persistently stored metadata to the memory after the device restarts abnormally and support the storage engine to continue providing services.

[0036] However, in the prior art, the centralized persistent storage of metadata usually adopts timed full-volume persistent storage, which occupies more resources, repeatedly persists some metadata each time, and is time-consuming, affecting the normal service of the storage engine. Especially for a storage engine based on ZNS (Zoned Namespaces), as the storage device capacity density becomes larger and larger, a storage partition of ZNS will also become larger, the amount of its metadata is larger, and there are more types of metadata. The centralized persistent storage occupies more resources and takes longer time, which will seriously affect the normal service of the storage engine.

[0037] To solve the above technical problems, the embodiments of the present disclosure provide a method for processing metadata of a storage engine, which persistently stores the unpersisted metadata in the memory in blocks and the unpersisted snapshot index information in blocks, can avoid the centralized persistent storage from occupying more resources and affecting the normal use of the storage engine by users; and can quickly find each snapshot based on the snapshot index information data block to restore it to the memory when restoring metadata, improving the metadata restoration efficiency, and further improving the reliability and availability of the storage engine system.

[0038] The application scenario of the method for processing metadata of the storage engine in the embodiments of the present disclosure is as Figure 1As shown, a snapshot is created for each unpersisted metadata block in the memory used by the storage engine, and each snapshot is successively and persistently stored in the metadata storage partition of the non-volatile storage medium, and the snapshot index information of each snapshot in the metadata storage partition is recorded in the memory; each metadata block includes the metadata of a data block managed by the storage engine; if the unpersisted snapshot index information in the memory reaches a preset data volume, the unpersisted snapshot index information is persistently stored in the metadata storage partition as a snapshot index information data block, and a pointer to the previous snapshot index information data block is added to the snapshot index information data block, and any snapshot index information data block is used to find the corresponding snapshots from the metadata storage partition according to the snapshot index information in any snapshot index information data block and restore them to the memory, and find the previous snapshot index information data block through the pointer to the previous snapshot index information data block.

[0039] It should be noted that the user information and data involved in the present disclosure (including but not limited to data for analysis, stored data, displayed data, etc.) are all information and data authorized by the user or fully authorized by all parties, and the collection, use, and processing of relevant data need to comply with the relevant laws, regulations, and standards of relevant countries and regions, and corresponding operation entrances are provided for users to choose to authorize or refuse.

[0040] The metadata processing method of the storage engine of the present disclosure will be introduced in detail below in combination with specific embodiments.

[0041] Refer to Figure 2 , Figure 2 which is a schematic flowchart of the metadata processing method of the storage engine provided by an embodiment of the present disclosure. The method of this embodiment can be applied to a terminal device or a server, and the metadata processing method of the storage engine includes:

[0042] S201. Create a snapshot for each unpersisted metadata block in the memory used by the storage engine, successively and persistently store each snapshot in the metadata storage partition of the non-volatile storage medium, and record the snapshot index information of each snapshot in the metadata storage partition in the memory; each metadata block includes the metadata of a data block managed by the storage engine.

[0043] In this embodiment, when the storage engine stores data, it stores the metadata in memory and then persistently stores it in a non-volatile storage medium in a centralized manner according to certain rules; when the ZNS-based storage engine stores data, since ZNS opens the management permission of the storage medium and transfers part of the management responsibility to the storage engine host side, a large amount of metadata will exist in the memory used by the ZNS storage engine, and the types of metadata are also relatively numerous. If centralized persistent storage is adopted in the non-volatile storage medium, it will occupy more resources and affect the normal use of the storage engine by users; therefore, in this embodiment, for the metadata in the memory used by the storage engine, snapshots (checkpoints) are created in blocks and sequentially persistently stored in the metadata storage partition (meta segment) of the non-volatile storage medium. Specifically, since the storage engine can manage data in terms of data chunks (which can also be called files), when persisting metadata, the metadata in memory can be chunked according to data chunks, that is, one data chunk corresponds to one metadata chunk, and the metadata chunk is persistently stored in the metadata storage partition of the non-volatile storage medium. In this way, the centralized persistence of metadata in memory can be avoided, the impact of the persistence process on the user's use of the storage engine can be avoided, and at the same time, the excessive occupation of storage space by metadata can be avoided, ensuring the amount of storage space available for users.

[0044] Considering that the metadata storage partition is usually relatively large and the number of simultaneously active metadata storage partitions is limited, a large number of metadata chunks and various types of metadata chunks can be stored in the metadata storage partition. Therefore, the snapshot index information of each snapshot in the metadata storage partition is required to quickly find each snapshot. Therefore, when each snapshot is persistently stored in the metadata storage partition, the snapshot index information of the snapshot in the metadata storage partition can be recorded in memory for subsequent metadata recovery. The snapshot index information can be restored in memory first, and the snapshot can be found according to the snapshot index information to accelerate the process of restoring metadata according to the snapshot.

[0045] S202. If the amount of unpersisted snapshot index information in the memory reaches the preset data volume, the unpersisted snapshot index information is persistently stored in the metadata storage partition as a snapshot index information data chunk, and a pointer to the previous snapshot index information data chunk is added to the snapshot index information data chunk.

[0046] In this embodiment, since the snapshot index information of each snapshot in the metadata storage partition is used to quickly find and load the snapshot during metadata recovery, it also needs to be persistently stored to avoid being lost due to abnormal restart only stored in memory. Since the metadata storage partition is relatively large and the number of snapshots may also be relatively large, the data volume of the snapshot index information is also very large. For example:

[0047] The larger the data block written by the user, the fewer the data index quantities, the less the metadata of the data block, and the less space required for the created snapshot. Taking the size of the data block chunk written by the user as 128k as an example, the snapshot data volume of a single chunk metadata is approximately: more than one hundred bytes (basic chunk metadata) + 64M / 128k * 8 (chunk data index) = 4096 + x bytes, and approximately 8k storage space is required. Roughly estimating that when the metadata storage partition has a capacity of 10G, the maximum number of snapshots that can be accommodated is: 10G * 1024 * 1024 / 8k = 1310720; and the size of a snapshot index information is 20 bytes, so the total amount of snapshot index information of all snapshots in the entire metadata storage partition is: 20 * 1310720 / 1024 / 1024 = 25MB.

[0048] If the snapshot index information is persistently stored in full at one time, it will also occupy a large amount of resources, affect the normal use of the storage engine by the user, and is also time-consuming when loading. If an abnormal restart occurs before the persistent storage is completed, it will be lost. Therefore, in this embodiment, the snapshot index information in the memory is also stored in a block-by-block persistent manner. The data volume of each snapshot index information data block is a preset data volume, that is, when the unpersisted snapshot index information in the memory reaches the preset data volume, it is persistently stored in the metadata storage partition as a snapshot index information data block. That is, the snapshot index information data block is stored in the metadata storage partition mixed with the snapshots. Its manifestation is that in the metadata storage partition, there is a snapshot index information data block behind some snapshots, and then there are some snapshots behind this snapshot index information data block, and there is a snapshot index information data block behind these snapshots, and so on.

[0049] A pointer to the previous snapshot index information data block can be added to each snapshot index information data block, and the respective snapshot index information data blocks are connected through the pointer. In this way, when restoring the metadata, the last snapshot index information data block can be first found from the metadata storage partition, and then the previous snapshot index information data block can be found from the metadata storage partition according to the pointer therein, and so on. Each snapshot index information data block can be quickly found from the metadata storage partition, and the corresponding snapshots can be found from the metadata storage partition based on the snapshot index information in the snapshot index information data block and restored to the memory.

[0050] Further, when the metadata storage partition is full, a pointer to the last snapshot index information data block is added to the tail information (SegmentFooter) of the metadata storage partition for finding the last snapshot index information data block through the pointer to the last snapshot index information data block. In specific implementation, when restoring metadata, it can be directly determined whether there is a pointer to the last snapshot index information data block according to the tail information of the metadata storage partition. If there is a pointer to the last snapshot index information data block, it indicates that the metadata storage partition is full. At this time, the last snapshot index information data block can be directly found according to the pointer to the last snapshot index information data block, and then the previous snapshot index information data block can be found according to the pointer included in the last snapshot index information data block, and so on, so that each snapshot index information data block can be quickly found from the metadata storage partition without additionally searching for the last snapshot index information data block from the metadata storage partition.

[0051] As an example, the structure of the metadata storage partition (MetaSegment) can be as Figure 3 shown, which includes the header information (SegmentHeader) of the metadata storage partition, each snapshot, various other types of metadata, each snapshot index information data block, and the tail information (SegmentFooter), and may also include a padding part (zero padding), where each snapshot index information data block is not concentrated together but is mixed with each snapshot and various other types of metadata. The tail information (SegmentFooter) of the metadata storage partition includes information such as the status, structure, and attributes of the metadata storage partition, and may also include a padding part, and if it is full, it will also include a pointer to the last snapshot index information data block. Of course, the snapshot index information data block also includes header information and tail information, but the content is different from that of the snapshot and other types of metadata, and can be used to distinguish the snapshot index information data block from the snapshot and other types of metadata.

[0052] Each snapshot, as well as other types of metadata, at least includes a metadata payload and a record footer, and may also include zero padding. The record footer records the index information of the current snapshot or the current metadata, including but not limited to location, length, type, etc., for use in the scanning process during subsequent metadata recovery. It should be noted that although the record footer is also index information, it is only the index information of the current snapshot or the current metadata, and is persisted together with the current snapshot or the current metadata. During recovery, the index information needs to be scanned one by one to obtain it. The snapshot index information data block, on the other hand, includes the snapshot index information of multiple snapshots and is batch-persisted. During recovery, the snapshot index information of multiple snapshots can be obtained at once without having to scan each snapshot individually.

[0053] The metadata processing method of the storage engine provided in this embodiment creates snapshots for each unpersisted metadata block in the memory used by the storage engine, persistently stores each snapshot in sequence in the metadata storage partition of the non-volatile storage medium, and records the snapshot index information of each snapshot in the metadata storage partition in the memory; each metadata block includes the metadata of a data block managed by the storage engine; if the unpersisted snapshot index information in the memory reaches a preset data volume, the unpersisted snapshot index information is persistently stored in the metadata storage partition as a snapshot index information data block, and a pointer to the previous snapshot index information data block is added to the snapshot index information data block. By persistently storing the unpersisted metadata in blocks and the unpersisted snapshot index information in blocks, this embodiment can avoid the situation where centralized persistence occupies too many resources and affects the normal use of the storage engine by users; and during metadata recovery, it can quickly find each snapshot based on the snapshot index information data block and restore it to the memory, improving the metadata recovery efficiency, while also avoiding the metadata occupying too much storage space and ensuring the amount of storage space available for users.

[0054] Based on any of the above embodiments, if the device where the storage engine is located experiences an abnormal restart, the metadata in the memory used by the storage engine is lost. Therefore, the metadata can be restored according to the snapshots in the metadata storage partition. The specific metadata recovery process can be as Figure 4 shown, including:

[0055] S301. When the instruction to restore metadata is triggered, find the last snapshot index information data block from the [due];

[0056] S302. Find the corresponding snapshots from the metadata storage partition according to the snapshot index information in the last snapshot index information data block, and load each snapshot into the memory to restore the metadata blocks corresponding to the snapshots into the memory;

[0057] S303. Find the previous snapshot index information data block according to the pointer to the previous snapshot index information data block included in the last snapshot index information data block.

[0058] In this embodiment, an instruction to restore metadata can be triggered after an abnormal restart, or an instruction to restore metadata can also be triggered in other cases where metadata needs to be restored. When the instruction to restore metadata is triggered, the metadata used by the storage engine is restored to the memory. Specifically, since the snapshots corresponding to the metadata blocks are persistently stored in the metadata storage partition of the non-volatile storage medium, if there is no snapshot index information data block, these snapshots need to be found in the metadata storage partition and then loaded into the memory; in this embodiment, since the snapshot index information data block is also persistently stored in the metadata storage partition, these snapshots can be found more quickly according to the snapshot index information data block. Therefore, it is necessary to first find the snapshot index information data block from the metadata storage partition. Since each snapshot index information data block includes a pointer to the previous snapshot index information data block, only the last snapshot index information data block needs to be found from the metadata storage partition, and then the previous snapshot index information data block can be found according to the pointer in the last snapshot index information data block. Further, according to the pointer in the previous snapshot index information data block, continue to search for the snapshot index information data block forward, and all the snapshot index information data blocks in the metadata storage partition can be found.

[0059] Each snapshot index information data block includes the snapshot index information of some snapshots in the metadata storage partition. Therefore, each snapshot can be found according to the snapshot index information, and then each snapshot can be loaded into the memory used by the storage engine, so as to restore the metadata blocks corresponding to the snapshots into the memory.

[0060] Optionally, when finding the last snapshot index information data block from the metadata storage partition in S301, it may include:

[0061] If there is a pointer to the last snapshot index information data block in the tail information of the metadata storage partition, find the last snapshot index information data block according to the pointer to the last snapshot index information data block; or

[0062] If there is no pointer to the last snapshot index information data block in the tail information of the metadata storage partition, scan the metadata storage partition from the back to the front to find the last snapshot index information data block.

[0063] In this embodiment, when the metadata storage partition is full, a pointer to the data block of the last snapshot index information is added to the tail information (SegmentFooter) of the metadata storage partition. Therefore, when looking for the data block of the last snapshot index information, it can first be determined whether there is a pointer to the data block of the last snapshot index information in the tail information of the metadata storage partition. If it exists, it means that the metadata storage partition is full, and at this time, the data block of the last snapshot index information can be directly found according to the pointer to the data block of the last snapshot index information; if it does not exist, it means that the metadata storage partition is not full. At this time, the metadata storage partition can be scanned from the back to the front, and it can be determined whether it is a data block of snapshot index information through the tail information of the scanned object until the data block of snapshot index information is found, that is, the data block of the last snapshot index information in the metadata storage partition.

[0064] Based on the above embodiment, since the snapshot index information data block persistence has a time interval, it is necessary to wait until the unpersisted snapshot index information in the memory reaches the preset data volume before a snapshot index information data block can be persistently stored. Therefore, if an abnormal restart occurs when the unpersisted snapshot index information in the memory does not reach the preset data volume, the unpersisted snapshot index information in the memory will be lost. Therefore, in fact, there may still be some snapshots after the data block of the last snapshot index information in the metadata storage partition, but the snapshot index information has not had time to be persisted. However, in the above embodiment, only the snapshot is found based on the snapshot index information data block, and the snapshots after the data block of the last snapshot index information will be missed. Therefore, before looking for the data block of the last snapshot index information, the scanned snapshots can be loaded into the memory, that is, the snapshots after the data block of the last snapshot index information are also loaded into the memory to ensure the integrity of the metadata. Of course, if there are no snapshots after the data block of the last snapshot index information, this process does not need to be executed.

[0065] In addition, in the case where there is a pointer to the data block of the last snapshot index information in the tail information of the metadata storage partition, there may or may not be snapshots after the data block of the last snapshot index information. If there are, the snapshots after the data block of the last snapshot index information are also loaded into the memory.

[0066] Based on any of the above embodiments, when loading each snapshot into memory, considering that the same data block (chunk) in the storage engine may be operated on multiple times, so there are multiple metadata blocks for this data block, that is, corresponding to multiple snapshots. Therefore, if multiple snapshots related to the same data block in the storage engine are found, the latest snapshot among the multiple snapshots related to the same data block is loaded into memory, and other snapshots among the multiple snapshots related to the same data block except the latest snapshot are ignored, to avoid the subsequent operations being overwritten by the previous operations. Specifically, the order of snapshot times can be compared according to the creation time information of each snapshot or the atomic operation count value, where the atomic operation count value can be generated by a globally atomically incrementing digital generator and is used to record the order of atomic operations.

[0067] Based on any of the above embodiments, considering that it also takes a certain amount of time to create snapshots and persistently store the metadata blocks, so there may still be a small part of the metadata blocks that have not had time to complete snapshot creation and persistent storage during an abnormal restart, and this part of the metadata in memory will be lost. Considering that the operations on the data blocks in the storage engine are recorded in the log of the storage engine, and the log will not be lost due to an abnormal restart, some metadata blocks that have not had time to complete snapshot creation and persistent storage can also be restored based on the log of the storage engine. Specifically, it can be as follows:

[0068] Obtain a first set of log data after the creation time of the last snapshot in the metadata storage partition from the log of the storage engine;

[0069] Replay the log data in the first set of log data in chronological order, and generate metadata after the creation time of the last snapshot in the memory.

[0070] In this embodiment, the log data after the creation time of the last snapshot in the metadata storage partition can be obtained from the log of the storage engine as the first set of log data. Specifically, the first set of log data after the creation time of the last snapshot can be found according to the timestamp or the atomic operation count value; further, the log data in the first set of log data can be replayed in chronological order (the order of timestamps or the atomic operation count value from small to large). During the replay process, some metadata will be generated in the memory, that is, the metadata after the creation time of the last snapshot, which can supplement the metadata for snapshot recovery and improve the integrity of the metadata.

[0071] Based on the above embodiments, since it is impossible for every piece of written data to trigger the creation of a snapshot or record a log data entry, otherwise the amount of metadata and log data would be too large, and it would affect the write performance of the storage engine. Therefore, based on the above snapshots and log data, there is still a part of the metadata that cannot be recovered, that is, the metadata corresponding to the data written after the creation time of the last snapshot and not recorded in the log data. The metadata corresponding to this data is also lost after an abnormal restart. To recover the metadata of this part of the data, in this embodiment, in the data storage partition of the storage engine, the data written after the creation time of the last snapshot is scanned, and the data written after the creation time of the last snapshot is generated into the corresponding metadata in memory, thus ensuring the integrity of the metadata in memory.

[0072] Based on any of the above embodiments, in addition to the metadata corresponding to the data blocks (chunks) in the memory used by the storage engine, there are also other types of metadata, such as the metadata of the data storage partition of the storage engine (SegmentMeta), which records information such as the status, usage amount, and garbage amount of the data storage partition. Among them, the status of the data storage partition includes a full state, an empty state, or a partially used state. When recovering the data index, it is determined whether the data storage partition is empty. If it is empty, there is no need to recover the data index corresponding to this data storage partition.

[0073] Since the amount of metadata of the data storage partition of the storage engine is not large, it can be persistently stored in a non-volatile storage medium during the persistence process. Specifically, during implementation, the metadata of the data storage partition of the storage engine can be persistently stored in a non-volatile storage medium in full at regular intervals.

[0074] Among them, considering that the metadata of the data storage partition is persistently stored in a non-volatile storage medium at regular intervals, and there is a certain time interval between the abnormal restart time and the persistent storage time of the metadata of the data storage partition, which may cause the metadata of the data storage partition to change during this time interval. Therefore, when restoring the metadata of the data storage partition to memory, after obtaining the metadata of the data storage partition from the non-volatile storage medium, the log data in the storage engine after the persistent storage time of the metadata of the data storage partition can also be combined to determine the changes in the metadata of the data storage partition during this time interval, and the metadata of the data storage partition can be completely restored.

[0075] Based on any of the above embodiments, the memory used by the storage engine further includes two other types of metadata. One is the metadata of the mapping relationship between the internal identifier (Inode Id) of the data block in the storage engine and the external identifier (Chunk Id) of the data block. The external identifier is the external name of the data block and has more physical significance. The internal identifier is the mapping of the storage engine to the external identifier, and both metadata management and restart use the internal identifier. The other is the basic metadata information of the data block (chunk), which records information such as the logical length and physical length of the data block. The basic metadata of a data block is approximately dozens of K or more than one hundred K. It should be noted that if a data block fails or is deleted, then the corresponding mapping relationship in the mapping relationship must not exist. Therefore, it is possible to determine whether a data block is a valid data block based on the mapping relationship.

[0076] Similarly, since the data volume is not large, during the persistence process, the metadata of the mapping relationship can be fully persistently stored in a non-volatile storage medium, and the basic metadata of the data block can also be fully persistently stored in a non-volatile storage medium. Specifically, it can also be stored in the metadata storage partition.

[0077] Correspondingly, the finding of the corresponding snapshots from the metadata storage partition according to the snapshot index information in the data block of the last snapshot index information and the loading of each snapshot in the memory includes:

[0078] Restoring the mapping relationship between the internal identifier of the data block in the storage engine and the external identifier of the data block to the memory;

[0079] Finding the corresponding snapshots from the metadata storage partition according to the snapshot index information in the data block of the last snapshot index information, screening valid snapshots according to the mapping relationship, and loading each valid snapshot in the memory.

[0080] In this embodiment, the above mapping relationship can be restored to the memory first. Then, after finding each snapshot, valid snapshots can be screened according to the mapping relationship, and the snapshots corresponding to the failed or deleted data blocks can be screened out. Thus, only valid snapshots need to be loaded, avoiding the waste of resources caused by the loading of invalid snapshots and improving the metadata recovery speed.

[0081] Among them, considering that the mapping relationship is persistently stored in a non-volatile storage medium at regular intervals, and there is a certain time interval between the abnormal restart time and the persistent storage time of the mapping relationship, which may cause the mapping relationship to change during this time interval. Therefore, when restoring the mapping relationship to memory, after obtaining the mapping relationship from the non-volatile storage medium, the log data in the storage engine's log after the persistent storage time of the mapping relationship can be combined to determine the data blocks that have become invalid or deleted during this time interval, and delete the mapping relationships of these data blocks, and / or determine the newly created data blocks during this time interval, and add the mapping relationships of the newly created data blocks, so as to achieve the complete restoration of the mapping relationship, and thus also ensure the accuracy of the effective snapshot judgment.

[0082] Based on any of the above embodiments, as an example, this embodiment provides a metadata recovery method, which may specifically include:

[0083] 1) Search for the metadata of the data storage partition of the latest storage engine, the metadata of the mapping relationship between the internal identifier of the data block in the storage engine and the external identifier of the data block, and the basic metadata information of the data block from the non-volatile storage medium;

[0084] Optionally, various types of metadata can be further parsed into the corresponding metadata management modules;

[0085] 2) Obtain the log data of the storage engine and parse the log data and save it in memory;

[0086] 3) Update the metadata of the data storage partition of the storage engine and the metadata of the mapping relationship according to the log data;

[0087] Furthermore, the basic metadata information of the data block can also be updated according to the updated mapping relationship, that is, the invalid data blocks are determined according to the updated mapping relationship, and the basic metadata information of the invalid data blocks is deleted;

[0088] 4) Judge whether the data storage partition is empty according to the status information in the metadata of the data storage partition of the storage engine;

[0089] If it is not empty, search for the last snapshot index information data block associated with the data storage partition from the metadata storage partition;

[0090] According to the pointer of the previous snapshot index information data block included in the last snapshot index information data block, search for the previous snapshot index information data block, and so on until all the snapshot index information data blocks are found;

[0091] Find the corresponding snapshots in the metadata storage partition according to the snapshot index information in each snapshot index information data block, filter the valid snapshots according to the mapping relationship, and load each valid snapshot into the memory to restore the metadata blocks corresponding to each valid snapshot to the memory;

[0092] For the snapshots after the last snapshot index information data block in the metadata storage partition, they are also loaded into the memory after filtering the valid snapshots;

[0093] 5) Obtain the first set of log data after the creation time of the last snapshot in the metadata storage partition from the log of the storage engine; replay the log data in the first set of log data in chronological order, and generate the metadata after the creation time of the last snapshot in the memory.

[0094] 6) Scan the data written after the creation time of the last snapshot in the data storage partition of the storage engine, and generate the metadata corresponding to the data written after the creation time of the last snapshot in the memory.

[0095] Through the above metadata recovery process, the integrity of metadata recovery can be improved as much as possible.

[0096] Based on any of the above embodiments, for an exit scenario of the storage engine, if there is enough time for persistent storage of metadata (i.e., graceful exit), all unpersisted metadata blocks in the memory can be persistently stored at once and then exit. The specific process is as follows:

[0097] In response to the storage engine exit instruction, create snapshots for all unpersisted metadata blocks in the memory, persistently store each snapshot into the metadata storage partition in sequence, and add a first identifier after the last snapshot and then execute the storage engine exit instruction, where the first identifier is used to indicate that the storage engine exits normally;

[0098] When the instruction to recover metadata is triggered, scan the metadata storage partition from the back to the front. If the first identifier is detected, load the snapshots in the metadata storage partition into the memory to restore the full amount of metadata in the memory; add a second identifier after the first identifier, where the second identifier is used to indicate that the first identifier is invalid.

[0099] In this embodiment, in the case of graceful exit, snapshots can be created for all unpersisted metadata blocks in memory and all of them can be persistently stored in the metadata storage partition. Of course, the unpersisted snapshot index information in memory will also be persistently stored in the metadata storage partition. To facilitate quick metadata recovery when restarting again, a first identifier can be added to the storage space after the last snapshot in the metadata storage partition to indicate that the storage engine has exited normally (graceful exit). Therefore, when the instruction to recover metadata is triggered, the metadata storage partition is scanned from the back to the front. If the first identifier is scanned, the snapshots in the metadata storage partition can be loaded into memory. The search and recovery of snapshots are the same as those in the above embodiments and will not be elaborated here. By the method of this embodiment, there is no need to execute the process of replaying log data and scanning the data in the data storage partition of the storage engine to recover some metadata, making the metadata recovery process simpler and more efficient.

[0100] However, it should be noted that the first identifier needs to be invalidated immediately after metadata recovery. Otherwise, after the next abnormal exit (non-graceful exit), if the metadata storage partition is scanned from the back to the front and the first identifier is scanned again, a wrong judgment will be made and the process of replaying log data and scanning the data in the data storage partition of the storage engine to recover some metadata will not be executed, resulting in the loss of some metadata. In this embodiment, a second identifier can be written after the first identifier. The second identifier is used to indicate that the first identifier is invalid. In this way, when the metadata storage partition is scanned from the back to the front, if the second identifier is scanned first, it means that the first identifier has become invalid and the metadata needs to be recovered according to the normal process, including recovering metadata based on snapshots, replaying log data, and scanning the data in the data storage partition of the storage engine to recover some metadata; if the first identifier is scanned first, it means a graceful exit, and only the metadata needs to be recovered based on snapshots, without executing the process of replaying log data and scanning the data in the data storage partition of the storage engine to recover some metadata.

[0101] In addition, during garbage collection, when relocating metadata, a second identifier is first written in the metadata storage partition. Whether there is a first identifier in front of it or not, this second identifier can be used to tell the storage engine system that other operations have been performed after restart and all the previous first identifiers have become invalid.

[0102] Based on any of the above embodiments, one of the most important tasks in restoring metadata in memory is to restore the data index, where the data index is the location information of each piece of data in a data chunk (chunk); since it is possible that no data was written to the storage engine after the creation time of the last snapshot before the abnormal restart, the data index information in the snapshot is complete; if data was continuously written to the storage engine, the data index information in the snapshot is incomplete, and the data index information of the data written after the creation time of the last snapshot will be lost during the abnormal restart.

[0103] Each snapshot includes the data index information of the corresponding data chunk, denoted as the first data index information; and the data index information of the data written to the storage engine after the creation time of the last snapshot in the metadata storage partition, denoted as the second data index information.

[0104] It is possible to determine whether data was written to the storage engine after the creation time of the last snapshot, and then select different ways to load the data index information. Specifically, it can be as follows:

[0105] Compare the write time of each data chunk in the data storage partition of the storage engine with the creation time of its corresponding snapshot respectively;

[0106] If the write time of all data chunks is less than the creation time of the corresponding snapshot, then load the first data index information in the snapshot into the memory; or

[0107] If the write time of at least one data chunk is not less than the creation time of the corresponding snapshot, then load the first data index information in the snapshot and the second data index information of the data written to the storage engine after the creation time of the last snapshot in the metadata storage partition into the memory.

[0108] In this embodiment, compare the write time of each data chunk in the data storage partition of the storage engine with the creation time of its corresponding snapshot respectively. If the write time of all data chunks is less than the creation time of the corresponding snapshot, it means that no data was written to the storage engine after the creation time of the last snapshot, and the first data index information in the snapshot is the full amount of data index information, and only the first data index information in the snapshot needs to be loaded into the memory; if the write time of at least one data chunk is not less than the creation time of the corresponding snapshot, it means that data was written to the storage engine after the creation time of the last snapshot, and the first data index information in the snapshot is incomplete. The first data index information in the snapshot and the second data index information of the data written to the storage engine after the creation time of the last snapshot in the metadata storage partition can be loaded into the memory to complete the loading of the full amount of data index information.

[0109] Based on the above embodiments, considering that loading the first data index information in each snapshot into the memory is very time-consuming, resulting in the storage engine being unable to provide services for a long time after restarting, which affects the service quality of the storage engine, while obtaining the second data index information takes less time. Therefore, in this embodiment, the first data index information in the snapshot can be loaded on demand, and the specific process is as follows:

[0110] Scan the data storage partition of the storage engine to obtain the second data index information, and load the second data index information into the memory;

[0111] When receiving any operation instruction for any target data in the data storage partition, if the data index information of the target data does not exist in the memory, determine the target snapshot where the data index information of the target data is located, and load the first data index information in the target snapshot into the memory and fuse it with the second data index information.

[0112] In this embodiment, the second data index information obtained by scanning can be first loaded into the memory. For example, it can be temporarily saved in the memory data structure. After the storage engine provides services, for any operation instruction for any target data in the data storage partition of the storage engine, including but not limited to read, write, delete, freeze, invalidate and other operation instructions, search for the data index information of the target data in the memory. If the data index information of the target data is not found in the memory, determine the target snapshot where the data index information of the target data is located, that is, determine the target data block where the target data is located, and then determine the snapshot of the metadata block corresponding to the data block. Load the first data index information in the target snapshot into the memory and fuse it with the second data index information to obtain the complete data index information of the target data block. Further, the data index information of the target data can be found, and then the operation instruction can be executed on the target data.

[0113] In another alternative embodiment, scan the data storage partition of the storage engine to obtain the second data index information, and load the second data index information into the memory; when receiving any operation instruction for any target data block in the data storage partition, including but not limited to read, write, delete, freeze, invalidate and other operation instructions for the target data block, load the first data index information in the target snapshot of the metadata block corresponding to the target data block into the memory and fuse it with the second data index information to obtain the complete data index information of the target data block. Further, the operation instruction can be executed on the target data block.

[0114] Based on the above embodiments, in the data storage partition of the storage engine, in addition to including the data of the storage engine (i.e., user data), it also includes the data index information of the data in the data storage partition for the storage engine to quickly restore the data index information after restart. However, since the data storage partition may be very large while the user-written data is relatively small, there will be a large amount of data index information, which may reach dozens of MB or even hundreds of MB. To accelerate the quick restoration of the data index information after restart and also to reduce the data storage partition switching jitter caused by the one-time persistence of the data index information in the data storage partition, in this embodiment, the data index information is persistently stored in blocks, that is, when the unpersisted data index information reaches the second preset data volume, a persistent storage of the data index information will be performed, that is, the unpersisted data index information will be persistently stored in the data storage partition as a data index information data block. The position of the data index information data block is not fixed and is scattered in the data storage partition and mixed in the data of the data storage partition. A pointer to the previous data index information data block can be added to each data index information data block, and all data index information data blocks can be linked through the pointer. As long as any data index information data block is found, all the data index information data blocks before this data index information data block can be sequentially found, and then the data index information included in each data index information data block can be obtained. The position of the data index information data block is as Figure 5 shown.

[0115] Therefore, when scanning the data storage partition of the storage engine to obtain the second data index information in the above embodiments, the second data index information after the creation time of the last snapshot can be found through the data index information data block in the data storage partition.

[0116] When searching for the data index information data block in the data storage partition, generally, the last data index information data block can be searched first, such as Figure 5The data index information data block n in it, and then sequentially look up the remaining data index information data blocks forward according to the pointer. However, considering that the data storage partition supports concurrent write requests, when processing concurrent write requests, the address for data writing will be allocated first and then the data will be written. Therefore, it is possible to obtain the data index information first, while the data has not been completely written. As a result, when restarting abnormally, the data index information data block may have been persistently stored in the data storage partition, but the data has not been completely written. And when each data index information data block is persistently stored, it is necessary to ensure that the previous data index information data block and its related data have been persistently stored before continuing to persistently store. Therefore, the last data index information data block in the data storage partition is not necessarily valid, but the data index information data blocks before the last data index information data block are valid, that is, the penultimate data index information data block n-1 and the data index information data blocks n-2, n-3... before it are valid. Therefore, the data index information in the last data index information data block can be discarded, and the data index information in the penultimate data index information data block and the data index information data blocks before it can be used. For the data after the penultimate data index information data block, the data index information can be generated by scanning.

[0117] Specifically, scanning the data storage partition of the storage engine to obtain the second data index information and loading the second data index information into the memory may include:

[0118] Scan the data storage partition from the back to the front until the penultimate data index information data block in the data storage partition is scanned, and sequentially look up the remaining data index information data blocks in the data storage partition forward according to the pointer of the previous data index information data block;

[0119] Obtain the data index information for the data after the penultimate data index information data block;

[0120] From the data index information of the data after the penultimate data index information data block, and the data index information included in the penultimate data index information data block and the remaining data index information data blocks, screen out the data index information of the data written into the data storage partition after the creation time of the last snapshot as the second data index information and load it into the memory.

[0121] In this embodiment, the data storage partition can be scanned from the back to the front, ignoring the last data index information data block in the data storage partition (i.e., the first data index information data block scanned), until the penultimate data index information data block in the target storage area is scanned (i.e., the second data index information data block scanned). According to the pointer of the previous data index information data block in the penultimate data index information data block, the previous data index information data block is searched, and the data index information data block is continuously searched forward according to the pointer in the previous data index information data block until the remaining data index information data blocks are found. For the index information of the data after the penultimate data index information data block, it can be obtained by scanning.

[0122] In this way, the index information of the data written into the data storage partition after the creation time of the last snapshot can be filtered out from the index information of the data after the penultimate data index information data block, the penultimate data index information data block, and the index information of the remaining data index information data blocks before the penultimate data index information data block, and used as the second index information, which can be loaded into the memory.

[0123] Optionally, the index information of the data after the penultimate data index information data block is obtained by scanning, and it can be specifically implemented through the following scheme:

[0124] The data in the data storage partition is stored in basic storage units (sectors). The minimum storage unit is generally 4K. The metadata related to 4K data is recorded in the sector footer of each basic storage unit. The metadata includes, but is not limited to, the internal identifier (Inode ID) of the data block to which the data in the basic storage unit belongs, data index information, etc. The data index information includes offset information (offset, poffset), length information, etc. The sector footer and the data of the basic storage unit are persistently stored together. Therefore, when scanning the data after the penultimate data index information data block, the data index information can be obtained according to the sector footer of each basic storage unit.

[0125] It should be noted that since it supports data in any case, in order to save physical space, multiple user-written data are supported to be merged into one basic storage unit. The metadata of multiple written data is recorded in the sector footer of the basic storage unit, including the internal identifier and index information of the data block to which each piece of data belongs. Because the merged writing of multiple data blocks within the basic storage unit is supported, the length of the sector footer of the basic storage unit includes a part of variable data block index information and fixed other information.

[0126] If the user's write data is relatively large, it may span multiple basic storage units. To ensure the atomicity of a user's write operation during the restart scan, the tail information of the basic storage unit also records the total number of basic storage units (sector num) that a piece of data spans, as well as the order (sector index) of the current basic storage unit among the spanned basic storage units. During the scan, if all the spanned basic storage units are scanned successfully, it indicates that the piece of data is valid, and the data index information in the spanned basic storage units can be obtained; if at least one of the spanned basic storage units fails to be scanned successfully (e.g., lost), it indicates that the piece of data is invalid, and the data index information in all the spanned basic storage units does not need to be obtained anymore.

[0127] Based on any of the above embodiments, in the metadata of the data storage partition of the storage engine, the garbage amount information is a key input parameter for the garbage collection selection strategy of the data storage partition. Its accuracy determines the level of garbage collection efficiency, and thus determines the write amplification during the operation of the storage engine system. The metadata of the data storage partition of the storage engine is usually persistently stored from memory to the metadata storage partition at regular intervals. If there is a certain time interval between the abnormal restart time and the persistent storage of the metadata of the data storage partition, it will cause the metadata of the data storage partition to be inaccurate, and the garbage amount information in it will no longer be accurate. Therefore, in this embodiment, the metadata of the data storage partition can be obtained from the metadata storage partition, the first garbage amount information can be obtained from it, and then the first garbage amount information can be supplemented in combination with the log of the storage engine to more accurately restore the garbage amount information of the data storage partition. Specifically, it can be as follows:

[0128] Obtain the first garbage amount information included in the metadata of the data storage partition of the storage engine;

[0129] Obtain a second set of log data after the creation time of the metadata of the data storage partition from the log of the storage engine;

[0130] Determine the second garbage amount information after the creation time of the metadata of the data storage partition according to the log data in the second set of log data, and supplement the second garbage amount information to the first garbage amount information.

[0131] In this embodiment, the first garbage amount information included in the metadata of the data storage partition of the storage engine can be obtained; log data after the creation time of the metadata of the data storage partition can be obtained from the log of the storage engine to form a second log data set, and operations that generate garbage can be searched for in the log data in the second log data set to determine the second garbage amount information after the creation time of the metadata of the data storage partition. Then, the second garbage amount information and the first garbage amount information are fused, that is, the second garbage amount information is supplemented into the first garbage amount information, so that the garbage amount information of the data storage partition can be restored more accurately, facilitating the determination of an appropriate garbage collection strategy.

[0132] Corresponding to the method for processing metadata of the storage engine in the above embodiment, Figure 6 is a structural block diagram of a device for processing metadata of a storage engine provided by an embodiment of the present disclosure. For ease of description, only parts related to the embodiment of the present disclosure are shown. Referring to Figure 6 the device 600 for processing metadata of the storage engine includes a metadata persistence unit 601 and an index unit 602.

[0133] Among them, the metadata persistence unit 601 is used to create a snapshot for each unpersisted metadata block in the memory used by the storage engine, persistently store each snapshot in sequence into the metadata storage partition of the non-volatile storage medium, and record the snapshot index information of each snapshot in the metadata storage partition in the memory; each metadata block includes the metadata of a data block managed by the storage engine;

[0134] The index unit 602 is used to, if the unpersisted snapshot index information in the memory reaches a preset data volume, persistently store the unpersisted snapshot index information into the metadata storage partition as a snapshot index information data block, and add a pointer to the previous snapshot index information data block in the snapshot index information data block.

[0135] In an embodiment of the present disclosure, the index unit 602 is further used for:

[0136] When the metadata storage partition is full, add a pointer to the last snapshot index information data block to the tail information of the metadata storage partition, for finding the last snapshot index information data block through the pointer to the last snapshot index information data block.

[0137] In an embodiment of the present disclosure, the device further includes a recovery unit 603, which is used for:

[0138] When an instruction to recover metadata is triggered, find the last snapshot index information data block from the metadata storage partition;

[0139] Find the corresponding snapshots in the metadata storage partition according to the snapshot index information in the last snapshot index information data block, and load each snapshot in the memory to restore the metadata blocks corresponding to each snapshot into the memory;

[0140] Find the previous snapshot index information data block according to the pointer to the previous snapshot index information data block included in the last snapshot index information data block.

[0141] In an embodiment of the present disclosure, when the recovery unit 603 finds the last snapshot index information data block from the metadata storage partition, it is used for:

[0142] If there is a pointer to the last snapshot index information data block in the tail information of the metadata storage partition, find the last snapshot index information data block according to the pointer to the last snapshot index information data block; or

[0143] If there is no pointer to the last snapshot index information data block in the tail information of the metadata storage partition, scan the metadata storage partition from the back to the front to find the last snapshot index information data block.

[0144] In an embodiment of the present disclosure, when the recovery unit 603 scans the metadata storage partition from the back to the front, it is further used for:

[0145] Before finding the last snapshot index information data block, load the scanned snapshots in the memory.

[0146] In an embodiment of the present disclosure, when the recovery unit 603 loads each snapshot in the memory, it is used for:

[0147] If multiple snapshots related to the same data block in the storage engine are found, load the latest snapshot in the memory and ignore other snapshots except the latest snapshot.

[0148] In an embodiment of the present disclosure, the recovery unit 603 is further used for:

[0149] Obtain a first set of log data after the creation time of the last snapshot in the metadata storage partition;

[0150] Play back the log data in the first set of log data in chronological order to generate metadata after the creation time of the last snapshot in the memory.

[0151] In one embodiment of the present disclosure, the recovery unit 603 is further configured to: scan the data written after the creation time of the last snapshot in the data storage partition of the storage engine, and generate metadata corresponding to the data written after the creation time of the last snapshot in the memory.

[0152] In one embodiment of the present disclosure, the recovery unit 603 is further configured to: persistently store the mapping relationship between the internal identifier of a data block in the storage engine and its external identifier in the non-volatile storage medium;

[0153] Correspondingly, when the recovery unit 603 searches for corresponding snapshots from the metadata storage partition according to the snapshot index information in the data block of the last snapshot index information and loads each snapshot into the memory, it is configured to:

[0154] Restore the mapping relationship to the memory;

[0155] Search for corresponding snapshots from the metadata storage partition according to the snapshot index information in the data block of the last snapshot index information, filter valid snapshots according to the mapping relationship, and load each valid snapshot into the memory.

[0156] In one embodiment of the present disclosure, the recovery unit 603 is further configured to:

[0157] In response to a storage engine exit instruction, create snapshots for all unpersisted metadata blocks in the memory, sequentially persistently store each snapshot into the metadata storage partition, and add a first identifier after the last snapshot and then execute the storage engine exit instruction, where the first identifier is used to indicate the normal exit of the storage engine;

[0158] When a recovery metadata instruction is triggered, scan the metadata storage partition from the back to the front. If the first identifier is detected, load the snapshots in the metadata storage partition into the memory to restore all metadata in the memory;

[0159] Add a second identifier after the first identifier, where the second identifier is used to indicate the invalidation of the first identifier.

[0160] In one embodiment of the present disclosure, the recovery unit 603 is further configured to:

[0161] Compare the write time of each data block in the data storage partition of the storage engine with the creation time of its corresponding snapshot respectively;

[0162] If the write time of all data blocks is less than the creation time of the corresponding snapshot, load the first data index information in the snapshot into the memory; or

[0163] If the write time of at least one data block is not less than the creation time of the corresponding snapshot, load the first data index information in the snapshot and the second data index information of the data written to the storage engine after the creation time of the last snapshot in the metadata storage partition into the memory.

[0164] In an embodiment of the present disclosure, when the recovery unit 603 loads the first data index information in the snapshot and the second data information index information of the data written to the storage engine after the creation time of the last snapshot in the metadata storage partition into the memory, it is configured to:

[0165] Scan the data storage partition of the storage engine to obtain the second data index information, and load the second data index information into the memory;

[0166] When receiving any operation instruction for any target data in the data storage partition, if the data index information of the target data does not exist in the memory, determine the target snapshot where the data index information of the target data is located, load the first data index information in the target snapshot into the memory, and fuse it with the second data index information.

[0167] In an embodiment of the present disclosure, the data storage partition includes the data of the storage engine and multiple data index information data blocks, and the multiple data index information data blocks are scattered in the data storage partition. Any data index information data block includes the data index information of part of the data and a pointer to the previous data index information data block;

[0168] When the recovery unit 603 scans the data storage partition of the storage engine to obtain the second data index information and loads the second data index information into the memory, it is configured to:

[0169] Scan the data storage partition from the back to the front until the penultimate data index information data block in the data storage partition is scanned, and sequentially search for the remaining data index information data blocks in the data storage partition according to the pointer of the previous data index information data block;

[0170] Obtain data index information for the data after the penultimate data index information data block;

[0171] Filter out the data index information of the data written to the data storage partition after the creation time of the last snapshot from the data index information of the data after the penultimate data index information data block and the data index information included in the penultimate data index information data block and the remaining data index information data blocks, and load it into the memory as the second data index information.

[0172] In an embodiment of the present disclosure, the recovery unit 603 is further configured to:

[0173] Obtain the first garbage volume information included in the metadata of the data storage partition of the storage engine;

[0174] Obtain a second set of log data after the creation time of the metadata of the data storage partition;

[0175] Determine the second garbage volume information after the creation time of the metadata of the data storage partition according to the log data in the second set of log data, and supplement the second garbage volume information to the first garbage volume information.

[0176] The device provided in this embodiment can be used to execute the technical solutions of the above method embodiments, and its implementation principles and technical effects are similar, which will not be elaborated here.

[0177] To implement the above embodiments, the present disclosure also provides an electronic device.

[0178] Refer to Figure 7 , which shows a schematic structural diagram of an electronic device 700 suitable for implementing the embodiments of the present disclosure. The electronic device 700 can be a terminal device or a server. Among them, the terminal device can include, but is not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, personal digital assistants (Personal Digital Assistant, abbreviated as PDA), tablet computers (Tablet Computer), portable media players (Portable Media Player, abbreviated as PMP), in-vehicle terminals (such as in-vehicle navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 7 The electronic device shown is only an example and should not impose any limitations on the functions and usage scopes of the embodiments of the present disclosure.

[0179] As Figure 7As shown, the electronic device 700 may include a processing device (such as a central processing unit, a graphics processing unit, etc.) 701, which may perform various appropriate actions and processes according to a program stored in a read only memory (ROM) 702 or a program loaded from a storage device 708 into a random access memory (RAM) 703. In the RAM 703, various programs and data required for the operation of the electronic device 700 are also stored. The processing device 701, the ROM 702, and the RAM 703 are connected to each other through a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0180] Generally, the following devices may be connected to the I / O interface 705: an input device 706 including, for example, a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 707 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 708 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 709. The communication device 709 may allow the electronic device 700 to communicate with other devices wirelessly or wirelessly to exchange data. Although Figure 7 an electronic device 700 with various devices is shown, it should be understood that it is not required to implement or have all the shown devices. Instead, more or fewer devices may be implemented or had.

[0181] Specifically, according to an embodiment of the present disclosure, the process described above with reference to the flowchart may be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable storage medium, and the computer program includes program codes for performing the method shown in the flowchart. In such an embodiment, the computer program may be downloaded and installed from a network through the communication device 709, or installed from the storage device 708, or installed from the ROM 702. When the computer program is executed by the processing device 701, the above functions defined in the method of the embodiment of the present disclosure are executed.

[0182] It should be noted that the above-mentioned computer-readable storage medium in the present disclosure can be a computer-readable signal medium, a computer-readable storage medium, or any combination of the two. A computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the above. More specific examples of the computer-readable storage medium can include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, the computer-readable storage medium can be any tangible medium that contains or stores a program, and this program can be used by or in combination with an instruction execution system, apparatus, or device. In the present disclosure, the computer-readable signal medium can include a data signal propagated in a baseband or as part of a carrier wave, which carries computer-readable program code. Such a propagated data signal can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer-readable signal medium can also be any computer-readable storage medium other than the computer-readable storage medium, and this computer-readable signal medium can send, propagate, or transmit a program for use by or in combination with an instruction execution system, apparatus, or device. The program code contained on the computer-readable storage medium can be transmitted by any appropriate medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0183] The above-mentioned computer-readable storage medium can be included in the above-mentioned electronic device; it can also exist separately without being assembled into the electronic device.

[0184] The above-mentioned computer-readable storage medium carries one or more programs, and when the one or more programs are executed by the electronic device, the electronic device is caused to execute the method shown in the above embodiment.

[0185] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages or combinations thereof. The programming languages include object-oriented programming languages such as Java, Smalltalk, C++, and also include conventional procedural programming languages such as the "C" language or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, executed as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0186] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architectures, functions, and operations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagram may represent a module, a program segment, or a part of code that contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than marked in the accompanying drawings. For example, two consecutive blocks shown may actually be executed substantially in parallel, and they may sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each block in the block diagram and / or flowchart, and combinations of blocks in the block diagram and / or flowchart, may be implemented by a dedicated hardware-based system for performing the specified functions or operations, or may be implemented by a combination of dedicated hardware and computer instructions.

[0187] The units involved in the embodiments described in the present disclosure may be implemented in software or in hardware. Among them, the name of the unit does not constitute a limitation to the unit itself in some cases. For example, the first acquisition unit may also be described as "the unit for acquiring at least two Internet protocol addresses".

[0188] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, by way of non-limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGA), application specific integrated circuits (ASIC), application specific standard products (ASSP), system on a chip (SOC), complex programmable logic devices (CPLD), and so on.

[0189] The electronic device, computer-readable storage medium, and computer program product provided by the embodiments of the present disclosure can be used to execute the technical solutions of the above method embodiments. Their implementation principles and technical effects are similar, and will not be elaborated here in this embodiment.

[0190] In a first aspect, according to one or more embodiments of the present disclosure, a method for processing metadata of a storage engine is provided, including:

[0191] Create a snapshot for each unpersisted metadata block in the memory used by the storage engine, sequentially persistently store each snapshot in the metadata storage partition of the non-volatile storage medium, and record the snapshot index information of each snapshot in the metadata storage partition in the memory; where each metadata block includes the metadata of a data block managed by the storage engine.

[0192] If the unpersisted snapshot index information in the memory reaches a preset data volume, persistently store the unpersisted snapshot index information in a snapshot index information data block in the metadata storage partition, and add a pointer to the previous snapshot index information data block in the snapshot index information data block.

[0193] According to one or more embodiments of the present disclosure, the method further includes:

[0194] When the metadata storage partition is full, add a pointer to the last snapshot index information data block in the tail information of the metadata storage partition for finding the last snapshot index information data block through the pointer to the last snapshot index information data block.

[0195] According to one or more embodiments of the present disclosure, the method further includes:

[0196] When an instruction to restore metadata is triggered, find the last snapshot index information data block from the metadata storage partition;

[0197] Find the corresponding snapshots from the metadata storage partition according to the snapshot index information in the last snapshot index information data block, and load each snapshot in the memory to restore the metadata blocks corresponding to each snapshot to the memory;

[0198] Find the previous snapshot index information data block according to the pointer to the previous snapshot index information data block included in the last snapshot index information data block.

[0199] According to one or more embodiments of the present disclosure, the finding the last snapshot index information data block from the metadata storage partition includes:

[0200] If there is a pointer to the last snapshot index information data block in the tail information of the metadata storage partition, find the last snapshot index information data block according to the pointer to the last snapshot index information data block; or

[0201] If there is no pointer to the last snapshot index information data block in the tail information of the metadata storage partition, scan the metadata storage partition from the back to the front to find the last snapshot index information data block.

[0202] According to one or more embodiments of the present disclosure, when scanning the metadata storage partition from the back to the front, it further includes:

[0203] Before finding the last snapshot index information data block, load the scanned snapshots into the memory.

[0204] According to one or more embodiments of the present disclosure, the method further includes:

[0205] Obtain a first set of log data after the creation time of the last snapshot in the metadata storage partition;

[0206] Play back the log data in the first set of log data in chronological order, and generate metadata after the creation time of the last snapshot in the memory.

[0207] According to one or more embodiments of the present disclosure, the method further includes:

[0208] In the data storage partition of the storage engine, scan the data written after the creation time of the last snapshot, and generate the data written after the creation time of the last snapshot into the corresponding metadata in the memory.

[0209] According to one or more embodiments of the present disclosure, the method further includes:

[0210] Persistently store the mapping relationship between the internal identifier of the data block in the storage engine and the external identifier of the data block in the non-volatile storage medium;

[0211] Correspondingly, the finding the corresponding snapshots from the metadata storage partition according to the snapshot index information in the last snapshot index information data block and loading each snapshot into the memory includes:

[0212] Restore the mapping relationship to the memory;

[0213] Find the corresponding snapshots from the metadata storage partition according to the snapshot index information in the last snapshot index information data block, filter out valid snapshots according to the mapping relationship, and load each valid snapshot into the memory.

[0214] According to one or more embodiments of the present disclosure, the method further includes:

[0215] In response to a storage engine exit instruction, create snapshots for all unpersisted metadata blocks in the memory respectively, persistently store each snapshot in the metadata storage partition in sequence, and add a first identifier after the last snapshot and then execute the storage engine exit instruction, where the first identifier is used to indicate that the storage engine exits normally;

[0216] When an instruction to restore metadata is triggered, scan the metadata storage partition from the back to the front. If the first identifier is detected, load the snapshots in the metadata storage partition into the memory to restore all metadata in the memory;

[0217] Add a second identifier after the first identifier, where the second identifier is used to indicate that the first identifier fails.

[0218] According to one or more embodiments of the present disclosure, the method further includes:

[0219] Compare the write time of each data block in the data storage partition of the storage engine with the creation time of its corresponding snapshot respectively;

[0220] If the write time of all data blocks is less than the creation time of the corresponding snapshot, load the first data index information in the snapshot into the memory; or

[0221] If the write time of at least one data block is not less than the creation time of the corresponding snapshot, load the first data index information in the snapshot and the second data index information of the data written to the storage engine after the creation time of the last snapshot in the metadata storage partition into the memory.

[0222] According to one or more embodiments of the present disclosure, loading the first data index information in the snapshot and the second data information index information of the data written to the storage engine after the creation time of the last snapshot in the metadata storage partition into the memory includes:

[0223] Scan the data storage partition of the storage engine to obtain the second data index information, and load the second data index information into the memory;

[0224] When receiving any operation instruction for any target data in the data storage partition, if the data index information of the target data does not exist in the memory, determine the target snapshot where the data index information of the target data is located, load the first data index information in the target snapshot into the memory, and fuse it with the second data index information.

[0225] According to one or more embodiments of the present disclosure, the data storage partition includes data of a storage engine and a plurality of data index information data blocks, and the plurality of data index information data blocks are dispersed in the data storage partition. Any data index information data block includes data index information of part of the data and a pointer to the previous data index information data block;

[0226] The scanning the data storage partition of the storage engine to obtain the second data index information and loading the second data index information into the memory includes:

[0227] Scan the data storage partition from the back to the front until the second-to-last data index information data block in the data storage partition is scanned, and sequentially search for the remaining data index information data blocks in the data storage partition according to the pointer of the previous data index information data block;

[0228] Obtain data index information for the data after the second-to-last data index information data block;

[0229] From the data index information of the data after the second-to-last data index information data block and the data index information included in the second-to-last data index information data block and the remaining data index information data blocks, filter out the data index information of the data written into the data storage partition after the creation time of the last snapshot, and use it as the second data index information to be loaded into the memory.

[0230] According to one or more embodiments of the present disclosure, the method further includes:

[0231] Obtain the first garbage amount information included in the metadata of the data storage partition of the storage engine;

[0232] Obtain a second set of log data after the creation time of the metadata of the data storage partition;

[0233] Determine the second garbage amount information after the creation time of the metadata of the data storage partition according to the log data in the second set of log data, and supplement the second garbage amount information to the first garbage amount information.

[0234] In a second aspect, one or more embodiments of the present disclosure provide a metadata processing device for a storage engine, including:

[0235] A metadata persistence unit, configured to create a snapshot for each unpersisted metadata block in the memory used by the storage engine, persistently store each snapshot in sequence into a metadata storage partition of a non-volatile storage medium, and record the snapshot index information of each snapshot in the metadata storage partition in the memory; wherein each metadata block includes the metadata of a data block managed by the storage engine;

[0236] An indexing unit, configured to, if the unpersisted snapshot index information in the memory reaches a preset data volume, persistently store the unpersisted snapshot index information into the metadata storage partition as a snapshot index information data block, and add a pointer to the previous snapshot index information data block to the snapshot index information data block.

[0237] According to one or more embodiments of the present disclosure, the indexing unit is further configured to:

[0238] When the metadata storage partition is full, add a pointer to the last snapshot index information data block to the tail information of the metadata storage partition, for finding the last snapshot index information data block through the pointer to the last snapshot index information data block.

[0239] According to one or more embodiments of the present disclosure, the device further includes a recovery unit, configured to:

[0240] When an instruction to recover metadata is triggered, find the last snapshot index information data block from the metadata storage partition;

[0241] Find the corresponding snapshots from the metadata storage partition according to the snapshot index information in the last snapshot index information data block, and load each snapshot in the memory, so as to restore the metadata blocks corresponding to each snapshot into the memory;

[0242] Find the previous snapshot index information data block according to the pointer to the previous snapshot index information data block included in the last snapshot index information data block.

[0243] According to one or more embodiments of the present disclosure, when the recovery unit finds the last snapshot index information data block from the metadata storage partition, it is configured to:

[0244] If there is a pointer to the last snapshot index information data block in the tail information of the metadata storage partition, find the last snapshot index information data block according to the pointer to the last snapshot index information data block; or

[0245] If there is no pointer to the last snapshot index information data block in the tail information of the metadata storage partition, scan the metadata storage partition from the end to the front to find the last snapshot index information data block.

[0246] According to one or more embodiments of the present disclosure, when the recovery unit scans the metadata storage partition from the end to the front, it is further configured to:

[0247] Before finding the last snapshot index information data block, load the scanned snapshots into the memory.

[0248] According to one or more embodiments of the present disclosure, the recovery unit is further configured to:

[0249] Obtain a first set of log data after the creation time of the last snapshot in the metadata storage partition;

[0250] Play back the log data in the first set of log data in chronological order to generate metadata after the creation time of the last snapshot in the memory.

[0251] According to one or more embodiments of the present disclosure, the recovery unit is further configured to:

[0252] In the data storage partition of the storage engine, scan the data written after the creation time of the last snapshot, and generate the data written after the creation time of the last snapshot into the corresponding metadata in the memory.

[0253] According to one or more embodiments of the present disclosure, the recovery unit is further configured to:

[0254] Persistently store the mapping relationship between the internal identifier of the data block in the storage engine and the external identifier of the data block in the non-volatile storage medium;

[0255] Correspondingly, when the recovery unit finds the corresponding snapshots from the metadata storage partition according to the snapshot index information in the last snapshot index information data block and loads each snapshot into the memory, it is configured to:

[0256] Restore the mapping relationship to the memory;

[0257] Find the corresponding snapshots from the metadata storage partition according to the snapshot index information in the last snapshot index information data block, filter out the valid snapshots according to the mapping relationship, and load each valid snapshot into the memory.

[0258] According to one or more embodiments of the present disclosure, the recovery unit is further configured to:

[0259] In response to a storage engine exit instruction, create snapshots for all unpersisted metadata blocks in the memory respectively, persistently store each snapshot in the metadata storage partition in sequence, and add a first identifier after the last snapshot and then execute the storage engine exit instruction, where the first identifier is used to indicate the normal exit of the storage engine;

[0260] When an instruction to restore metadata is triggered, scan the metadata storage partition from the back to the front. If the first identifier is detected, load the snapshots in the metadata storage partition into the memory to restore the full amount of metadata in the memory;

[0261] Add a second identifier after the first identifier, where the second identifier is used to indicate the invalidation of the first identifier.

[0262] According to one or more embodiments of the present disclosure, the restoration unit is further configured to:

[0263] Compare the write time of each data block in the data storage partition of the storage engine with the creation time of its corresponding snapshot respectively;

[0264] If the write time of all data blocks is less than the creation time of the corresponding snapshot, load the first data index information in the snapshot into the memory; or

[0265] If the write time of at least one data block is not less than the creation time of the corresponding snapshot, load the first data index information in the snapshot and the second data index information of the data written to the storage engine after the creation time of the last snapshot in the metadata storage partition into the memory.

[0266] According to one or more embodiments of the present disclosure, when the restoration unit loads the first data index information in the snapshot and the second data information index information of the data written to the storage engine after the creation time of the last snapshot in the metadata storage partition into the memory, it is configured to:

[0267] Scan the data storage partition of the storage engine to obtain the second data index information and load the second data index information into the memory;

[0268] When receiving any operation instruction for any target data in the data storage partition, if the data index information of the target data does not exist in the memory, determine the target snapshot where the data index information of the target data is located, load the first data index information in the target snapshot into the memory, and fuse it with the second data index information.

[0269] According to one or more embodiments of the present disclosure, the data storage partition includes data of a storage engine and a plurality of data index information data blocks, and the plurality of data index information data blocks are dispersed in the data storage partition. Any one of the data index information data blocks includes data index information of some data and a pointer to the previous data index information data block;

[0270] When the recovery unit scans the data storage partition of the storage engine to obtain the second data index information and loads the second data index information into the memory, it is used for:

[0271] Scan the data storage partition from the back to the front until the penultimate data index information data block in the data storage partition is scanned, and sequentially search for the remaining data index information data blocks in the data storage partition according to the pointer of the previous data index information data block;

[0272] Obtain data index information for the data after the penultimate data index information data block;

[0273] From the data index information of the data after the penultimate data index information data block and the data index information included in the penultimate data index information data block and the remaining data index information data blocks, screen out the data index information of the data written into the data storage partition after the creation time of the last snapshot, and use it as the second data index information to be loaded into the memory.

[0274] According to one or more embodiments of the present disclosure, the recovery unit is further used for:

[0275] Obtain the first garbage amount information included in the metadata of the data storage partition of the storage engine;

[0276] Obtain a second set of log data after the creation time of the metadata of the data storage partition;

[0277] Determine the second garbage amount information after the creation time of the metadata of the data storage partition according to the log data in the second set of log data, and supplement the second garbage amount information to the first garbage amount information.

[0278] In a third aspect, according to one or more embodiments of the present disclosure, there is provided an electronic device, including: at least one processor and a memory;

[0279] The memory stores computer execution instructions;

[0280] The at least one processor executes the computer-executable instructions stored in the memory, such that the at least one processor executes the metadata processing method of the storage engine as described in the first aspect above and various possible designs of the first aspect.

[0281] In a fourth aspect, according to one or more embodiments of the present disclosure, there is provided a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the metadata processing method of the storage engine as described in the first aspect above and various possible designs of the first aspect.

[0282] In a fifth aspect, according to one or more embodiments of the present disclosure, there is provided a computer program product including a computer program, which, when executed by a processor, implements the metadata processing method of the storage engine as described in the first aspect above and various possible designs of the first aspect.

[0283] The above description is only a preferred embodiment of the present disclosure and an explanation of the applied technical principles. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by the specific combination of the above technical features, and should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosure concept. For example, the technical solutions formed by mutually replacing the above features with the (but not limited to) technical features having similar functions disclosed in the present disclosure.

[0284] In addition, although the operations are depicted in a particular order, this should not be construed as requiring that the operations be performed in the particular order shown or in sequential order. In certain environments, multitasking and parallel processing may be advantageous. Similarly, although several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of the present disclosure. Certain features described in the context of separate embodiments may also be implemented combinatorially in a single embodiment. Conversely, the various features described in the context of a single embodiment may also be implemented separately or in any suitable sub-combination in multiple embodiments.

[0285] Although the subject matter has been described in language specific to structural features and / or methodological act logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are merely example forms for implementing the claims.

Claims

1. A method for processing metadata of a storage engine, characterized in that, including: Create a snapshot for each unpersisted metadata block in the memory used by the storage engine, sequentially persist each snapshot to the metadata storage partition of the non-volatile storage medium, and record the snapshot index information of each snapshot in the metadata storage partition in the memory; Each metadata block includes the metadata of a data block managed by the storage engine; If the unpersisted snapshot index information in the memory reaches the preset data volume, persist the unpersisted snapshot index information as a snapshot index information data block to the metadata storage partition, and add a pointer to the previous snapshot index information data block in the snapshot index information data block.

2. The method according to claim 1, wherein The method further includes: When the metadata storage partition is full, add a pointer to the last snapshot index information data block in the tail information of the metadata storage partition, for finding the last snapshot index information data block through the pointer to the last snapshot index information data block.

3. The method according to claim 1, wherein The method further includes: When the instruction to restore metadata is triggered, find the last snapshot index information data block from the metadata storage partition; Find the corresponding snapshots from the metadata storage partition according to the snapshot index information in the last snapshot index information data block, and load each snapshot in the memory to restore the metadata blocks corresponding to each snapshot to the memory; Find the previous snapshot index information data block according to the pointer to the previous snapshot index information data block included in the last snapshot index information data block.

4. The method according to claim 3, characterized in that, The finding the last snapshot index information data block from the metadata storage partition includes: If there is a pointer to the last snapshot index information data block in the tail information of the metadata storage partition, find the last snapshot index information data block according to the pointer to the last snapshot index information data block; or If there is no pointer to the last snapshot index information data block in the tail information of the metadata storage partition, scan the metadata storage partition from the back to the front to find the last snapshot index information data block.

5. The method according to claim 4, characterized in that When scanning the metadata storage partition from the back to the front, it further includes: Before finding the last snapshot index information data block, load the scanned snapshots in the memory.

6. The method according to claim 3, characterized in that The method further includes: Obtain a first set of log data after the creation time of the last snapshot in the metadata storage partition; Play back the log data in the first set of log data in chronological order, and generate metadata after the creation time of the last snapshot in the memory.

7. The method according to claim 6, wherein The method further includes: In the data storage partition of the storage engine, scan the data written after the creation time of the last snapshot, and generate the metadata corresponding to the data written after the creation time of the last snapshot in the memory.

8. The method according to any one of claims 3-7, characterized in that, The method further includes: Persist the mapping relationship between the internal identifier and the external identifier of the data block in the storage engine to the non-volatile storage medium; Correspondingly, finding corresponding snapshots from the metadata storage partition according to the snapshot index information in the last snapshot index information data block and loading each snapshot into the memory includes: Restoring the mapping relationship into the memory; Finding corresponding snapshots from the metadata storage partition according to the snapshot index information in the last snapshot index information data block, screening valid snapshots according to the mapping relationship, and loading each valid snapshot into the memory.

9. The method according to claim 3, characterized in that, The method further includes: In response to a storage engine exit instruction, creating snapshots for all unpersisted metadata blocks in the memory, sequentially and persistently storing each snapshot into the metadata storage partition, and adding a first identifier after the last snapshot and then executing the storage engine exit instruction, where the first identifier is used to indicate that the storage engine exits normally; When an instruction to restore metadata is triggered, scanning the metadata storage partition from the back to the front. If the first identifier is detected, loading the snapshots in the metadata storage partition into the memory to restore all metadata in the memory; Adding a second identifier after the first identifier, where the second identifier is used to indicate that the first identifier becomes invalid.

10. The method according to claim 3, characterized in that, The method further includes: Respectively comparing the write time of each data block in the data storage partition of the storage engine with the creation time of its corresponding snapshot; If the write time of all data blocks is less than the creation time of the corresponding snapshot, loading the first data index information in the snapshot into the memory; or If the write time of at least one data block is not less than the creation time of the corresponding snapshot, loading the first data index information in the snapshot and the second data index information of the data written into the storage engine after the creation time of the last snapshot in the metadata storage partition into the memory.

11. The method according to claim 10, wherein Loading the first data index information in the snapshot and the second data information index information of the data written into the storage engine after the creation time of the last snapshot in the metadata storage partition into the memory includes: Scanning the data storage partition of the storage engine to obtain the second data index information and loading the second data index information into the memory; When receiving any operation instruction for any target data in the data storage partition, if the data index information of the target data does not exist in the memory, determining the target snapshot where the data index information of the target data is located, loading the first data index information in the target snapshot into the memory, and fusing it with the second data index information.

12. The method according to claim 11, wherein The data storage partition includes the data of the storage engine and multiple data index information data blocks, and the multiple data index information data blocks are scattered in the data storage partition. Any data index information data block includes the data index information of part of the data and a pointer to the previous data index information data block; Scanning the data storage partition of the storage engine to obtain the second data index information and loading the second data index information into the memory includes: Scan the data storage partition from the back to the front until the penultimate data index information data block in the data storage partition is scanned, and sequentially search for the remaining data index information data blocks in the data storage partition according to the pointer of the previous data index information data block; Obtain data index information for the data after the penultimate data index information data block; From the data index information of the data after the penultimate data index information data block and the data index information included in the penultimate data index information data block and the remaining data index information data blocks, filter out the data index information of the data written into the data storage partition after the creation time of the last snapshot, and load it into the memory as the second data index information.

13. The method according to claim 3, wherein The method further includes: Obtain the first garbage amount information included in the metadata of the data storage partition of the storage engine; Obtain a second set of log data after the creation time of the metadata of the data storage partition; Determine the second garbage amount information after the creation time of the metadata of the data storage partition according to the log data in the second set of log data, and supplement the second garbage amount information to the first garbage amount information.

14. A metadata processing device for a storage engine, characterized in that Includes: A metadata persistence unit for creating snapshots for each unpersisted metadata block in the memory used by the storage engine, sequentially persistently storing the snapshots in the metadata storage partition of the non-volatile storage medium, and recording the snapshot index information of each snapshot in the metadata storage partition in the memory; Each metadata block includes the metadata of a data block managed by the storage engine; An index unit for, if the unpersisted snapshot index information in the memory reaches a preset data volume, persistently storing the unpersisted snapshot index information in a snapshot index information data block in the metadata storage partition, and adding a pointer to the previous snapshot index information data block to the snapshot index information data block.

15. An electronic device, characterized in that, Includes: A processor and a memory; The memory stores computer execution instructions; The processor executes the computer execution instructions stored in the memory, so that the processor executes the method according to any one of claims 1-13.

16. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer execution instructions, and when the processor executes the computer execution instructions, the method according to any one of claims 1-13 is implemented.

17. A computer program product, comprising a computer program, characterized in that, The computer program, when executed by the processor, implements the method according to any one of claims 1-13.

Images