Mapping table snapshot generation method and electronic device

Abstract

The application discloses a mapping table snapshot generation method and an electronic device, relates to the technical field of data management, and performs block processing on a logical-to-physical mapping table of a full allocation volume and counts an empty block rate, adaptively selects a snapshot generation strategy according to an actual empty block distribution working condition of the mapping table, and obtains a complete mapping table snapshot that can be directly loaded. Compared with a recovery mode of full-disk traversal reconstruction of the mapping table, the recovery mode does not need to scan all data blocks in a fault recovery stage, does not need to reconstruct a mapping index in real time, can directly load a pre-generated snapshot to quickly restore a complete logical-to-physical mapping table, eliminates high time consumption overhead of traversal reconstruction of a large-capacity volume, shortens a fault recovery time length, reduces a business interruption time, and thus improves stability of business execution.

Description

Technical Field

[0001] This application relates to the field of data management technology, and in particular to a method for generating mapping table snapshots and an electronic device. Background Technology

[0002] In large-scale data storage scenarios, the primary-backup mirroring fault recovery mechanism requires a full traversal of data blocks to rebuild the mapping table during fault takeover. The recovery time increases significantly with the increase of volume capacity, resulting in technical problems such as long fault recovery time for large-capacity volumes and prolonged business interruption. Summary of the Invention

[0003] This application provides a method for generating mapping table snapshots and an electronic device to at least solve the problem of poor business execution stability in related technologies.

[0004] This application provides a method for generating a mapping table snapshot, including:

[0005] Obtain the logical-to-physical mapping table of the fully allocated volume, and divide the logical-to-physical mapping table into fixed-size data blocks;

[0006] Traverse the data blocks, count the number of non-empty data blocks and calculate the empty block rate;

[0007] When the empty block rate is greater than or equal to the preset ratio, compression processing is performed on the non-empty data blocks to obtain compressed data blocks and index entries. A sparse index snapshot structure is then constructed based on the compressed data blocks and index entries.

[0008] When the empty block rate is less than the preset ratio, the logic is directly copied to the physical mapping table in a dense format to obtain a dense snapshot structure.

[0009] Write the sparse index snapshot structure or dense snapshot structure to the pre-allocated shared storage snapshot slot.

[0010] This application also provides a mapping table snapshot generation apparatus, including:

[0011] The acquisition module is used to obtain the logical-to-physical mapping table of the fully allocated volume and divide the logical-to-physical mapping table into data blocks of fixed size;

[0012] The statistics module is used to traverse data blocks, count the number of non-empty data blocks, and calculate the empty block rate.

[0013] The sparse construction module is used to perform compression processing on non-empty data blocks when the empty block rate is greater than or equal to a preset ratio, to obtain compressed data blocks and index entries, and to construct a sparse index snapshot structure based on the compressed data blocks and index entries.

[0014] The dense building module is used to directly copy logic to the physical mapping table in a dense format when the empty block rate is less than a preset ratio, so as to obtain a dense snapshot structure.

[0015] The write module is used to write sparse index snapshot structures or dense snapshot structures to pre-allocated shared storage snapshot slots.

[0016] This application also provides an electronic device, including: a memory for storing a computer program; and a processor for executing the computer program to implement the steps of any of the above-described mapping table snapshot generation methods.

[0017] This application also provides a non-volatile computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the steps of any of the above-described mapping table snapshot generation methods.

[0018] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of any of the above-described mapping table snapshot generation methods.

[0019] This application utilizes a method to segment the logical-to-physical mapping table of a fully allocated volume and calculate the empty block rate. Based on the actual empty block distribution of the mapping table, an adaptive snapshot generation strategy is selected to obtain a complete mapping table snapshot that can be directly loaded. Compared to the recovery method of reconstructing the mapping table from scratch, this method eliminates the need to scan all data blocks during the fault recovery phase and rebuild the mapping index in real time. It allows for the direct loading of pre-generated snapshots to quickly restore the complete logical-to-physical mapping table, eliminating the high time-consuming overhead of large-capacity volume traversal and reconstruction, shortening fault recovery time, reducing business interruption time, and thus improving the stability of business operations. Attached Figure Description

[0020] To more clearly illustrate the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is a schematic diagram illustrating an application scenario of a mapping table snapshot generation method provided in an embodiment of this application;

[0022] Figure 2 A flowchart illustrating a method for generating a mapping table snapshot provided in an embodiment of this application;

[0023] Figure 3 A flowchart illustrating another method for generating a mapping table snapshot provided in this application embodiment;

[0024] Figure 4 A schematic diagram of the snapshot generation architecture provided in the embodiments of this application;

[0025] Figure 5 A schematic diagram of the sparse index snapshot structure provided in the embodiments of this application;

[0026] Figure 6 A schematic diagram of slot allocation provided for embodiments of this application;

[0027] Figure 7 A schematic diagram illustrating snapshot management provided in an embodiment of this application;

[0028] Figure 8 A schematic diagram illustrating WAL log expansion and recovery filtering provided in an embodiment of this application;

[0029] Figure 9 A schematic diagram of the metadata version chain structure provided in the embodiments of this application;

[0030] Figure 10 A schematic diagram illustrating fault recovery provided in an embodiment of this application;

[0031] Figure 11 A schematic diagram of a mapping table snapshot generation device provided in this application embodiment;

[0032] Figure 12 A schematic diagram of another mapping table snapshot generation device provided in this application embodiment;

[0033] Figure 13 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0034] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of this application.

[0035] It should be noted that, in the description of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. The terms "first," "second," etc., in this application are used to distinguish similar objects and are not used to describe a specific order or sequence.

[0036] For example, in high-availability storage environments within enterprise-level data centers, cloud computing platforms, and mission-critical systems (such as core financial transactions, virtualization clusters, and large relational databases), fully allocated volumes (thick provisioned volumes) are widely used due to their stable input / output (I / O) performance and high determinism in space pre-allocation. The metadata mapping from logical addresses to physical addresses directly determines the efficiency and consistency of data read and write operations.

[0037] When the primary storage control node goes down due to hardware failure, system crash, or network interruption, the backup node must immediately take over the business to ensure data continuity and service availability.

[0038] Since the capacity of a fully allocated volume typically reaches TB or even PB levels, and the mapping table is enormous, traditional recovery methods require a full scan to rebuild the index, resulting in excessively long business interruption times. This makes it difficult to meet the stringent requirements of modern core businesses for second-level recovery and strict recovery point targets.

[0039] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0040] The specific application environment architecture upon which the mapping table snapshot generation method depends is described here. (References) Figure 1 The primary control entity is used to issue storage service input / output requests. It receives these requests, maintains a logical-to-physical mapping table, and mirrors and synchronizes the service data to the backup control entity. The backup control entity receives the synchronized service data from the primary control entity; it only stores the service data and does not maintain the logical-to-physical mapping table. When the primary control entity fails, the backup control entity takes over the service operation.

[0041] In related technologies, a fault diagnosis mechanism based on heartbeat detection and IO timeout is adopted for a primary-backup high-availability storage system with fully allocated volumes. During normal operation, the primary control entity is responsible for handling host read and write requests and maintaining the logical-to-physical mapping table in memory. At the same time, it mirrors the written data blocks unidirectionally to the backup control entity through a high-speed interconnect channel. However, the backup entity does not maintain the index structure only to reduce synchronization overhead.

[0042] When a primary node failure triggers a switchover, the backup control entity must perform a full index reconstruction before taking over services. This involves traversing multiple mirrored data blocks in the shared storage area or local cache, extracting the logical block address (LBA) and physical address one by one to reconstruct the correspondence. Only after the mapping table is fully assembled in memory will the system officially start the service takeover process.

[0043] However, the single-stage recovery mode relies entirely on full data scanning, and the recovery time is linearly positively correlated with the volume capacity. Furthermore, the front-end business cannot respond during the reconstruction period, resulting in long-term service interruptions. At the same time, it is difficult to guarantee the Recovery Point Objective (RPO) in extreme cases.

[0044] Figure 2 This is a flowchart illustrating the mapping table snapshot generation method provided in the embodiments of this application, as shown below. Figure 2 As shown, embodiments of this application provide a method for generating a mapping table snapshot. The method is described in detail below:

[0045] S201. Obtain the logical-to-physical mapping table of the fully allocated volume and divide the logical-to-physical mapping table into data blocks of fixed size.

[0046] For example, when creating a fully allocated volume, the system pre-allocates physical storage space for all logical addresses of the volume and establishes a complete mapping relationship from logical address to physical address (Logical-to-Physical Mapping, or L2P for short), which is the logical-to-physical mapping table.

[0047] For example, the L2P mapping table of a fully allocated volume is organized in the form of entries, with each logical address corresponding to a mapping entry. The entry records the physical address that the logical address currently points to or indicates an invalid flag.

[0048] For example, to facilitate subsequent statistics and compression, the system divides the L2P mapping table into multiple data blocks of a fixed size. Each data block covers a continuous logical address range and contains a predetermined number of mapping entries. For example, each data block covers a predetermined logical space or contains a predetermined number of mapping entries. The data blocks are arranged continuously in the mapping table, which facilitates sequential traversal.

[0049] For example, using the same granularity for different snapshot periods of the same logical volume ensures that the boundaries of data blocks in consecutive snapshots are aligned. This allows for precise location of the corresponding content based on the block number when it is necessary to roll back along the version chain or perform incremental comparisons. The system assigns a unique block number to each data block, which can be used as the basis for traversal order or as a location field in the compressed index.

[0050] Optionally, the L2P mapping table can be obtained by the mapping management module reading the full mapping table from the mapping cache in memory or the persistent mapping storage, or by aggregating the scattered mapping entries into a continuous table structure through layer-by-layer address translation during the first access.

[0051] S202. Traverse the data blocks, count the number of non-empty data blocks and calculate the empty block rate.

[0052] For example, the system sequentially traverses all data blocks, reading the mapping entries contained in each data block one by one. For a fully allocated volume, although logical addresses are mapped during the creation phase, as upper-layer applications delete or overwrite, and storage system reclaims space, some physical blocks corresponding to logical addresses are released, and the mapping entries are marked as invalid. Therefore, not all data blocks contain valid mapping content.

[0053] The system identifies data blocks consisting entirely of invalid entries as empty data blocks, and data blocks containing at least one valid mapping as non-empty data blocks, and counts the number of the two types of data blocks accordingly.

[0054] For example, the empty block rate is calculated based on the number of empty data blocks and the number of non-empty data blocks. The empty block rate represents the proportion of empty data blocks in the entire L2P mapping table, reflecting the sparsity of the mapping table content.

[0055] With the help of scenario examples, the higher the empty block rate, the more invalid content in the mapping table can be compressed and skipped, and the greater the space gain of storing snapshots using sparse indexing; the lower the empty block rate, the denser the mapping table content, and the more limited the space that sparse compression can save.

[0056] For example, traversal and statistics can be completed in memory at once, or a pipelined approach can be used to perform statistics while traversing. During traversal, the mapping entries themselves do not need to be rewritten; only the valid tags of the entries are read. Therefore, traversal overhead is low, and the impact on the normal operation of the mapping table is minimal. After the system completes the statistics, it obtains the empty block rate used for adaptively selecting the snapshot format.

[0057] S203. When the empty block rate is greater than or equal to the preset ratio, perform compression processing on the non-empty data blocks to obtain compressed data blocks and index entries, and construct a sparse index snapshot structure based on the compressed data blocks and index entries.

[0058] For example, the system compares the empty block rate with a preset ratio, such as 30%. When the empty block rate is greater than or equal to the preset ratio, it determines that the L2P mapping table as a whole exhibits sparse characteristics and uses a sparse index snapshot structure to store the snapshot. At this time, the system only performs compression processing on non-empty data blocks; empty data blocks are skipped directly because they do not contain valid mapping content and are no longer written to the snapshot data area.

[0059] Optionally, the compression process can employ general lossless compression algorithms such as LZ4 or ZSTD, or it can use special encoding for the repetition pattern of the mapping entries, resulting in compressed data blocks with reduced size after compression.

[0060] For example, after compressed data blocks are written to the snapshot data area, the system generates corresponding index entries for each compressed data block. These index entries record the block number of the compressed data block in the original mapping table, as well as its storage location and size within the snapshot. This information is used to locate and restore any non-empty data block from the snapshot during recovery. By combining index entries with compressed data blocks, a sparse index snapshot structure is constructed. This structure only stores compressed data containing content and the indexes used to locate this compressed data. The logical intervals corresponding to empty blocks are treated as invalid content during recovery, thus preserving complete logical mapping information with a smaller snapshot size.

[0061] Optionally, the storage layout of the sparse index snapshot structure can be divided into a header area, an index area, and a data area. The header area stores global snapshot information, the index area stores all index entries contiguously, and the data area stores each compressed data block sequentially. Since both the index area and the data area are organized in order by block number, the compressed content of any data block can be quickly located based on the index entries during recovery, enabling on-demand reading and parallel reading.

[0062] S204. When the empty block rate is less than the preset ratio, the logic is directly copied to the physical mapping table using a dense format to obtain a dense snapshot structure.

[0063] For example, when the empty block rate is less than a preset ratio, the system determines that the mapping table content is sufficiently dense, with non-empty data blocks accounting for the vast majority. Compressing this dense data would save limited space and would instead introduce additional compression and decompression overhead. In this case, the system uses a dense format, directly copying the entire logical-to-physical mapping table to the snapshot slot, resulting in a dense snapshot structure. The data blocks in the dense snapshot structure maintain the same order and granularity as the original mapping table; the mapped content remains unchanged, the writing process is simple and direct, requiring no compression operations and generating no index entries.

[0064] For example, since the dense snapshot structure retains the original content of all data blocks, the data in the snapshot can be loaded back into memory as a whole during recovery, and the complete L2P mapping table can be quickly restored. This avoids the step of decompressing by index block as in the sparse method, and can achieve faster snapshot generation speed and lower recovery latency in dense mapping table scenarios.

[0065] For example, the system adaptively switches between sparse index snapshot structure and dense snapshot structure based on the quantitative indicator of empty block rate, so that the choice of snapshot format matches the actual content characteristics of the mapping table. When the mapping table is sparse, snapshots are saved with less storage overhead, and when the mapping table is dense, snapshots are saved with less computational overhead and higher read and write speed.

[0066] S205. Write the sparse index snapshot structure or dense snapshot structure to the pre-allocated shared storage area snapshot slot.

[0067] For example, multiple snapshot slots are pre-allocated in the shared storage area. Each snapshot slot has a predetermined capacity and a fixed slot identifier. The system determines the target snapshot slot for this snapshot write before generating the snapshot. After the sparse index snapshot structure or dense snapshot structure is built in memory, the system writes the header information, index information, or data content of the snapshot structure into the storage area corresponding to the target snapshot slot in layout order. Since the snapshot slots are pre-allocated, the writing process does not depend on dynamic space allocation at runtime, avoiding write failures caused by insufficient space or fragmentation.

[0068] For example, multiple generations of snapshots can be stored in a snapshot slot. The snapshots of different generations are organized through a version list. The head of the version list records the identifier of the latest valid snapshot slot and stores it at a preset offset position in the shared storage area.

[0069] For example, after the write is completed, the system updates the version list information so that the newly written snapshot becomes a valid snapshot that can be accessed by the recovery program. At the same time, the old generation snapshot can still be read by backtracking through the pointer, ensuring that if any generation snapshot is corrupted, it is possible to backtrack to the previous generation available snapshot along the version list.

[0070] For example, the snapshot writing process and the write request processing of the fully allocated volume are carried out concurrently. In order to ensure the consistency between the snapshot and the write operation, the snapshot boundary information can be recorded with the help of write-ahead logs while the snapshot is being written. This allows the logs to determine the order of the snapshot and subsequent write operations after an abnormal power outage, providing a consistent recovery benchmark for fault recovery.

[0071] For example, the shared storage area has pre-allocated snapshot slots with fixed capacity. The slot capacity is planned according to the maximum space that the snapshot may occupy, so as to avoid snapshot interruption due to failure of temporary space allocation during snapshot writing.

[0072] The mapping table snapshot generation method provided in this application embodiment obtains the logical-to-physical mapping table of a fully allocated volume and divides the logical-to-physical mapping table into data blocks of fixed size; iterates through the data blocks, counts the number of non-empty data blocks and calculates the empty block rate; when the empty block rate is greater than or equal to a preset ratio, it performs compression processing on the non-empty data blocks to obtain compressed data blocks and index entries, and constructs a sparse index snapshot structure based on the compressed data blocks and index entries; when the empty block rate is less than the preset ratio, it directly copies the logical-to-physical mapping table using a dense format to obtain a dense snapshot structure; and writes the sparse index snapshot structure or the dense snapshot structure into a pre-allocated shared storage area snapshot slot.

[0073] The above solution divides the logical-to-physical mapping table of the fully allocated volume into blocks and calculates the empty block rate. It adaptively selects a snapshot generation strategy based on the actual empty block distribution of the mapping table, resulting in a complete mapping table snapshot that can be directly loaded. Compared to the recovery method of reconstructing the mapping table from scratch, this approach eliminates the need to scan all data blocks during the fault recovery phase and rebuild the mapping index in real time. It allows for the direct loading of pre-generated snapshots to quickly restore the complete logical-to-physical mapping table, eliminating the high time-consuming overhead of large-capacity volume traversal and reconstruction, shortening fault recovery time, reducing business interruption time, and thus improving the stability of business operations.

[0074] Based on any of the above embodiments, the following, in conjunction with Figure 3 The detailed process of generating the mapping table snapshot is explained.

[0075] Figure 3 This is a flowchart illustrating another method for generating a mapping table snapshot provided in an embodiment of this application. Figure 3 As shown, the method includes the following steps:

[0076] S301. Obtain the logical-to-physical mapping table of the fully allocated volume and divide the logical-to-physical mapping table into data blocks of fixed size.

[0077] Below, in conjunction with Figure 4 The snapshot generation architecture is explained.

[0078] Figure 4 This is a schematic diagram of the snapshot generation architecture provided in an embodiment of this application. Figure 4 As shown, the primary control entity is the primary storage controller in the controller-level primary / standby strategy and the primary storage device in the device-level primary / standby strategy; the standby control entity is similar. The high-speed interconnect channel for controller-level primary / standby is the Peripheral Component Interconnect Express (PCIe) bus / shared memory. In the device-level primary / standby strategy, it is the Remote Direct Memory Access (RDMA) network (converged Ethernet RDMA / unlimited bandwidth). The controller-level shared storage area is the shared solid-state drive (SSD) memory within the chassis, and the device-level shared storage area is the flash cluster / Ethernet-connected flash cluster shared disk enclosure.

[0079] S302. Traverse the data blocks to obtain the entries for each data block.

[0080] For example, a table entry for a data block is a mapping entry contained within the data block. The system accesses each data block sequentially according to its block number, reads the mapping entries within each data block from the mapping cache or mapping storage, and associates the read table entries with the block number of the data block.

[0081] Optionally, during the traversal, the system can read table entries in batches at the block level. After each batch of data blocks is read, the validity of the entries is immediately determined without waiting for all data blocks to be read, thereby reducing memory usage during the traversal in a pipelined manner.

[0082] S303. In the table entries of each data block, the data blocks marked as invalid are determined to be empty data blocks.

[0083] For example, an invalid flag in a mapping entry indicates that the logical address is not currently mapped to a valid physical address, which may be caused by upper-level deletion or space reclamation operations. The system checks the flags of each entry in a data block one by one. If all entries in a data block are marked as invalid, the data block containing all invalid entries is determined to be an empty data block; as long as there is at least one entry in a data block that is not marked as invalid, the data block is not an empty data block.

[0084] For example, the determination of whether all data blocks have empty or non-empty attributes can be completed in a single traversal. The determination process only relies on the tag field of the entry and does not involve the parsing of the physical address content.

[0085] S304. Determine that all data blocks in the data block except for empty data blocks as non-empty data blocks.

[0086] For example, after determining whether a data block is empty, the remaining data blocks are non-empty data blocks. Each non-empty data block contains at least one valid mapping entry, which is the logical mapping information that needs to be actually saved in the snapshot. The process of determining non-empty data blocks does not require re-traversal; the system can mark the remaining data blocks as non-empty while traversing and determining empty data blocks, and record the block number of the non-empty data blocks as the data range for subsequent compression processing.

[0087] S305. Count the number of empty data blocks and the number of non-empty data blocks, and determine the empty block rate based on the number of empty data blocks and the number of non-empty data blocks.

[0088] For example, after completing the traversal, the system obtains the number of empty data blocks and the number of non-empty data blocks. The sum of the number of empty data blocks and the number of non-empty data blocks is the total number of data blocks. The empty block rate is equal to the ratio of the number of empty data blocks to the total number of data blocks, or it can be calculated by dividing the number of empty data blocks by the sum of the number of empty data blocks and the number of non-empty data blocks.

[0089] Optionally, the calculation process for the empty block rate can be represented by the following formula:

[0090]

[0091] in, Indicates the empty block rate. Number of non-empty blocks, Indicates the total number of blocks.

[0092] For example, if the total number of data blocks is 4096, of which 1600 are empty data blocks and 2496 are non-empty data blocks, then the empty block rate is approximately 39%.

[0093] For example, the empty block ratio is used to compare with a preset ratio to select the snapshot format. The preset ratio can be configured during storage system initialization or dynamically adjusted by the administrator based on the actual usage characteristics of the volume. The accuracy of the empty block ratio calculation depends on the granularity of the data block partitioning. The smaller the data block partitioning, the more refined the description of the sparsity of the mapping table by the empty block ratio.

[0094] Based on the above implementation method, by reading table entries block by block, determining empty data blocks based on invalid tags, distinguishing non-empty data blocks and counting their numbers to calculate the empty block rate, the system can obtain an accurate sparsity index in one sequential traversal. The statistical process does not rewrite the mapping content and has low overhead, thereby improving the stability of business execution.

[0095] S306. When the empty block rate is greater than or equal to the preset ratio, perform compression processing on the non-empty data blocks to obtain compressed data blocks and index entries, and construct a sparse index snapshot structure based on the compressed data blocks and index entries.

[0096] One feasible implementation method for performing compression processing on non-empty data blocks is as follows: performing compression processing on non-empty data blocks to obtain compressed non-empty data blocks; generating an index entry for the compressed non-empty data blocks, wherein the index entry includes at least one of the following: block number, compressed data offset, compressed size, and original size.

[0097] For example, the system only compresses data blocks that are determined to be non-empty; empty data blocks are not included in the compression process. The compression process is performed on a unit basis, with each non-empty data block being compressed independently to obtain the corresponding compressed non-empty data block.

[0098] For example, the system generates an index entry for each compressed non-empty data block. The block number in the index entry records the number of the data block in the original mapping table, the compressed data offset records the starting position of the compressed data block in the snapshot data area, the compressed size records the number of bytes in the compressed data block, and the original size records the number of bytes in the data block before compression.

[0099] For example, the block number is used to restore the compressed data block to the correct logical location during recovery; the compressed data offset allows the recovery program to directly address and read the specified compressed data block, supporting on-demand loading and parallel reading; the compressed size and the original size are used for buffer allocation during decompression and for statistics on the compression ratio. The index entry can also additionally record the type identifier of the compression algorithm so that the corresponding decompression algorithm can be selected during decompression.

[0100] With the example of a scenario, when the original size of a non-empty data block is 64KB and the size is 18KB after LZ4 compression, the index entry records the block number as 1024, the compressed data offset as the 3.5MB offset of the snapshot data area, the compressed size as 18KB, and the original size as 64KB. The recovery program can locate and restore the data block based on this index entry.

[0101] In this feasible implementation, only non-empty data blocks are compressed, and each compressed data block is described by an index item containing the block number, compressed data offset, compressed size and original size. This significantly reduces the snapshot size while ensuring the location and restoreability of the compressed data blocks, thereby improving the stability of business execution.

[0102] One feasible implementation method for constructing a sparse index snapshot structure is as follows: aggregating index items into an index array; writing snapshot global verification information and block statistics information into the snapshot header; and constructing a sparse index snapshot structure based on the snapshot header, the index array, and the compressed data blocks.

[0103] For example, the system arranges the index entries corresponding to all compressed data blocks in ascending order of block number and aggregates them into an index array. Each element in the index array corresponds to a compressed data block, and the entire index array is stored in the index area of ​​the snapshot structure.

[0104] For example, the system generates a snapshot header. The snapshot global verification information in the snapshot header is used to verify the integrity of the entire snapshot. The block statistics information is used to record metadata such as the total number of data blocks, the number of empty data blocks, the number of non-empty data blocks, and the number of compressed data blocks contained in the snapshot, so that the recovery program can perform a preliminary check before loading the snapshot.

[0105] For example, the snapshot header, index array, and compressed data blocks are organized sequentially according to the header area, index area, and data area, thus forming a sparse index snapshot structure. The recovery program first reads the snapshot header to complete verification and statistical checks, and then locates and decompresses the compressed data blocks in the data area block by block according to the index array, thereby reconstructing the complete L2P mapping table.

[0106] Optionally, the global check information in the header can be generated using Cyclic Redundancy Check (CRC) to detect bit flips or block loss that occur during snapshot storage.

[0107] Below, in conjunction with Figure 5 The sparse index snapshot structure is explained.

[0108] Figure 5 This is a schematic diagram of a sparse index snapshot structure provided in an embodiment of this application. Figure 5 As shown, the implementation steps of the sparse compression algorithm are as follows:

[0109] The mapping table is divided into fixed-size blocks, each block being BLOCK_SIZE=64KB, with a total number of blocks... .in, This indicates the size of each data block.

[0110] Iterate through all blocks and count the number of non-empty blocks (block_count). To determine if a block is empty, check if all entries within the block are 0 or have an invalid flag (0xFFFFFFFF).

[0111] Calculate empty block rate If empty_ratio < 0.3, then dense format is used for direct copying; otherwise, SIS sparse format is used. The 0.3 threshold can be adjusted according to the actual scenario.

[0112] Compress each non-empty block and record the index entry {block_id, offset, comp_size, orig_size}. The compression algorithm can be LZ4 or ZSTD.

[0113] Fill in the SIS Header and Snapshot Slot Header, calculate the checksum, and write it to disk.

[0114] After performing the above steps, a parallel decompression algorithm is required during recovery. This involves reading the SIS Header and index array, and then using multi-threaded parallel decompression of each block. Let the number of threads be num_threads, then each thread is responsible for decompression. Each block. After decompression, empty block positions will be filled with zeros.

[0115] In this feasible implementation, by aggregating index items into an index array and writing global snapshot verification information and block statistics information into the snapshot header, and then combining the snapshot header, index array and compressed data blocks in a fixed layout, the sparse index snapshot structure has self-description and self-verification capabilities. The recovery program can complete the integrity pre-check before loading and efficiently restore the mapping table based on the index array during the loading process, thereby improving the reliability and recovery efficiency of the snapshot and thus improving the stability of business execution.

[0116] S307. When the empty block rate is less than the preset ratio, the logic is directly copied to the physical mapping table using a dense format to obtain a dense snapshot structure.

[0117] It should be noted that the execution process of S307 is the same as that of S204, and will not be repeated here.

[0118] S308. Write the sparse index snapshot structure or dense snapshot structure to the pre-allocated shared storage area snapshot slot.

[0119] Below, in conjunction with Figure 6 Explanation of slot allocation.

[0120] Figure 6 This is a schematic diagram of slot allocation provided in an embodiment of this application. Figure 6 As shown, the snapshot slots in the shared storage area adopt a pre-allocation and circular reuse management strategy. Each volume reserves K snapshot slots (K≥2) in the shared storage area, and the size of each slot is the same as the active mapping table slot in the fixed_lp area (e.g., 32MB). Each slot maintains a status field, including: FREE: Free, can be used by new snapshots; VALID: Contains valid snapshot data; OBSOLETE: Replaced by an updated snapshot, awaiting reclamation.

[0121] When the snapshot condition is triggered, the snapshot management module scans the slot list of the current volume and looks for the first slot with a status of FREE. If there is no FREE slot, it selects the slot with the smallest version number in the OBSOLETE slot and overwrites it (i.e., FIFO eviction). The selected slot is marked as WRITING, and after the snapshot data is written, the status is updated to VALID.

[0122] Each VALID slot's header records a `prev_slot` field, pointing to the slot ID of the previous valid snapshot, forming a version linked list. The `prev_slot` of the latest snapshot points to the old snapshot, and the head of the link is indicated by the `active_slot` pointer.

[0123] Location during fault recovery: The recovery module reads the active_slot pointer and directly obtains the latest snapshot slot ID, without needing to scan all slots.

[0124] One feasible implementation method is to trigger snapshot storage by the following steps: after each full allocation volume write request is completed, determine the time difference between the current system time and the last snapshot generation time, and count the number of write operations since the last snapshot generation; when the time difference is greater than or equal to a time threshold, or the number of write operations is greater than or equal to a write operation count threshold, write the sparse index snapshot structure or dense snapshot structure to the pre-allocated shared storage area snapshot slot.

[0125] For example, the timing of snapshot generation is jointly controlled by the time dimension and the frequency of write operations. After each full-allocation volume write request is processed, the system reads the current system time and calculates the time difference between the current system time and the last snapshot generation time, while maintaining a counter of the cumulative write operations processed since the last snapshot generation.

[0126] When the time difference reaches a time threshold (e.g., triggered once every 10 minutes), or the number of write operations reaches a write operation threshold (e.g., accumulated to 1 million write operations), the system determines that a new snapshot needs to be generated and executes the aforementioned snapshot structure construction and writing process.

[0127] Optionally, the snapshot triggering condition can be represented by the following formula:

[0128]

[0129] in, This indicates that a snapshot has been triggered. Indicates the current moment. Indicates the last snapshot time. This indicates the number of write operations since the last snapshot. Indicates the time threshold. This represents the threshold for the number of write operations.

[0130] For example, using a dual-dimensional approach of time and write operation count can avoid frequent snapshot generation during periods of sparse write operations, while generating snapshots promptly during periods of intensive write operations to shorten the recovery point interval. The system performs the determination immediately after each write request is processed, ensuring that the determination timing is synchronized with the write operation rhythm, and that there is no long-term deviation between the snapshot generation time and the actual data state.

[0131] With the example of the scenario, when the number of write operations processed by the system within 60 seconds quickly exceeds the write operation count threshold, the system will immediately trigger the generation of a snapshot even though the time threshold has not yet been reached, thereby tightening the recovery point to the moment closer to the time before the failure occurred.

[0132] Below, in conjunction with Figure 7 Explanation of snapshot management.

[0133] Figure 7 This is a schematic diagram illustrating snapshot management as provided in an embodiment of this application. Figure 7 As shown, the snapshot threshold is configured by setting a time threshold at system initialization. and write count threshold Write a snapshot start flag (SNAPSHOT_START) to the Write-Ahead Logging (WAL) that includes the snapshot version number. and timestamp Copy the mapping table to the snapshot slot. Allocate a reserved snapshot slot from the fixed_lp area and copy the current LP mapping table to that slot. Sparse compression format storage is optional. Write the snapshot completion flag. After the snapshot is written to disk, write the snapshot completion SNAPSHOT_COMPLETE flag to the WAL, containing... , And metadata offset information. Update the version chain pointer, link the new snapshot slot to the version chain, the prev_slot field of the new snapshot points to the currently active slot, and then atomically update the version chain head so that active_slot points to the new slot. Increment SnapID, Prepare for the next snapshot cycle.

[0134] Below, in conjunction with Figure 8 This section explains the WAL log expansion and recovery filtering.

[0135] Figure 8 This is a schematic diagram illustrating WAL log expansion and recovery filtering provided in an embodiment of this application. Figure 8 As shown, the WAL index file is read into memory. A binary search is performed in the index array to locate the position of the last entry whose SnapID equals the target SnapID. If no exact match is found, the first position with a SnapID greater than the target SnapID is returned. Starting from that position, the WAL log is read sequentially, filtering out all write entries with SnapID ≥ N. The entries are sorted in ascending order of sequence number. Since the WAL is inherently ordered, no additional sorting is needed in a single-controller scenario. Each entry is then applied to the memory LP mapping table.

[0136] Parallel filtering in multi-controller scenarios: In multi-controller scenarios, a sliding window is used to parse the WAL logs of each controller in parallel. The window size is WINDOW_SIZE=10MB, and the number of threads is equal to the number of controllers.

[0137] In this feasible implementation, snapshots are generated by triggering the two conditions of time difference and number of write operations. This balances the frequency control of snapshot generation with the accuracy of recovery points, reducing snapshot storage overhead while ensuring the up-to-dateness of recoverable data during fault recovery, thereby improving the stability of business execution.

[0138] One feasible implementation method for writing the snapshot structure to the snapshot slot is as follows: generating a snapshot version identifier based on the time of the write operation; writing the snapshot version identifier to the snapshot header; and writing the sparse index snapshot structure or dense snapshot structure carrying the snapshot header to the pre-allocated shared storage area snapshot slot.

[0139] For example, when the system triggers snapshot generation, it generates a snapshot version identifier based on the time when the write operation occurs (i.e., the system timestamp at the time of snapshot generation). The snapshot version identifier can adopt a monotonically increasing sequence number, such as a combination of a global sequence number and a timestamp, to ensure that the versions of snapshots from different generations are comparable and distinguishable.

[0140] The system writes the snapshot version identifier into the snapshot header, so that the snapshot header carries version information along with the aforementioned global verification information and block statistics information.

[0141] For example, a snapshot header carrying a snapshot version identifier is written to the target snapshot slot along with the snapshot structure. The snapshot version identifier is used to determine the relationship between different generations of snapshots during recovery, and is also used to compare with the version information in the write-ahead log to determine the range of logs that need to be replayed.

[0142] In this feasible implementation, by writing a snapshot version identifier generated based on the writing time into the snapshot header, each generation of snapshots has a globally unique version mark. The recovery program can accurately select the target snapshot and determine the log playback boundary based on this, which provides a basis for multi-generation snapshot management and accurate recovery, thereby improving the stability of business execution.

[0143] One feasible implementation method for generating a mapping table snapshot further includes: obtaining a write-ahead log and writing a snapshot start marker into the write-ahead log; after writing to the pre-allocated shared storage area snapshot slot, generating a snapshot completion marker based on the snapshot version identifier, snapshot slot offset, and snapshot metadata information; writing the snapshot completion marker into the write-ahead log; and writing the write-ahead log into the write-ahead log area.

[0144] For example, the write-ahead log is a log recorded by the storage system before data is written to persistence, used for data replay after abnormal power outages or failures. During snapshot generation, the system first obtains the write-ahead log and writes a snapshot start marker into it, which identifies the starting point of the data state corresponding to this snapshot; after the snapshot is completely written to the snapshot slot, the system generates a snapshot completion marker based on the snapshot version identifier, snapshot slot offset, and snapshot metadata information, and appends the snapshot completion marker to the write-ahead log. Finally, the write-ahead log is written to the write-ahead log area for persistent storage.

[0145] For example, the snapshot start marker and snapshot completion marker define the snapshot generation process as a complete interval in the write-ahead log. By retrieving the snapshot start marker and snapshot completion marker in the write-ahead log, the recovery program can determine which generation of snapshots has been fully written and which write operations occurred after the snapshot and need to be replayed, thereby avoiding misclassifying incomplete snapshots as valid snapshots.

[0146] In this feasible implementation, by recording the snapshot start mark and snapshot completion mark in the write-ahead log and persisting the write-ahead log, a clear time boundary is established between snapshots and write operations. After an abnormal power outage, the completed snapshot and subsequent write data can be accurately distinguished, thereby improving the stability of business execution.

[0147] One feasible implementation method for generating a mapping table snapshot further includes: extracting multiple node information of key nodes in the write-ahead log, wherein the multiple node information includes at least one of the following: timestamp, snapshot version number, write-ahead log file offset, and entry number information; generating a write-ahead log index file based on the multiple node information; and storing the write-ahead log index file.

[0148] For example, the write-ahead log grows continuously as write operations continue. If the entire log is scanned sequentially from beginning to end during recovery, the recovery time will increase linearly with the log length. Therefore, the system extracts node information from key nodes in the write-ahead log during or after the write process. Key nodes can be the location where each snapshot is completed, or the segment boundaries after the write-ahead log file is divided into segments of a predetermined size.

[0149] The timestamp in the node information records the system time corresponding to the node, the snapshot version number records the snapshot version associated with the node, the write-ahead log file offset records the byte position of the node in the log file, and the entry count information records the number of log entries contained in the node's range.

[0150] For example, the system organizes all node information into a write-ahead log index file and stores it persistently. During recovery, the system first loads the write-ahead log index file and quickly locates the log range near the target snapshot version based on the node information, without having to scan all log data, significantly shortening the log location time.

[0151] In this feasible implementation, by extracting the node information of key nodes in the write-ahead log and generating a write-ahead log index file, an index structure that can be quickly queried is established for the write-ahead log, enabling the recovery program to skip irrelevant log intervals and directly locate the target position, thereby improving the stability of business execution.

[0152] One feasible implementation method for generating a mapping table snapshot further includes: obtaining the version list head stored at a preset offset position from the shared storage area, and obtaining a valid snapshot slot identifier from the version list head; reading the restored snapshot data from the snapshot slot corresponding to the valid snapshot slot identifier; obtaining verification information from the snapshot header of the restored snapshot data, performing integrity verification on the restored snapshot data based on the verification information to obtain a verification result, which is either verification passed or verification failed; if the verification result is verification passed, data recovery is performed using the restored snapshot data; if the verification result is verification failed, backtracking along the version chain using a pointer field, reading the data from the previous generation snapshot slot, and re-performing the integrity verification.

[0153] For example, the preset offset position of the shared storage area stores the version list head, which records the valid snapshot slot identifier in the current version chain, and the snapshots of each generation can be accessed along the version chain through the pointer field.

[0154] During fault recovery, the storage controller first reads the version list head from the shared storage area to obtain the latest valid snapshot slot identifier, and then reads the recovery snapshot data from that snapshot slot. After reading, the system obtains verification information from the snapshot header and performs integrity verification on the recovery snapshot data. If the verification result is successful, data recovery is directly performed based on that snapshot.

[0155] For example, if the latest generation snapshot is interrupted during writing or the storage medium is damaged, the integrity check will fail. In this case, the system rolls back along the version chain to the previous generation snapshot slot via a pointer field, reads the previous generation snapshot data, and re-executes the integrity check. Since the slots for each generation of snapshots in the version chain are pre-allocated and linked by pointers, the rollback operation can be repeated without destroying existing snapshots.

[0156] Below, in conjunction with Figure 9 Explain the metadata version chain structure.

[0157] Figure 9 This is a schematic diagram of the metadata version chain structure provided in an embodiment of this application. For example... Figure 9 As shown, when creating a new snapshot, the `prev_slot` field of the new snapshot is set to point to the currently active slot, then the new snapshot is written to the slot area, and finally the version list head is atomically updated. , .

[0158] When version rollback is required, recovery begins with the `active_slot`, sequentially reading and verifying each snapshot slot. Verification steps include: verifying the magic number field, verifying the header checksum, and verifying the data checksum. If the current snapshot verification fails, rollback proceeds along the `prev_slot` pointer to the previous snapshot. If all snapshots are corrupted, a traditional full rebuild is performed.

[0159] With the example scenario, when the verification result of the latest generation snapshot is that the verification fails, the system rolls back from the third generation snapshot to the second generation snapshot along the pointer field. After the second generation snapshot is verified, data recovery is started based on the second generation snapshot, and then the data is pushed to the latest state by combining the write-ahead log.

[0160] In this feasible implementation, a valid snapshot is located by the head of the version list and an integrity check is performed. When the latest snapshot is unavailable, the system automatically rolls back to the previous generation of available snapshots along the pointer field. This ensures that the recovery process can still obtain a consistent and usable recovery baseline even when the snapshot is partially damaged, thus improving the reliability of fault recovery.

[0161] One feasible implementation method for generating mapping table snapshots further includes: obtaining a write-ahead log index file; determining the index entry position corresponding to the target snapshot version using a binary search based on the write-ahead log index file; reading the write-ahead log from the index entry position and filtering out multiple write-ahead log entries whose snapshot version identifier is greater than or equal to the target snapshot version; and parsing the write-ahead logs corresponding to each controller in parallel using a sliding window method based on the multiple write-ahead log entries.

[0162] For example, after completing the snapshot loading, the system also needs to replay the write-ahead logs after the snapshot to advance the mapping table to the latest state before the failure. The system first obtains the write-ahead log index file, and determines the position of the index entry corresponding to the target snapshot version through binary search based on the node information recorded in the index entries. This position is the starting point of the log replay. Starting from this position, the system reads the write-ahead logs and filters out write-ahead log entries whose snapshot version identifier is greater than or equal to the target snapshot version. These entries record all write operations that occurred after the snapshot was generated.

[0163] For example, in a multi-controller architecture, write-ahead log entries generated by different controllers are interleaved and distributed throughout the log. A sliding window approach is used to parse the write-ahead logs corresponding to each controller in parallel, allowing multiple processor cores to process log entries from different windows, thus improving the parallelism of log parsing. The window size can be configured based on the number of available processor cores.

[0164] In this feasible implementation, the log starting point corresponding to the target snapshot version is located by binary search using the write-ahead log index file, and the write-ahead logs corresponding to each controller are parsed in parallel using a sliding window. This significantly reduces the log location time and improves the playback throughput, shortens the overall time for fault recovery, and thus improves the stability of business execution.

[0165] One feasible implementation method for determining the index entry position corresponding to the target snapshot version through binary search is as follows: performing a binary search in the index array of the write-ahead log index file; if an index entry matching the target snapshot version identifier exists in the index array, locating the index entry position based on the index entry; if no index entry matching the target snapshot version identifier exists in the index array, returning the index entry position greater than the target snapshot version identifier.

[0166] For example, the node information in the write-ahead log index file is arranged in order by snapshot version number to form an index array. The system performs a binary search in the index array, and each comparison reduces the search interval by half. The search time complexity is logarithmic.

[0167] If there is an index entry in the index array that exactly matches the target snapshot version identifier, the system will directly locate the index entry based on that index entry, that is, start replaying from the log offset corresponding to that entry.

[0168] For example, if a new log segment node has not been triggered after the snapshot corresponding to the target snapshot version identifier is generated, and there is no completely matching index entry in the index array, then the binary search returns the position of the most recent index entry greater than the target snapshot version identifier. Reading the log from this position can cover all write operations after the target snapshot, ensuring that no log content that needs to be replayed is missed.

[0169] In this feasible implementation, a binary search is performed on the ordered index array, and the position of the first index entry with a value greater than the target snapshot version is returned when no exact match is found. This makes the log replay starting point location both fast and complete, taking into account both location efficiency and the integrity of replay data, thereby improving the stability of business execution.

[0170] One feasible implementation method for generating the mapping table snapshot includes: after loading the snapshot and restoring the basic logic to the physical mapping table, taking over the host service read and write operations; replaying the selected write-ahead log entries to gradually advance the basic logic to the physical mapping table to the latest state before the failure; after the service read and write operations resume, writing the newly generated write requests into the recovery period write-ahead log; and merging the recovery period write-ahead log into the write-ahead log area after the historical write-ahead log entries have been replayed.

[0171] For example, after the system loads a snapshot and restores the underlying logic to the physical mapping table, it takes over the host's read and write operations. This allows the host to resume access before the mapping table is fully updated, shortening service interruption time. After takeover, the system replays the selected write-ahead log entries in the background, gradually pushing the underlying logic to the physical mapping table to its latest state before the failure.

[0172] For example, during the business read / write recovery process, newly generated write requests are written to the recovery period write-ahead log in addition to updating the mapping table normally, to avoid confusion between new write operations and historical logs being replayed. After all historical write-ahead log entries have been replayed, the system merges the recovery period write-ahead logs into the write-ahead log area, so that the log data is reorganized and used for subsequent recovery.

[0173] Below, in conjunction with Figure 10 Explanation of fault recovery.

[0174] Figure 10 This is a schematic diagram illustrating fault recovery provided in an embodiment of this application. Figure 10 As shown:

[0175] Step S501: Fault Detection and Service Takeover. The front-end host detects faults in the main control entity through a heartbeat mechanism: if there are N consecutive (N=3) IO timeouts without response, it is determined to be a fault, triggering a switchover, and the backup control entity takes over the service IP / port.

[0176] Step S502: Locate the latest complete snapshot scan WAL index file, use binary search to locate the last SNAPSHOT_COMPLETE marker, and obtain its version number SnapID=N. If no complete snapshot marker is found, downgrade to traditional full recovery.

[0177] Step S503: Load snapshot and quickly restore business (Phase 1) Read snapshot data sequentially from the fixed slot area of ​​the shared storage area according to the slot offset recorded in the snapshot metadata.

[0178] Verify the magic number, header checksum, and data checksum. If verification fails, roll back along the version chain.

[0179] Determine the snapshot format (dense or SIS). If it is SIS, invoke parallel decompression.

[0180] After loading the mapping table into memory, the standby control entity is immediately switched to an active state and begins receiving and processing host I / O requests. At this time, the mapping table state is... (Corresponding to the moment when SnapID=N).

[0181] Step S504: Background Asynchronous Replay of Incremental WAL (Phase Two) While business I / O recovers, a background task is started to filter all write entries with SnapID ≥ N from the WAL. These entries are sorted in ascending order by global sequence number (SN) (global merging is required in multi-controller scenarios). Each entry is applied to the memory-mapped table, changing the state from... Progressing to the latest status .

[0182] The background playback adopts a lock-free design (such as using atomic operations or segmented locks) and can dynamically adjust the playback rate according to the system load, minimizing the impact on the front-end business.

[0183] Step S505: After verifying and completing the incremental WAL playback, check the mapping relationship of key locations (such as random sampling verification) to confirm that the recovery is complete. The system then switches to normal dual-controller redundancy mode.

[0184] In this feasible implementation, by first taking over the host business read and write and then replaying the logs in the background, and merging the recovery period write-ahead logs with the historical write-ahead logs, the advancement of the mapping table and the business recovery are executed in parallel. This significantly shortens the business interruption window while ensuring data consistency, thereby improving the stability of business execution.

[0185] One feasible implementation method for generating a mapping table snapshot further includes: when generating a new snapshot, determining the corresponding new snapshot slot and pointing the pointer field of the new snapshot slot to the currently active snapshot slot; atomically updating the version list head to point the version list head to the new snapshot slot and updating the version list length information; and storing the updated version list head to a preset offset position in the shared storage area.

[0186] For example, when generating a new snapshot, the system first determines the new snapshot slot and points the pointer field of the new snapshot slot to the currently active snapshot slot, thereby connecting the new snapshot to the existing version chain to form a unidirectional linked list structure.

[0187] The version chain length increases with each new snapshot. After the update is complete, the system atomically sets the version chain head to the new snapshot slot. Atomic updates can be implemented using atomic compare-and-swap operations or lock-based atomic write operations, ensuring that no intermediate state is read when multiple controllers concurrently access the version chain head. Finally, the updated version chain head is stored at a preset offset in the shared memory area.

[0188] For example, multiple generations of snapshots are chained together through a pointer field. When the number of snapshots exceeds the capacity of the snapshot slot, the system can mark the snapshot with the smallest version number as pending retirement and reclaim its slot, thus realizing the intergenerational rolling of snapshots.

[0189] In this feasible implementation, the new snapshot slot is added to the version chain through a pointer field, and the version chain head is updated atomically, so that multiple generations of snapshots form a traceable and rollback chain structure. At the same time, atomic updates avoid version inconsistencies under concurrent access, thereby improving the stability of business execution.

[0190] One feasible implementation method for determining the corresponding new snapshot slot is as follows: determining the slot status field of the snapshot slot, which includes at least one of the following: idle state, valid state, and pending retirement state; when there is no idle snapshot slot, determining the snapshot slot with the smallest version number that is in the pending retirement state as the new snapshot slot; setting the selected slot to the writing state; and updating the slot status to valid state after the snapshot data has been completely written to the slot.

[0191] For example, each snapshot slot maintains a slot status field, which describes the current lifecycle state of the slot.

[0192] In this system, "idle" indicates that the slot is unused or has been reclaimed and is ready for rewriting; "valid" indicates that the snapshot data stored in the slot is complete and usable; and "pending retirement" indicates that the snapshot version in the slot is older and will be prioritized for reclamation and reuse when a new snapshot is generated and there are no free slots. When the system generates a new snapshot, it first searches for snapshot slots in an idle state, and if an idle snapshot slot exists, it is selected directly.

[0193] For example, when no snapshot slots are available, the system selects the snapshot slot with the smallest version number from among the snapshot slots in the expiring state as the new snapshot slot to evict the oldest snapshot. After selection, the slot status is set to "writing," indicating that snapshot data is being written to the slot to prevent misreading by other processes. Once the snapshot data is completely written to the slot and the header and verification information are written, the slot status is updated to "valid," and the slot becomes a valid snapshot that can be accessed by the recovery program.

[0194] With a scenario example, in the event of a failure, the mapping table is loaded from the snapshot slot, and the write-ahead log after the snapshot boundary marker is replayed. Let the volume capacity be V, the mapping table size be S, the incremental data volume within the snapshot interval be Δ, and the read bandwidth be BW. The recovery time for the relevant technical solution is:

[0195]

[0196] Recovery time in this application embodiment:

[0197]

[0198] in, Indicates the fault detection time. This indicates the bandwidth of a full disk scan of the image cache. Indicates the bandwidth for reading the write-ahead log. The bandwidth of sequential reading of snapshot slots in BW_seq is much higher than that of random scanning in BW_random, and Δ (incremental WAL data volume) is much smaller than V (full volume data volume), which can eliminate the huge overhead of full volume traversal.

[0199] With the example of the scenario, the implementation of this application divides the recovery process into two stages: Stage 1: Loading snapshot, ensuring the mapping table has basic integrity, and restoring services.

[0200] Phase Two: Replay incremental WAL and synchronize to the latest status.

[0201] Phase 1 takes only a fraction of the time depending on the size of the snapshot slot, which is typically 32MB. Sequential reads take less than 0.1 seconds. Phase 2 runs smoothly in the background without blocking host I / O, so the impact on business is minimal.

[0202] Using a scenario example, implement the following: divide the mapping table into fixed-size blocks, each block size BLOCK_SIZE=64KB, traverse all blocks, count the number of non-empty blocks (block_count), and calculate the empty block rate. If empty_ratio ≥ 0.3, a sparse index structure is used: only the compressed data of non-empty blocks and the block number index are stored.

[0203] Let the mapping table size be S, the empty block rate be r, and the compression ratio be c (typically 2-3 times that of LZ4), then the storage space of the sparse format is:

[0204]

[0205] S_index represents the overhead of the index array, which is 16 bytes per non-empty block. Through sparse index compression, approximately 70% of space is saved in scenarios with a high empty block rate, while also reducing loading bandwidth.

[0206] In this feasible implementation, the lifecycle of snapshot slots is managed through the slot status field. When there are no free slots, the slot with the smallest version number to be phased out is reclaimed. The writing process is protected by the writing status and marked as writing complete by the valid status. This achieves safe cyclic reuse of snapshot slots, which not only ensures that the slot capacity is fully utilized, but also avoids data misreading during the writing process, thereby improving the stability of business execution.

[0207] Figure 11 This is a schematic diagram of a mapping table snapshot generation device provided in an embodiment of this application. Figure 11As shown, the mapping table snapshot generation device 110 may include: an acquisition module 111, a statistics module 112, a sparse construction module 113, a dense construction module 114, and a writing module 115.

[0208] The acquisition module 111 is used to obtain the logical-to-physical mapping table of the fully allocated volume and divide the logical-to-physical mapping table into data blocks of fixed size.

[0209] The statistics module 112 is used to traverse data blocks, count the number of non-empty data blocks, and calculate the empty block rate.

[0210] The sparse construction module 113 is used to perform compression processing on non-empty data blocks when the empty block rate is greater than or equal to a preset ratio, to obtain compressed data blocks and index entries, and to construct a sparse index snapshot structure based on the compressed data blocks and index entries.

[0211] Dense building module 114 is used to directly copy logic to the physical mapping table in a dense format when the empty block rate is less than a preset ratio, so as to obtain a dense snapshot structure.

[0212] The write module 115 is used to write sparse index snapshot structures or dense snapshot structures to pre-allocated shared storage area snapshot slots.

[0213] Optionally, module 111 can be executed. Figure 2 S201 in the embodiment.

[0214] Optionally, the statistics module 112 can perform... Figure 2 S202 in the embodiment.

[0215] Optionally, sparse building block 113 can execute Figure 2 S203 in the embodiment.

[0216] Optionally, dense building block 114 can execute Figure 2 S204 in the embodiment.

[0217] Optionally, the write module 115 can perform... Figure 2 S205 in the embodiment.

[0218] In one possible implementation, the statistics module 112 is specifically used for:

[0219] Traverse the data blocks to obtain the entries for each data block.

[0220] In the table entries of each data block, the data blocks marked as invalid are determined to be empty data blocks.

[0221] All data blocks in the data block, excluding empty data blocks, are identified as non-empty data blocks.

[0222] Count the number of empty data blocks and the number of non-empty data blocks, and determine the empty block rate based on the number of empty data blocks and the number of non-empty data blocks.

[0223] In one possible implementation, the sparse construction module 113 is specifically used for:

[0224] Compress the non-empty data block to obtain the compressed non-empty data block.

[0225] Generate an index entry for the compressed non-empty data block. The index entry includes at least one of the following: block number, compressed data offset, compressed size, and original size.

[0226] In one possible implementation, the sparse construction module 113 is specifically used for:

[0227] Aggregate the index items into an index array.

[0228] Write global snapshot verification information and block statistics information to the snapshot header.

[0229] Construct a sparse index snapshot structure based on the snapshot header, index array, and compressed data blocks.

[0230] In one possible implementation, the writing module 115 is specifically used for:

[0231] After each full volume write request is processed, the time difference between the current system time and the last snapshot generation time is determined, and the number of write operations since the last snapshot generation is counted.

[0232] When the time difference is greater than or equal to the time threshold, or the number of write operations is greater than or equal to the number of write operations threshold, the sparse index snapshot structure or dense snapshot structure is written to the pre-allocated shared storage area snapshot slot.

[0233] In one possible implementation, the writing module 115 is specifically used for:

[0234] A snapshot version identifier is generated based on the time when the write operation occurs.

[0235] Write the snapshot version identifier to the snapshot header.

[0236] Write the sparse index snapshot structure or dense snapshot structure carrying the snapshot header to the pre-allocated shared storage snapshot slot.

[0237] It should be noted that the mapping table snapshot generation device shown in the embodiments of this application can execute the technical solution shown in the above method embodiments, and its implementation principle and beneficial effects are similar, so they will not be described again here.

[0238] Figure 12This is a schematic diagram of another mapping table snapshot generation device provided in an embodiment of this application. Figure 11 Based on the illustrated embodiments, as Figure 12 As shown, the mapping table snapshot generation device 110 also includes: a log recording module 116, an index generation module 117, a recovery module 118, a playback parsing module 119, a positioning module 120, a takeover module 121, a chain management module 122, and a slot determination module 123.

[0239] Log recording module 116 is used for:

[0240] Retrieve the write-ahead log and write the snapshot start marker into the write-ahead log.

[0241] After writing to the pre-allocated shared storage area snapshot slot, a snapshot completion marker is generated based on the snapshot version identifier, snapshot slot offset, and snapshot metadata information.

[0242] Write a snapshot completion flag to the write-ahead log.

[0243] Write the write-ahead log to the write-ahead log area.

[0244] Index generation module 117 is used for:

[0245] Extract multiple node information of key nodes in the write-ahead log. The multiple node information includes at least one of the following: timestamp, snapshot version number, write-ahead log file offset, and entry count information.

[0246] Generate a write-ahead log index file based on information from multiple nodes.

[0247] Store the write-ahead log index file.

[0248] Recovery module 118 is used for:

[0249] Retrieve the version list head saved at a preset offset position from the shared storage area, and retrieve the valid snapshot slot identifier from the version list head.

[0250] Read the recovery snapshot data from the snapshot slot corresponding to the valid snapshot slot identifier.

[0251] The verification information is obtained from the snapshot header of the restored snapshot data. The integrity of the restored snapshot data is verified based on the verification information to obtain the verification result, which is either verification passed or verification failed.

[0252] If the verification result is successful, data recovery is performed by restoring snapshot data.

[0253] If the verification result is that the verification fails, the pointer field is used to backtrack along the version chain, read the data of the previous generation snapshot slot, and re-execute the integrity verification.

[0254] Playback parsing module 119 is used for:

[0255] Retrieve the write-ahead log index file.

[0256] Based on the write-ahead log index file, the location of the index entry corresponding to the target snapshot version is determined by binary search.

[0257] Read the write-ahead log from the index entry position and filter out multiple write-ahead log entries whose snapshot version identifier is greater than or equal to the target snapshot version.

[0258] Based on multiple write-ahead log entries, the write-ahead logs corresponding to each controller are parsed in parallel using a sliding window approach.

[0259] Positioning module 120 is used for:

[0260] Perform a binary search within the index array of the write-ahead log index file.

[0261] If an index entry matching the target snapshot version identifier exists in the index array, locate the index entry position based on the index entry.

[0262] If no index entry matching the target snapshot version identifier exists in the index array, return the position of the index entry greater than the target snapshot version identifier.

[0263] The receiver module 121 is used for:

[0264] After loading the snapshot and restoring it to obtain the underlying logic to the physical mapping table, take over the host business read and write operations.

[0265] The pre-written log entries obtained from the screening are replayed to gradually advance the basic logic to the physical mapping table to the latest state before the failure.

[0266] After the read and write operations resume, newly generated write requests will be written to the recovery period write-ahead log.

[0267] After the historical write-ahead log entries are replayed, the recovery write-ahead logs are merged into the write-ahead log area.

[0268] Chain management module 122 is used for:

[0269] When a new snapshot is generated, the corresponding new snapshot slot is determined, and the pointer field of the new snapshot slot is set to the currently active snapshot slot.

[0270] The version list head is updated atomically to point to the new snapshot slot, and the version list length information is updated.

[0271] Store the updated version of the linked list head at a preset offset in the shared memory area.

[0272] Slot determination module 123 is used for:

[0273] Determine the slot status field of the snapshot slot. The slot status field includes at least one of the following: idle status, active status, and pending retirement status.

[0274] If no snapshot slot is available, the snapshot slot with the lowest version number that is in a pending phase-out state will be designated as the new snapshot slot.

[0275] Set the selected slot to the write-in status.

[0276] After the snapshot data is completely written to the slot, the slot status is updated to a valid state.

[0277] Figure 13 A schematic diagram of the structure of the electronic device provided in this application. Figure 13 As shown, the electronic device 130 provided in this embodiment includes at least one processor 1301 and a memory 1302. Optionally, the electronic device 130 also includes a communication component 1303. The processor 1301, the memory 1302, and the communication component 1303 are connected via a bus.

[0278] In a specific implementation, at least one processor 1301 executes computer execution instructions stored in memory 1302, causing at least one processor 1301 to execute the above-described mapping table snapshot generation method embodiment.

[0279] The specific implementation process of processor 1301 can be found in the above method embodiments, and its implementation principle and technical effect are similar. It will not be repeated here.

[0280] In the above embodiments, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in the application can be directly manifested as being executed by a hardware processor, or executed by a combination of hardware and software modules within the processor.

[0281] The memory may include random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage device.

[0282] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0283] Embodiments of this application also provide a non-volatile computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the steps in any of the above embodiments of the mapping table snapshot generation method when running.

[0284] In one exemplary embodiment, the aforementioned non-volatile computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0285] Embodiments of this application also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps in any of the above embodiments of the mapping table snapshot generation method.

[0286] Embodiments of this application also provide another computer program product, including a non-volatile computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps in any of the above embodiments of the mapping table snapshot generation method.

[0287] Any of the components, modules, units, parts, methods, and operations described herein can be implemented using software, firmware, hardware (e.g., fixed logic circuitry), manual processing, or any combination thereof. Alternatively or additionally, any functionality described herein can be performed at least in part by one or more hardware logic components, such as, but not limited to, a central processing unit (CPU), a field-programmable gate array (FPGA), an application-specific integrated circuit (ASIC), an application-specific standard product (ASSP), a system-on-a-chip (SoC), a complex programmable logic device (CPLD), a microprocessor (MCU), etc. The terms "system," "computing device," or "apparatus" as used herein encompass various means, devices, and machines for processing data, including, for example, one or more programmable processors, computers, SoCs, or combinations thereof. The apparatus may also include code that creates an execution environment for the computer program in question, such as code constituting processor firmware, a protocol stack, a database management system, an operating system, a cross-platform runtime environment, a virtual machine, or a combination thereof. The aforementioned computer program (also known as a program, software, software application, app, script, or code) can be written in any form of programming language, including compiled or interpreted languages, declarative or procedural languages, and can be deployed in any form, including as a standalone program or as a module, component, subroutine, object, or other unit suitable for a computing environment.

[0288] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0289] The above provides a detailed description of a mapping table snapshot generation method and electronic device provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only intended to help understand the method and core ideas of this application. It should be noted that those skilled in the art can make various improvements and modifications to this application without departing from its principles, and these improvements and modifications also fall within the protection scope of this application.

Claims

1. A method for generating a mapping table snapshot, characterized in that, include: Obtain the logical-to-physical mapping table of the fully allocated volume, and divide the logical-to-physical mapping table into data blocks of fixed size; Traverse the data blocks, count the number of non-empty data blocks and calculate the empty block rate; When the empty block rate is greater than or equal to a preset ratio, compression processing is performed on the non-empty data blocks to obtain compressed data blocks and index entries. A sparse index snapshot structure is then constructed based on the compressed data blocks and the index entries. When the empty block rate is less than the preset ratio, the logic is directly copied to the physical mapping table using a dense format to obtain a dense snapshot structure. Write the sparse index snapshot structure or the dense snapshot structure into the pre-allocated shared storage area snapshot slot.

2. The mapping table snapshot generation method according to claim 1, characterized in that, Traverse the data blocks, count the number of non-empty data blocks and calculate the empty block rate, including: Traverse the data blocks to obtain the table entries for each data block; In the table entries of each data block, the data blocks marked as invalid are determined to be empty data blocks; The data blocks other than the empty data blocks are identified as non-empty data blocks. The number of empty data blocks and the number of non-empty data blocks are counted, and the empty block rate is determined based on the number of empty data blocks and the number of non-empty data blocks.

3. The method for generating a mapping table snapshot according to claim 1, characterized in that, Compressing the non-empty data block yields compressed data blocks and index entries, including: Compress the non-empty data block to obtain the compressed non-empty data block; Generate an index entry for the compressed non-empty data block, the index entry including at least one of the following: block number, compressed data offset, compressed size, and original size.

4. The mapping table snapshot generation method according to claim 1, characterized in that, Constructing a sparse index snapshot structure based on the compressed data block and the index entry includes: Aggregate the index items into an index array; Write global snapshot verification information and block statistics information to the snapshot header; The sparse index snapshot structure is constructed based on the snapshot header, the index array, and the compressed data block.

5. The method for generating a mapping table snapshot according to claim 1, characterized in that, Writing the sparse index snapshot structure or the dense snapshot structure into a pre-allocated shared storage area snapshot slot includes: After each full volume write request is processed, the time difference between the current system time and the last snapshot generation time is determined, and the number of write operations since the last snapshot generation is counted. When the time difference is greater than or equal to the time threshold, or when the number of write operations is greater than or equal to the number of write operations threshold, the sparse index snapshot structure or the dense snapshot structure is written to the pre-allocated shared storage area snapshot slot.

6. The method for generating a mapping table snapshot according to claim 5, characterized in that, Writing the sparse index snapshot structure or the dense snapshot structure into the pre-allocated shared storage area snapshot slot includes: Generate a snapshot version identifier based on the time when the write operation occurred; Write the snapshot version identifier into the snapshot header; Write the sparse index snapshot structure or dense snapshot structure carrying the snapshot header into the pre-allocated snapshot slot in the shared storage area.

7. The method for generating a mapping table snapshot according to claim 1, characterized in that, The method further includes: Obtain the write-ahead log and write a snapshot start marker into the write-ahead log; After writing to the pre-allocated shared storage area snapshot slot, a snapshot completion marker is generated based on the snapshot version identifier, snapshot slot offset, and snapshot metadata information; Write the snapshot completion marker into the write-ahead log; Write the write-ahead log to the write-ahead log area.

8. The method for generating a mapping table snapshot according to claim 7, characterized in that, The method further includes: Extract multiple node information of key nodes in the write-ahead log, wherein the multiple node information includes at least one of the following: timestamp, snapshot version number, write-ahead log file offset, and entry number information; Generate a write-ahead log index file based on the information from the multiple nodes; Store the write-ahead log index file.

9. The method for generating a mapping table snapshot according to claim 8, characterized in that, The method further includes: Obtain the version list head stored at a preset offset position from the shared storage area, and obtain the valid snapshot slot identifier from the version list head; Read and restore snapshot data from the snapshot slot corresponding to the valid snapshot slot identifier; The verification information is obtained from the snapshot header of the restored snapshot data, and the integrity of the restored snapshot data is verified according to the verification information to obtain the verification result, which is either verification passed or verification failed. When the verification result is successful, data recovery is performed using the restored snapshot data; If the verification result is that the verification fails, the system will backtrack along the version chain using the pointer field, read the data of the previous generation snapshot slot, and re-execute the integrity verification.

10. The method for generating a mapping table snapshot according to claim 9, characterized in that, The method further includes: Obtain the write-ahead log index file; Based on the write-ahead log index file, the position of the index entry corresponding to the target snapshot version is determined by binary search. Read the write-ahead log from the index entry position and filter out multiple write-ahead log entries whose snapshot version identifier is greater than or equal to the target snapshot version; Based on the multiple write-ahead log entries, the write-ahead logs corresponding to each controller are parsed in parallel using a sliding window method.

11. The method for generating a mapping table snapshot according to claim 10, characterized in that, The location of the index entry corresponding to the target snapshot version is determined by binary search, including: Perform a binary search in the index array of the write-ahead log index file; If an index entry matching the target snapshot version identifier exists in the index array, the index entry position is located based on the index entry. If no index entry matching the target snapshot version identifier exists in the index array, the position of the index entry greater than the target snapshot version identifier is returned.

12. The method for generating a mapping table snapshot according to claim 10, characterized in that, The method further includes: After loading the snapshot and restoring it to obtain the basic logic to the physical mapping table, take over the host business read and write operations; The pre-written log entries obtained from the screening are replayed, and the basic logic to the physical mapping table is gradually advanced to the latest state before the failure. After the business read and write operations resume, newly generated write requests will be written to the recovery period write-ahead log; After the historical write-ahead log entries are replayed, the recovery period write-ahead logs are merged into the write-ahead log area.

13. The method for generating a mapping table snapshot according to any one of claims 1-12, characterized in that, The method further includes: When generating a new snapshot, the corresponding new snapshot slot is determined, and the pointer field of the new snapshot slot is set to the snapshot slot that is currently active. The version list head is updated atomically to point to the new snapshot slot, and the version list length information is updated. Store the updated version of the linked list head at a preset offset in the shared memory area.

14. The method for generating a mapping table snapshot according to claim 13, characterized in that, Determine the corresponding new snapshot slot, including: Determine the slot status field of the snapshot slot, wherein the slot status field includes at least one of the following: idle status, active status, and pending retirement status; When there are no idle snapshot slots, the snapshot slot with the smallest version number that is in a pending phase-out state is determined as the new snapshot slot; Set the selected slot to write status; After the snapshot data is completely written to the slot, the slot status is updated to a valid state.

15. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor, configured to implement the steps of the mapping table snapshot generation method as described in any one of claims 1 to 14 when executing the computer program.

Images