A metadata management method for deduplication

Abstract

The present application relates to a metadata management method for deduplication, which receives an I / O request, cuts the data into data blocks, and uses a hash algorithm to establish a unique fingerprint; in the fingerprint index structure, the fingerprint of the data block is queried; a transaction entry is created in the form of a bitmap, and the metadata manager scans the bitmap to obtain the transaction entry in a free state and assigns a globally unique Trans‑id; if a request that hits the fingerprint index is received, the logical block number accessed by the upper layer is mapped to the physical block number in the fingerprint map, and the reference count is increased by 1; if there is no request that hits the fingerprint index, a new free physical block number is assigned, a logical block number mapping is established, the mapping table and the reference count are written to the metadata area of ​​the PM, and the written data blocks are stored. In the present application, a transaction entry combined with a log mechanism is introduced to ensure the atomicity within the transaction entry and to ensure consistent metadata for deduplication in the event of a crash.

Description

Background Art

[0002] The development of modern information technology and the Internet generates a large amount of data every day. Society is entering an era of big data, and the data that needs to be managed is becoming more and more complex. Therefore, the development of deduplication technology solves some problems caused by the explosion of data.

[0003] Data deduplication is a popular data reduction technology that can effectively reduce the capacity of storage systems. It detects identical data objects in data streams based on the redundancy of the data itself, transmits and stores only one unique data object, and replaces other duplicate data copies with a pointer to the unique data object, thereby reducing network transmission and storage capacity requirements.

[0004] According to the granularity of deduplication operations, it can be divided into file-level and block-level deduplication. File-level deduplication ensures that files are not duplicated, while block-level deduplication requires files to be divided into data blocks for comparison. Block-level deduplication technology has become a mainstream research object because it is simple and efficient and can achieve a higher compression rate. As the amount of stored data increases, the difficulty of maintaining metadata for managing data increases.

[0005] In addition, metadata append, update, and delete operations need to be crash-consistent, otherwise unacceptable errors may occur in the event of a system crash or power outage. Figure 1 As shown in the figure, when processing repeated write requests for file A data block (LBNx) and file B data block (LBNy), due to system crash or power failure, the reference count of its physical block (PBNa) may fail to increase, and there are two duplicate data blocks in the system, but the reference count is 1. In subsequent operations, if file B is deleted and file C data is written, the reference count of data block PBNa will be set to zero, even if file A still needs to use this physical block. The system will recycle the physical block with a reference count of zero. If this physical block is assigned to LBNz of file C, this may cause the data block (LBNx) of file A that has not been deleted to be abnormally overwritten by the data block (LBNz) of file C. Finally, when the user accesses file A, the data of file C will be returned.

[0006] The above errors are caused by partially updating the deduplication metadata without providing crash consistency guarantee in the case of unexpected power loss or system crash. Therefore, the research on crash consistency of deduplication metadata will become very important.

[0007] It is worth noting that non-volatile memory has memory-level read and write latency and durability. This hardware feature is suitable for storing metadata information, which can effectively solve the conflict problems of deduplication and consistency. Summary of the invention

[0008] The present application provides a metadata management method for deduplication of data, so as to update metadata for deduplication of data which needs to ensure consistency in the event of a crash during an online process.

[0009] The technical solution of this application is as follows:

[0010] According to a first aspect of an embodiment of the present application, a metadata management method for deduplication is provided, the method comprising:

[0011] Receive an I / O request for data, cut the data into data blocks of fixed size, and perform fingerprint calculation using a hash algorithm to establish a unique fingerprint for the data block;

[0012] In the fingerprint index structure preloaded in the memory, query and compare the fingerprint of the data block to determine whether the same data block is hit;

[0013] Creating a transaction entry in a bitmap format, wherein one I / O request corresponds to one transaction entry, and the transaction entry is irrelevant to whether the fingerprint is matched;

[0014] Scanning the bitmap through the metadata manager to obtain a transaction entry in a free state, and assigning a globally unique Trans-id variable to the transaction entry;

[0015] If a request that hits the fingerprint index is received, the logical block number accessed by the upper layer is mapped to the physical block number in the fingerprint map, and the reference count of the physical block corresponding to the fingerprint is increased by 1;

[0016] If there is no request that hits the fingerprint index, a new free physical block number is allocated, a logical block number mapping is established, the I / O request is forwarded downward, and the mapping and reference count are written to the metadata area of ​​PM (persistent memory), and the written data block is stored, wherein the mapping also needs to be written to the corresponding transaction entry.

[0017] In some embodiments, the transaction entry includes a free state, a start state, and an end state.

[0018] In some embodiments, the transaction entry includes a Trans-id variable, a state variable, and at least one metadata update backup; wherein the state variable is used to characterize that the transaction entry is in one of the free state, start state, and end state mentioned above; the Trans-id variable is used as the serial number of the transaction entry and the order in which they are checkpointed.

[0019] In some embodiments, cutting the data into data blocks of fixed size specifically includes:

[0020] Based on the memory page and file system block size, set the deduplication data block size to 4KB.

[0021] In some embodiments, the memory layout related to deduplication metadata in the persistent memory is divided into a metadata area and a log area, wherein the metadata area stores the contents of the data block mapping table and the physical block reference count, and the log area is designed as a fixed-size area consisting of N transaction entries, and a bitmap is used to mark the usage of different entries.

[0022] According to a second aspect of an embodiment of the present application, another metadata management method for deduplication is provided, the method comprising:

[0023] Receive an I / O request for data, cut the data into data blocks of fixed size, and perform fingerprint calculation using a hash algorithm to establish a unique fingerprint for the data block;

[0024] In the fingerprint index structure preloaded in the memory, query and compare the fingerprint of the data block to determine whether the same data block is hit;

[0025] Creating a transaction entry in a bitmap format, wherein one I / O request corresponds to one transaction entry, and the transaction entry is irrelevant to whether the fingerprint is matched;

[0026] Scanning the bitmap through the metadata manager to obtain a transaction entry in a free state, and assigning a globally unique Trans-id to the transaction entry;

[0027] If a request that hits the fingerprint index is received, the logical block number accessed by the upper layer is mapped to the target physical block number in the fingerprint mapping, the reference count of the corresponding physical block is increased by 1, and the reference count of the original mapped physical block number is reduced;

[0028] If there is no request that hits the fingerprint index, the I / O request is forwarded downward, and the deduplication metadata is written to the metadata area of ​​the persistent memory, and the written data block is stored;

[0029] If the reference count of the physical block number of the original mapping is reduced to 0, the physical block number with the reference count of 0 is deleted.

[0030] In some embodiments, a transaction entry includes a free state, a start state, and an end state.

[0031] In some embodiments, the transaction entry includes a Trans-id variable, a state variable, and at least one metadata update backup; wherein the state variable is used to characterize that the transaction entry is in one of the free state, start state, and end state mentioned above; the Trans-id variable is used as the serial number of the transaction entry and the order in which they are checkpointed.

[0032] In some embodiments, cutting the data into data blocks of fixed size specifically includes:

[0033] Based on the memory page and file system block size, set the deduplication data block size to 4KB.

[0034] According to the third aspect of the embodiment of the present application, a non-volatile storage device is provided, including: a persistent memory storage; the persistent memory storage is used to implement the metadata management method for deduplication described above, and the persistent memory storage generally adopts a memory with good low latency and byte addressing; it can achieve speed and byte addressing close to the memory level. A metadata management method for deduplication involved in the present application can reduce a large amount of additional metadata I / O, whether at the time of writing or updating, introduces transaction entries combined with a log mechanism, ensures the atomicity within the transaction entries, and ensures consistent deduplication metadata in the event of a crash.

[0035] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] The drawings herein are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present application, and together with the specification are used to explain the principles of the present application, and do not constitute an improper limitation on the present application.

[0037] Figure 1 is a schematic diagram of a scenario in which data is damaged due to a system crash or power failure when processing repeated write requests for a file A data block (LBNx) and a file B data block (LBNy) according to an exemplary embodiment;

[0038] Figure 2 is a schematic diagram of a system architecture based on a non-volatile memory file system according to an exemplary embodiment;

[0039] Figure 3 It is a flowchart of a metadata management method for deduplication according to an exemplary embodiment;

[0040] Figure 4 is a schematic diagram showing the composition of a transaction entry according to an exemplary embodiment;

[0041] Figure 5 is a schematic diagram of a conversion process of three states of a transaction entry according to an exemplary embodiment;

[0042] Figure 6 It is a schematic diagram of an execution flow of a metadata management method for deduplication of data when writing new data according to an exemplary embodiment;

[0043] Figure 7 The diagram is a flow chart of a metadata management method for deduplication of data when writing new data according to an exemplary embodiment. DETAILED DESCRIPTION

[0044] In order to enable ordinary persons in the art to better understand the technical solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.

[0045] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application. On the contrary, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the attached claims.

[0046] like Figure 2 As shown, the present application exemplarily shows a schematic diagram of a system architecture based on a non-volatile memory file system.

[0047] The operating system generally divides the accessible memory space into two parts, one is the kernel space and the other is the user space. The kernel space is the area accessed by the operating system kernel, which is independent of ordinary applications and is a protected memory space. The user space is the memory area accessible by ordinary applications.

[0048] In the kernel space, the data deduplication system (hereinafter referred to as NC-DEDUP) consists of three parts: metadata management module (labeled as metadata manager in the figure), hash engine module and block interface module. Figure 2 As shown, NC-DEDUP is located at the bottom layer of the file system and the upper layer of the block device driver. In other words, the file system is the upper layer of NC-DEDUP, and the block device driver is the lower layer of NC-DEDUP. This structure ensures its good portability and provides good functional scalability. The specific functional modules are described as follows:

[0049] Metadata management module: Metadata mainly includes the mapping table from fingerprint index to PBN (physical block number), the mapping table from LBN (logical block number) to PBN (physical block number), and the reference count of PBN. The metadata management module is responsible for the allocation and recycling of metadata space, periodic persistence, fast query, and the execution of the log mechanism on PM (persistent memory) to ensure crash consistency.

[0050] Hash engine (Hash) module: Sequential reading of files for traversal search is very slow, so an index needs to be established. The most commonly used index is the hash table. If we create an index for the data in the file, the "Key" stores the value we want to query next time, and the "Value" corresponds to which file and which position. A fingerprint is selected based on the content of each block to identify it, where the block fingerprint can uniquely identify the data block. The hash algorithm is used to identify whether the data block is repeated, and the various hash methods in the deduplication are adjusted according to the different repetition rates to control the computing overhead.

[0051] Block interface module: Based on the information in the deduplication metadata of the block interface, the I / O (read / write) of the deduplicated data will be redirected or discarded.

[0052] In some embodiments, the metadata management of the present application is based on persistent memory (PM). The processing of any write request involves a series of modifications to the deduplication metadata. All possible metadata updates are divided into writing new blocks, duplicate blocks, and updating blocks. Writing new blocks means that the data block is not recorded, that is, the table entry cannot be found in the fingerprint index table. It is necessary to record the metadata information of this block for subsequent duplicate data operations, and it is necessary to add a fingerprint index, a LBN mapping, and a PBN reference count. Duplicate blocks refer to data blocks or logical block numbers that are not recorded, but fingerprint information is recorded. For such data blocks, the PBN of the unique data block is queried, and a mapping from a LBN to a unique PBN and a PBN reference count need to be updated. Update blocks mean that the data block is recorded, and the mapping relationship from LBN to PBN has been saved. When writing data, an LBN mapping needs to be updated, and two PBN reference counts and one or zero indexes need to be updated.

[0053] In a deduplication metadata management method provided by the present application, the implementation of NC-DEDUP is based on the open source framework Dmdedup, which inherits the bio processing of block I / O interface and block device. Dmdedup is an open source block-level deduplication and data deduplication module, which uses COW B-tree (copy on write) strategy to ensure the consistency of deduplication and data deduplication metadata.

[0054] like Figure 3 As shown, the method performs the following steps:

[0055] S11: receiving an I / O request for data, cutting the data into data blocks of a fixed size, and performing fingerprint calculation using a hash algorithm to establish a unique fingerprint for the data block.

[0056] In some embodiments, since the memory page and file system block sizes are usually 4KB, the deduplication data block size is set to 4KB, which can avoid block alignment problems and effectively reduce the overhead caused by block segmentation.

[0057] In some embodiments, the MD5 hash algorithm is used to calculate the data block fingerprint. The fingerprint size obtained by the traditional MD5 algorithm is 128 bits. Compared with other hash algorithms, the fingerprint index space can be effectively compressed.

[0058] S12: In the fingerprint index structure preloaded in the memory, the fingerprint of the data block is queried and compared to determine whether the same data block is hit.

[0059] S13: creating a transaction entry in a bitmap format, wherein one I / O request corresponds to one transaction entry, and the transaction entry is irrelevant to whether the fingerprint is matched.

[0060] In some embodiments, in the metadata management based on PM, the memory layout related to the deduplication metadata is mainly divided into a metadata area and a log area. The metadata area mainly stores the contents of the data block mapping table and the physical block reference count. Due to the low access latency and byte addressing characteristics of PM, access to deduplication metadata will be significantly accelerated. The log area is designed as a fixed-size area consisting of N transaction entries, and a bitmap is used to mark the usage of different entries.

[0061] In some embodiments, the number of transaction entries in the log area on the PM is set to 512. More transaction entries can cache more metadata updates, but will affect the identification of subsequent duplicate write requests.

[0062] S14: Scan the bitmap through the metadata manager to obtain the transaction entry in the free state, and allocate a globally unique Trans-id variable to the transaction entry.

[0063] In the embodiment of the present application, the transaction entry includes a free state, a start state and an end state.

[0064] In some embodiments, Figure 4As shown, each transaction entry includes a Trans-id, a state variable (status), and at least one metadata update backup (ref-op). The state variable is used to indicate that the transaction entry is in one of the above three states; the Trans-id variable is used as the serial number of the transaction entry and the order in which they are checked.

[0065] For each bio (block input output) write operation sent by the upper layer, all operations on deduplication metadata will be packaged into a transaction entry, modified and submitted uniformly. There is a checkpointed-id variable at the head of the PM space, which indicates the current maximum Trans-id.

[0066] like Figure 5 As shown, when the upper layer initiates an I / O write request, the metadata manager scans the bitmap, obtains a transaction entry in a free state (that is, a transaction entry marked as FREE), and converts the state of the transaction entry to BEGIN (that is, the start state). If no free entry is found in the bitmap, the operating system needs to reschedule the deduplication work thread and block the incoming read and write requests of the thread.

[0067] The transaction area is then filled with updated metadata operations and an I / O request is issued. When the next block device completes the I / O request, the driver of the next block device will issue an interrupt request and the status of the transaction will be converted to END (i.e., end state) in the device's software interrupt routine.

[0068] After each I / O operation on the data is completed, the metadata manager will start from the current checkpointed-id and check in id order whether there are transaction entries that can be checkpointed to the metadata area on the PM. After all eligible transactions are written to the metadata area, the transaction status is set to FREE and the flag in the bitmap is cleared.

[0069] The upper-level user program only needs to complete the write to the log data area on the PM and can return safely. Due to the low access latency of the PM, the log update operation can be completed quickly, so the write completion can be reported to the user quickly. When a system crash or power failure occurs, only the entries in the end state in the log area need to be scanned and updated to the metadata area.

[0070] S15: If a request for hitting the fingerprint index is received, the logical block number accessed by the upper layer is mapped to the physical block number in the fingerprint map, and the reference count of the physical block number corresponding to the fingerprint is increased by 1.

[0071] In some embodiments, when metadata corresponding to the physical block number is updated, it is written into an allocated transaction.

[0072] In some embodiments, the hit fingerprint index is used to indicate that the data currently undergoing I / O and the existing data are determined to be the same data block after fingerprint query.

[0073] For write requests that hit the fingerprint index, NC-DEDUP does not forward the upper-layer data to the I / O device, but records the updated deduplication metadata to the log area and then notifies the user that the I / O is completed.

[0074] S16: If there is no request that hits the fingerprint index, a new free physical block number is allocated, a logical block number mapping is established, the I / O request is forwarded downward, and the mapping and reference count are written to the metadata area of ​​PM (persistent memory), and the written data block is stored, wherein the mapping also needs to be written to the corresponding transaction entry.

[0075] like Figure 6 As shown, when NC-DEDUP receives a write request, it executes the above-mentioned metadata management method for deduplication. For a write request, NC-DEDUP adds the metadata update to a transaction to ensure the atomicity within the transaction entry. In addition, NC-DEDUP reduces a large amount of additional metadata I / O on the basis of ensuring data consistency.

[0076] like Figure 7 As shown, another metadata management method for deduplication provided by the present application is mainly aimed at the situation when NC-DEDUP updates data, including:

[0077] S21: receiving an I / O request for data, cutting the data into data blocks of a fixed size, and performing fingerprint calculation using a hash algorithm to establish a unique fingerprint for the data block.

[0078] S22: In the fingerprint index structure preloaded in the memory, query and compare the fingerprint of the data block to determine whether the same data block is hit.

[0079] S23: creating a transaction entry in a bitmap format, wherein one I / O request corresponds to one transaction entry, and the transaction entry is irrelevant to whether the fingerprint is matched.

[0080] In some embodiments, in the metadata management based on PM, the memory layout related to the deduplication metadata is mainly divided into a metadata area and a log area. The metadata area mainly stores the contents of the data block mapping table and the physical block reference count. Due to the low access latency and byte addressing characteristics of PM, access to deduplication metadata will be significantly accelerated. The log area is designed as a fixed-size area consisting of N transaction entries, and a bitmap is used to mark the usage of different entries.

[0081] S24: Scan the bitmap through the metadata manager to obtain the transaction entry in the free state, and allocate a globally unique Trans-id variable to the transaction entry.

[0082] In fact, the above four steps are relatively consistent with the execution steps when NC-DEDUP writes new data.

[0083] S25: If a request for a hit fingerprint index is received, the logical block number accessed by the upper layer is mapped to the target physical block number in the fingerprint mapping, the reference count of the corresponding physical block is increased by 1, and the reference count of the original mapped physical block number is reduced.

[0084] In some embodiments, when metadata corresponding to the target physical block number is updated, it is written into an allocated transaction.

[0085] For write requests that hit the fingerprint index, NC-DEDUP does not forward the upper-layer data to the I / O device, but records the updated deduplication metadata to the log area and then notifies the user that the I / O is completed.

[0086] S26: If there is no request that hits the fingerprint index, the I / O request is forwarded downward, and the deduplication metadata is written into the metadata area of ​​the PM, and the written data block is stored.

[0087] In some embodiments, the deduplication metadata includes a mapping table and a reference count.

[0088] S27: If the reference count of the physical block number of the original mapping is reduced to 0, the physical block number with the reference count of 0 is deleted.

[0089] In some embodiments, if the reference count of the PBN of the original mapping is reduced to 0, it means that no other file shares its data block. The storage space of the cleaned data block can be reclaimed through a garbage collection mechanism.

[0090] like Figure 7As shown in the figure, when NC-DEDUP updates data, NC-DEDUP adds the metadata update to a transaction to ensure the atomicity within the transaction entry. NC-DEDUP reduces a large amount of additional metadata I / O on the basis of ensuring data consistency. In addition, NC-DEDUP guarantees the order of metadata checking and data I / O, which can ensure that the deduplication metadata points to the correct and safe data block.

[0091] Compared with the execution method of existing deduplication technology when ensuring consistency between data, the deduplication metadata management method involved in the present application can reduce a large amount of additional metadata I / O both when writing and updating, introduce transaction entries combined with a log mechanism, ensure the atomicity within the transaction entries, and ensure consistent deduplication metadata in the event of a crash.

[0092] The application scenarios described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Ordinary technicians in this field can know that with the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0093] Those skilled in the art will appreciate that various aspects of the present application may be implemented as a system, method or program product. Therefore, various aspects of the present application may be specifically implemented in the following forms, namely: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software, which may be collectively referred to as "circuit", "module" or "system" herein.

[0094] In some possible implementations, the electronic device according to the present application may include at least one processor and at least one memory. The memory stores program code, and when the program code is executed by the processor, the processor executes the operation data management method according to various exemplary implementations of the present application described above in this specification. For example, the processor may execute the steps in the operation data management method.

[0095] In an exemplary embodiment, various aspects of a metadata management method for deduplication provided by the present application may also be implemented in the form of a program product, which includes program code. When the program product is run on a computer device, the program code is used to enable the computer device to execute the steps of the metadata management method for deduplication according to various exemplary embodiments of the present application described above in this specification.

[0096] It should be noted that, although several units or subunits of the device are mentioned in the above detailed description, this division is merely exemplary and not mandatory. In fact, according to the embodiments of the present application, the features and functions of two or more units described above can be embodied in one unit. Conversely, the features and functions of one unit described above can be further divided into multiple units to be embodied.

[0097] In addition, although the operations of the method of the present application are described in a specific order in the drawings, this does not require or imply that the operations must be performed in this specific order, or that all the operations shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps.

[0098] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0099] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable image scaling device to produce a machine, so that the instructions executed by the processor of the computer or other programmable image scaling device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0100] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable image scaling device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, the instruction device being implemented in the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0101] These computer program instructions may also be loaded onto a computer or other programmable image scaling device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, whereby the instructions executed on the computer or other programmable device provide for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0102] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present application.

[0103] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A metadata management method for deduplication, characterized in that: The method comprises: Receive an I / O request for data, cut the data into data blocks of fixed size, and perform fingerprint calculation using a hash algorithm to establish a unique fingerprint for the data block; In the fingerprint index structure preloaded in the memory, query and compare the fingerprint of the data block to determine whether the same data block is hit; Creating a transaction entry in a bitmap format, wherein one I / O request corresponds to one transaction entry, and the transaction entry is irrelevant to whether the fingerprint is matched; Scanning the bitmap through the metadata manager to obtain a transaction entry in a free state, and assigning a globally unique Trans-id variable to the transaction entry; If a request for a hit fingerprint index is received, the logical block number accessed by the upper layer is mapped to the physical block number in the fingerprint map, and the reference count of the physical block number corresponding to the fingerprint is increased by 1; If there is no request that hits the fingerprint index, a new free physical block number is allocated, a logical block number mapping is established, the I / O request is forwarded downward, and the mapping and reference count are written to the metadata area of ​​PM (persistent memory), and the written data block is stored, wherein the mapping also needs to be written to the corresponding transaction entry; The transaction entry includes a free state, a start state, and an end state; The transaction entry includes a Trans-id variable, a state variable, and at least one metadata update backup; wherein the state variable is used to characterize that the transaction entry is in one of the free state, start state, and end state mentioned above; the Trans-id variable is used as the serial number of the transaction entry and the order in which they are checkpointed.

2. The method according to claim 1, characterized in that: The step of cutting the data into data blocks of fixed size specifically includes: Based on the memory page and file system block size, set the deduplication data block size to 4KB.

3. The method according to claim 2, characterized in that The memory layout related to deduplication metadata in the persistent memory is divided into a metadata area and a log area, wherein the metadata area stores the contents of the data block mapping table and the physical block reference count, and the log area is designed as a fixed-size area consisting of N transaction entries, and a bitmap is used to mark the usage of different entries.

4. A metadata management method for deduplication, characterized in that: The method comprises: Receive an I / O request for data, cut the data into data blocks of fixed size, and perform fingerprint calculation using a hash algorithm to establish a unique fingerprint for the data block; In the fingerprint index structure preloaded in the memory, query and compare the fingerprint of the data block to determine whether the same data block is hit; Creating a transaction entry in a bitmap format, wherein one I / O request corresponds to one transaction entry, and the transaction entry is irrelevant to whether the fingerprint is matched; Scanning the bitmap through the metadata manager to obtain a transaction entry in a free state, and assigning a globally unique Trans-id variable to the transaction entry; If a request that hits the fingerprint index is received, the logical block number accessed by the upper layer is mapped to the target physical block number in the fingerprint mapping, the reference count of the corresponding physical block number is increased by 1, and the reference count of the original mapped physical block number is reduced; If there is no request that hits the fingerprint index, forward the I / O request downward, write the deduplication metadata into the metadata area of ​​the persistent memory, and store the written data block; If the reference count of the physical block number of the original mapping is reduced to 0, the physical block number with the reference count of 0 is deleted; Transaction entries include free state, start state, and end state; The transaction entry includes a Trans-id variable, a state variable, and at least one metadata update backup; wherein the state variable is used to characterize that the transaction entry is in one of the free state, start state, and end state mentioned above; the Trans-id variable is used as the serial number of the transaction entry and the order in which they are checkpointed.

5. The method according to claim 4, characterized in that The step of cutting the data into data blocks of fixed size specifically includes: Based on the memory page and file system block size, set the deduplication data block size to 4KB.

Images