Deduplication methods, apparatus and readable media for storage architecture

Abstract

This invention discloses a deduplication method, apparatus, and readable medium for storage architectures. It constructs a source address index structure and a fingerprint index bucket structure in collaboration with a memory cache. The source address index structure includes a temporary index cache with a source address bucket structure, a moderately accessed source address index structure, a cold-accessed source address index structure, and a frequently accessed source address index structure. The fingerprint index bucket structure includes a fingerprint index structure with a fingerprint bucket structure, a historical fingerprint index structure, a historical fingerprint Bloom filter structure, and a count Bloom filter. Access data is obtained, and a search is performed in the source address index structure and the fingerprint index bucket structure based on the access data. Deduplication is then performed based on the search results. This reduces the space and time overhead associated with traditional data deduplication in new storage architectures and improves the overall access performance of the storage system.

Description

Technical Field

[0001] This invention relates to the field of deduplication, and more specifically to a method, apparatus, and readable medium for deduplication of data in storage architectures. Background Technology

[0002] The ever-increasing data volume of data-intensive applications puts enormous pressure on storage systems, consuming significant amounts of storage space and causing I / O congestion. As an effective data reduction solution, real-time deduplication has been integrated into existing storage systems to reduce data storage pressure and improve access performance. Deduplication uses collision-resistant hash algorithms to calculate a unique fingerprint for each input data item to identify whether it is a duplicate of existing data. Traditional deduplication techniques, such as... Figure 1 As shown.

[0003] To achieve better performance, modern storage systems typically employ a caching architecture within their storage system hierarchy. This involves placing high-speed cache devices between DRAM and main storage. Because flash solid-state drives (SSDs) offer superior read / write performance compared to hard disk drives (HDDs), SSDs have been widely adopted as storage system cache devices, while HDDs, with their large capacity, are used as main storage. Integrating real-time deduplication into the storage system can further improve storage device performance and lifespan while reducing wear and tear on SSDs. However, with the emergence of new non-volatile storage media, such as 3D-XPoint and ZNAND, traditional SSDs in the storage hierarchy are gradually being replaced by these high-speed new media, and even battery-powered DRAM is being used as a cache device. Considering the different characteristics of various storage media, real-time deduplication operations should further consider their space and time overhead under new storage architectures. Summary of the Invention

[0004] In view of the aforementioned technical problems, the purpose of the embodiments of this application is to provide a deduplication method, apparatus, and readable medium for storage architectures to solve the technical problems mentioned in the background section.

[0005] In a first aspect, the present invention provides a deduplication method for storage architectures, comprising the following steps:

[0006] Construct a source address index structure and a fingerprint index bucket structure that are coordinated with memory cache. The source address index structure includes a temporary index cache with a source address bucket structure, a medium-access source address index structure, a cold-access source address index structure, and a frequently accessed source address index structure. The fingerprint index bucket structure includes a fingerprint index structure with a fingerprint bucket structure, a historical fingerprint index structure, a historical fingerprint Bloom filter structure, and a count Bloom filter.

[0007] Retrieve access data, search the source address index structure and fingerprint index bucket structure based on the access data, and perform duplicate data deletion operation based on the search results.

[0008] Preferably, the system searches the source address index structure and fingerprint index bucket structure based on the access data, and performs duplicate data deletion based on the search results, specifically including:

[0009] Obtain the source address and data content of the accessed data, and calculate the fingerprint information based on the data content;

[0010] Search the fingerprint index bucket structure based on the fingerprint information, and perform index entry migration management in the fingerprint index bucket structure based on the search results;

[0011] In response to the determination that fingerprint information identical to the fingerprint information exists in the fingerprint index bucket structure, the source address information of the accessed data is updated to the source address index structure, and no data is written at the same time;

[0012] In response to the determination that there is no fingerprint information identical to the fingerprint information in the fingerprint index bucket structure, a new index entry is generated based on the fingerprint information of the accessed data and added to the fingerprint index bucket structure.

[0013] The system searches the source address index structure based on the source address of the accessed data, and then performs migration management of the index entries in the source address index structure based on the search results.

[0014] In response to determining that a source address identical to the source address exists in the source address index structure, the cache physical address corresponding to the source address in the source address index structure is updated to the cache physical address corresponding to the source address accessing the data.

[0015] In response to the determination that there is no source address in the source address index structure that is identical to the source address, a new index entry is generated based on the source address of the accessed data and its corresponding cache physical address, and added to the source address index structure.

[0016] Preferably, the fingerprint information is searched within the fingerprint index bucket structure, and the search results are used to manage the migration of index entries within the fingerprint index bucket structure. Specifically, this includes:

[0017] Search the fingerprint index structure, historical fingerprint index structure, and historical fingerprint Bloom filter in sequence until the fingerprint information matches one of the index structures.

[0018] When fingerprint information matches the fingerprint index structure, its corresponding index entry is moved to the frequently accessed source address index structure; when fingerprint information matches the historical fingerprint index structure, its corresponding index entry is moved to the fingerprint index structure; when fingerprint information matches the historical fingerprint Bloom filter, its corresponding index entry is moved to the fingerprint index structure.

[0019] When a new index entry is added to the fingerprint index structure and the fingerprint index structure is full, the index entry with the lowest heat count in the fingerprint index structure is moved to the historical fingerprint index structure; when a new index entry is added to the historical fingerprint index structure and the historical fingerprint index structure is full, the index entry with the lowest heat count in the historical fingerprint index structure is moved to the historical fingerprint Bloom filter.

[0020] Preferably, the source address index structure is searched based on the source address of the accessed data, and the index entries are migrated and managed within the source address index structure based on the search results. This specifically includes:

[0021] Search the frequently accessed source address index structure, the temporary index cache, the moderately accessed source address index structure, and the cold accessed source address index structure in turn, until the source address of the accessed data hits one of the index structures.

[0022] When the source address hits the temporary index cache, the corresponding index entry is moved to the medium-access source address index structure; when the source address hits the cold-access source address index structure, the corresponding index entry is moved to the medium-access source address index structure; when the source address hits the medium-access source address index structure, the corresponding index entry is moved to the frequent-access source address index structure; when the source address hits the frequent-access source address index structure, the corresponding index entry is updated to the most frequently accessed position in the frequent-access source address index structure, and deduplication is skipped, and the data is directly written to the storage device.

[0023] When a new index entry is added to the frequently accessed source address index structure and the space of the frequently accessed source address index structure is full, the index entry with the lowest access frequency in the frequently accessed source address index structure is moved to the temporary index cache; when a new index entry is added to the temporary index cache and the space of the temporary index cache is full, the index entry with the lowest access frequency count in the temporary index cache is moved to the cold accessed source address index structure; when a new index entry is added to the medium accessed source address index structure and the space of the medium accessed source address index structure is full, the index entry with the lowest access frequency count in the medium accessed source address index structure is moved to the temporary index cache.

[0024] As a preferred approach, a management method coupling reverse index management and fingerprint index bucket structure is adopted to manage the dirty page list through dirty page data address indexing, specifically including:

[0025] For each fingerprint index structure, a dirty page list is maintained. A reverse mapping from cached physical address to source address and a one-to-one index management method that couples the dirty page list with the fingerprint index structure are used to manage the dirty page data address index.

[0026] Preferably, the source address bucket structure uses the source address as the key and the cache physical address and version number as the value; the fingerprint bucket structure uses fingerprint information as the key and the hot count, reference count, version number and source address as the value; the temporary index cache, the medium-access source address index structure and the frequent access source address index structure are stored in memory; the fingerprint index structure, the historical fingerprint index structure and the dirty page list are stored in memory; and the cold access source address index structure and the historical fingerprint Bloom filter structure are stored in the cache.

[0027] In a second aspect, the present invention provides a deduplication apparatus for storage architectures, comprising:

[0028] The index structure construction module is configured to build a source address index structure and a fingerprint index bucket structure that are coordinated with the memory cache. The source address index structure includes a temporary index cache with a source address bucket structure, a medium-access source address index structure, a cold-access source address index structure, and a frequently accessed source address index structure. The fingerprint index bucket structure includes a fingerprint index structure with a fingerprint bucket structure, a historical fingerprint index structure, a historical fingerprint Bloom filter structure, and a count Bloom filter.

[0029] The deletion operation module is configured to obtain access data, search in the source address index structure and fingerprint index bucket structure based on the access data, and perform duplicate data deletion operation based on the search results.

[0030] Thirdly, the present invention provides an electronic device including one or more processors; and a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any implementation of the first aspect.

[0031] Fourthly, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method as described in any of the implementations of the first aspect.

[0032] Compared with the prior art, the present invention has the following beneficial effects:

[0033] (1) The deduplication method for storage architecture proposed in this invention can reduce the space and time overhead caused by traditional data deduplication under new storage architecture and improve the overall access performance of storage system.

[0034] (2) The deduplication method for storage architecture proposed in this invention can improve the index retrieval efficiency of deduplication technology and improve the storage space utilization of storage system. Attached Figure Description

[0035] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0036] Figure 1 This is a schematic diagram of the traditional deduplication technology architecture;

[0037] Figure 2 This is a flowchart illustrating a storage-oriented deduplication method according to an embodiment of this application.

[0038] Figure 3 This is a schematic diagram of the storage structure of the fingerprint index bucket structure and the source address index structure during the deduplication operation of the storage architecture-oriented deduplication method according to an embodiment of this application.

[0039] Figure 4 This is a schematic diagram of the access flow of the fingerprint index bucket structure in the deduplication method for storage architecture according to an embodiment of this application;

[0040] Figure 5 This is a schematic diagram illustrating the access flow of the source address index structure in the storage architecture-oriented deduplication method according to an embodiment of this application.

[0041] Figure 6 This is a schematic diagram of a storage-oriented deduplication apparatus according to an embodiment of this application. Detailed Implementation

[0042] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this invention, and not all of them. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0043] Figure 2 An embodiment of this application illustrates a storage-oriented deduplication method, comprising the following steps:

[0044] S1. Construct a source address index structure and a fingerprint index bucket structure that are coordinated with a memory cache. The source address index structure includes a temporary index cache with a source address bucket structure, a medium-access source address index structure, a cold-access source address index structure, and a frequently accessed source address index structure. The fingerprint index bucket structure includes a fingerprint index structure with a fingerprint bucket structure, a historical fingerprint index structure, a historical fingerprint Bloom filter structure, and a count Bloom filter.

[0045] In a specific embodiment, the source address bucket structure uses the source address as the key and the cache physical address and version number as the value; the fingerprint bucket structure uses fingerprint information as the key and the hot count, reference count, version number, and source address as the value; the temporary index cache, the medium-access source address index structure, and the frequent-access source address index structure are stored in memory; the fingerprint index structure, the historical fingerprint index structure, and the dirty page list are stored in memory; and the cold-access source address index structure and the historical fingerprint Bloom filter structure are stored in the cache.

[0046] Specifically, this method is applied to data access computer storage systems and storage devices, including memory, cache, and underlying storage devices. (Reference) Figure 3The `hLBA` bucket represents a frequently accessed source address index structure, `TempBuffer` represents a temporary cache, `wLBA` bucket represents a moderately accessed source address index structure, `BloomFilter` represents a Bloom filter, `cLBA` bucket represents a cold accessed source address index structure, `C-BloomFilter` represents a counting Bloom filter, `FP` bucket represents a fingerprint index structure, `gFP` bucket represents a historical fingerprint index structure, `FP-BloomFilter` represents a historical fingerprint Bloom filter, `Dirty list` represents a dirty page list, `CA` represents a cached physical address, `LBA` represents a source address, `vID` represents a version number, `PRY` represents a hot count, and `RefCnt` represents a reference count. For the source address bucket structure, bucket numbers are assigned based on the weak hash prefix of the source address. For the fingerprint bucket structure, bucket numbers are assigned based on the weak hash prefix of the fingerprint information. Bucket numbering determines the bucket number based on the first few bits of the hash value, and the offset within the bucket is inserted sequentially. The contents of the bucket are managed according to the LRU (Least Recently Used) management strategy. Using weak hash prefixes can evenly distribute accessed data across different bucket structures, resulting in more balanced bucket access. For the physical address of the cache device in the fingerprint bucket structure, a mapping relationship that completely corresponds to the source address is adopted. That is, the cache data area is equally divided according to the fingerprint bucket structure, and the data page within each area corresponds to an entry in the fingerprint bucket structure. Therefore, the physical address of the cache device can be determined by the fingerprint bucket structure based on the bucket number and the offset within the bucket. This physical address of the cache device is stored in the corresponding source address bucket structure as the value in the corresponding source address index result. In addition, to maintain the cache management algorithm for the data within the bucket, an LRU list structure is maintained for each bucket structure.

[0047] Furthermore, the source address bucket structure and fingerprint bucket structure maintain the mapping relationship from the source address to the cache physical address and the mapping relationship from the fingerprint information to the cache physical address, respectively. First, by directly pointing the source address and fingerprint information to the cache device physical address, deduplication of write requests can be achieved by maintaining only one fingerprint. Both the source address and the cache physical address are 8 bytes in size; therefore, each index entry in the source address bucket structure can save 12 bytes of storage space. Second, by maintaining the mapping relationship from the source address to the cache physical address in the source address bucket structure, read requests can achieve read request address lookup by accessing the source address bucket structure once, thereby reducing the lookup time overhead of read requests. However, the above decoupled index structure has a consistency problem, namely, the separate management of the source address bucket structure and the fingerprint bucket structure leads to inconsistencies in the cache device physical addresses between indexes. For example, when a fingerprint bucket structure removes an index entry, to avoid the high-time-cost source address traversal, this method does not traverse the source address bucket structure, resulting in the corresponding cache physical address value in the source address bucket structure being unable to read valid data content. To address this consistency issue, version number information (Version ID, vID) is maintained in each of the two bucket structures, with each version number containing 4 bytes of information. Therefore, after performing a source address index structure lookup, if a source address index structure is hit, its corresponding cached physical address and vID are obtained. The bucket address and intra-bucket offset address of the fingerprint bucket structure are calculated using the cached physical address, thus obtaining the vID of the fingerprint bucket structure. By comparing the two vID values, it can be determined whether the currently hit source address index entry is valid. To address the vID maintenance issue, an incrementing method of 1 is used for management.

[0048] In addition, each index entry in the fingerprint bucket structure maintains a heat count (PRY) and a reference count (RefCnt) for the fingerprint information. For the heat count, when new fingerprint information hits the fingerprint index bucket structure, the heat count of the corresponding index entry is incremented by 1. For the reference count, when updating the source address index structure, if the source address of the current write request is not hit, the reference count of the index entry in the responding fingerprint bucket structure is incremented by 1 using the cached physical address (CA) of the source address bucket structure. When removing entries from the fingerprint bucket structure, the least recently referenced index entry is selected from the end of the LRU list. If its corresponding heat count is greater than 1, its heat count is decremented by 1, and the corresponding index entry is added to the head of the list. This process continues until an index entry with a heat count of 0 is found. In the coupled memory and cache management process, for the source address index structure, additional mapping relationships are stored in the cache device, and a weak hash algorithm is used to calculate the hash value of the source address for maintaining the address information of the counting Bloom filter. For the fingerprint index bucket structure, the two fingerprint management mechanisms first perform weak hash calculation on the fingerprint information to obtain historical fingerprint information. If a historical fingerprint index structure is hit, it indicates a high probability that the fingerprint information was removed prematurely. Therefore, when updating the fingerprint index bucket structure, its heat count is incremented by 2, thereby reducing its removal probability. If the space of the historical fingerprint index structure is full, the weak hash value of the historical fingerprint is calculated again and stored in the cache storage device as the address information of the counting Bloom filter. Therefore, if the counting Bloom filter is hit, it indicates a high probability that the historical fingerprint was removed prematurely. When updating the fingerprint index bucket structure, its heat count is incremented by 2, thereby reducing its removal probability. The above two coupled memory and cache management mechanisms fully consider the different design requirements of the two index structures. For the source address index structure, considering its many-to-one mapping relationship, only additional cold index entries are stored in the cache device, thereby reducing the consumption of memory space. For the fingerprint index bucket structure, considering its completely one-to-one mapping relationship, a cache management strategy based on historical fingerprint information is implemented by introducing a two-layer fingerprint management mechanism and hot counting, thereby improving the data deduplication rate.

[0049] S2, obtain access data, search in the source address index structure and fingerprint index bucket structure based on the access data, and perform duplicate data deletion operation based on the search results.

[0050] In a specific embodiment, the process involves searching the source address index structure and fingerprint index bucket structure based on the access data, and then performing a duplicate data deletion operation based on the search results. Specifically, this includes:

[0051] Obtain the source address and data content of the accessed data, and calculate the fingerprint information based on the data content;

[0052] Search the fingerprint index bucket structure based on the fingerprint information, and perform index entry migration management in the fingerprint index bucket structure based on the search results;

[0053] In response to the determination that fingerprint information identical to the fingerprint information exists in the fingerprint index bucket structure, the source address information of the accessed data is updated to the source address index structure, and no data is written at the same time;

[0054] In response to the determination that there is no fingerprint information identical to the fingerprint information in the fingerprint index bucket structure, a new index entry is generated based on the fingerprint information of the accessed data and added to the fingerprint index bucket structure.

[0055] The system searches the source address index structure based on the source address of the accessed data, and then performs migration management of the index entries in the source address index structure based on the search results.

[0056] In response to determining that a source address identical to the source address exists in the source address index structure, the cache physical address corresponding to the source address in the source address index structure is updated to the cache physical address corresponding to the source address accessing the data.

[0057] In response to the determination that there is no source address in the source address index structure that is identical to the source address, a new index entry is generated and added to the source address index structure based on the source address of the accessed data and its corresponding cache physical address.

[0058] In a specific embodiment, refer to Figure 4 The process involves searching the fingerprint index bucket structure based on the fingerprint information, and then managing the migration of index entries within the fingerprint index bucket structure based on the search results. Specifically, this includes:

[0059] Search the fingerprint index structure, historical fingerprint index structure, and historical fingerprint Bloom filter in sequence until the fingerprint information matches one of the index structures.

[0060] When fingerprint information matches the fingerprint index structure, its corresponding index entry is moved to the frequently accessed source address index structure; when fingerprint information matches the historical fingerprint index structure, its corresponding index entry is moved to the fingerprint index structure; when fingerprint information matches the historical fingerprint Bloom filter, its corresponding index entry is moved to the fingerprint index structure.

[0061] When a new index entry is added to the fingerprint index structure and the fingerprint index structure is full, the index entry with the lowest heat count in the fingerprint index structure is moved to the historical fingerprint index structure; when a new index entry is added to the historical fingerprint index structure and the historical fingerprint index structure is full, the index entry with the lowest heat count in the historical fingerprint index structure is moved to the historical fingerprint Bloom filter.

[0062] In a specific embodiment, refer to Figure 5 The process involves searching the source address index structure based on the source address of the accessed data, and then managing the migration of index entries within the source address index structure based on the search results. Specifically, this includes:

[0063] Search the frequently accessed source address index structure, the temporary index cache, the moderately accessed source address index structure, and the cold accessed source address index structure in turn, until the source address of the accessed data hits one of the index structures.

[0064] When the source address hits the temporary index cache, the corresponding index entry is moved to the medium-access source address index structure; when the source address hits the cold-access source address index structure, the corresponding index entry is moved to the medium-access source address index structure; when the source address hits the medium-access source address index structure, the corresponding index entry is moved to the frequent-access source address index structure; when the source address hits the frequent-access source address index structure, the corresponding index entry is updated to the most frequently accessed position in the frequent-access source address index structure, and deduplication is skipped, and the data is directly written to the storage device.

[0065] When a new index entry is added to the frequently accessed source address index structure and the space of the frequently accessed source address index structure is full, the index entry with the lowest access frequency in the frequently accessed source address index structure is moved to the temporary index cache; when a new index entry is added to the temporary index cache and the space of the temporary index cache is full, the index entry with the lowest access frequency count in the temporary index cache is moved to the cold accessed source address index structure; when a new index entry is added to the medium accessed source address index structure and the space of the medium accessed source address index structure is full, the index entry with the lowest access frequency count in the medium accessed source address index structure is moved to the temporary index cache.

[0066] Specifically, embodiments of this application propose a selective deduplication technique based on data update feature awareness, fully considering the impact of data update features on deduplication. If a duplicate data item is updated in a subsequent access, it will waste previous index update operations and cause additional subsequent index update operations, thus affecting system performance overhead. For each source address index structure, a frequently accessed source address index structure is maintained, using the source address as the index key. Simultaneously, the heat count of the source addresses within the frequently accessed source address index structure is maintained in list form. Before the deduplication operation, a source address lookup is performed first. Based on the source address lookup result, it is determined whether to perform the deduplication operation. During the deduplication operation, fingerprint information is searched first, followed by the source address lookup.

[0067] The read / write access process for frequently accessing the source address index structure can be described as follows:

[0068] During the write request access process, the data first undergoes a source address lookup. If a match is found, deduplication is not performed, and the source address is updated in the frequently accessed source address index structure. If a match is not found, deduplication is executed.

[0069] In the read request access process, the data uses the source address to first look up the frequently accessed source address index structure. If a match is found, the corresponding cache physical address is returned.

[0070] Selective deduplication will result in more data being written to the cache storage device. However, modern storage devices generally feature high performance and high durability. Therefore, fully utilizing the high durability of modern storage devices will help leverage their performance advantages. Furthermore, selective deduplication is strongly coupled with the source address index structure. Frequent access to the eviction index entries within the source address index structure will continue to execute deduplication, thus minimizing the problem of duplicate data being written multiple times.

[0071] In a specific embodiment, a management method coupling reverse index management and fingerprint index bucket structure is used to manage the dirty page list by indexing the dirty page data address, specifically including:

[0072] For each fingerprint index structure, a dirty page list is maintained. A reverse mapping from cached physical address to source address and a one-to-one index management method that couples the dirty page list with the fingerprint index structure are used to manage the dirty page data address index.

[0073] Specifically, in write-back scenarios, the dirty page list incurs significant space and traversal overhead. To address this, the one-to-one and many-to-one mapping relationships are decoupled. The one-to-one mapping is maintained by the fingerprint index, while the many-to-one mapping remains maintained by the dirty page list, thus forming a data management mechanism across multiple devices for managing the dirty page data address index. Furthermore, when a write-back strategy is used across multiple devices, a reverse mapping based on cached physical addresses to source addresses is constructed. A one-to-one index management method coupling the dirty page data address index and the fingerprint index structure is employed. This allows the index entries showing the cached physical address to the latest source address after duplicate data deletion to be stored in the fingerprint index structure, thereby reducing the space and traversal overhead of the dirty page data address index. When the fingerprint index structure is full, the dirty page data address index is found using the cached physical address corresponding to the index entry with the lowest hot count, and the data is read from the cache space and written to the underlying storage device.

[0074] For maintaining a one-to-one mapping, the bucket structure of the fingerprint index maintains additional source addresses, which mark the most recently accessed source address for the current fingerprint information. Therefore, when the bucket structure in the fingerprint index is full, the eviction process can directly obtain the corresponding source address, thus writing the data from the cache storage device back to the main storage device. For maintaining a many-to-one mapping, the dirty page list uses a mapping method from cache physical addresses to source addresses; that is, each cache physical address maintains a set of source addresses it references. Therefore, during the eviction process, by obtaining the cache physical address, all referenced source addresses can be retrieved, avoiding the source address traversal process.

[0075] The steps S1-S2 above do not represent the order of the steps, but are merely symbolic representations of the steps.

[0076] Further reference Figure 6 As an implementation of the methods shown in the above figures, this application provides an embodiment of a deduplication apparatus for storage architectures, which is similar to... Figure 2 Corresponding to the method embodiments shown, this device can be specifically applied to various electronic devices.

[0077] This application provides a deduplication apparatus for storage architecture, comprising:

[0078] The index structure construction module 1 is configured to construct a source address index structure and a fingerprint index bucket structure that are coordinated with a memory cache. The source address index structure includes a temporary index cache with a source address bucket structure, a medium-access source address index structure, a cold-access source address index structure, and a frequently accessed source address index structure. The fingerprint index bucket structure includes a fingerprint index structure with a fingerprint bucket structure, a historical fingerprint index structure, a historical fingerprint Bloom filter structure, and a count Bloom filter.

[0079] The deletion operation module 2 is configured to obtain access data, search in the source address index structure and fingerprint index bucket structure based on the access data, and perform duplicate data deletion operation based on the search results.

[0080] It should be noted that the computer-readable medium described in this application can be a computer-readable signal medium, a computer-readable medium, or any combination thereof. A computer-readable medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor device, or any combination thereof. More specific examples of a computer-readable medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this application, a computer-readable medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution device, apparatus, or device. In this application, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable medium other than a computer-readable medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution device, apparatus, or apparatus. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0081] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or it can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0082] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using dedicated hardware-based means to perform the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0083] The modules described in the embodiments of this application can be implemented in software or hardware. These modules can also be located within a processor.

[0084] In another aspect, this application also provides a computer-readable medium, which may be included in the electronic device described in the above embodiments; or it may exist independently and not assembled into the electronic device. The computer-readable medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to: construct a memory cache-coordinated source address index structure and a fingerprint index bucket structure; the source address index structure includes a temporary index cache with a source address bucket structure, a moderately accessed source address index structure, a cold-accessed source address index structure, and a frequently accessed source address index structure; the fingerprint index bucket structure includes a fingerprint index structure with a fingerprint bucket structure, a historical fingerprint index structure, a historical fingerprint Bloom filter structure, and a count Bloom filter; acquire access data; search in the source address index structure and the fingerprint index bucket structure based on the access data; and perform duplicate data deletion operations based on the search results.

[0085] The above description is merely a preferred embodiment of this application and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in this application is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described inventive concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features with similar functions disclosed in this application.

Claims

1. A deduplication method for storage architecture, characterized in that, Includes the following steps: Construct a source address index structure and a fingerprint index bucket structure that are coordinated with a memory cache. The source address index structure includes a temporary index cache with a source address bucket structure, a medium-access source address index structure, a cold-access source address index structure, and a frequently accessed source address index structure. The fingerprint index bucket structure includes a fingerprint index structure with a fingerprint bucket structure, a historical fingerprint index structure, a historical fingerprint Bloom filter structure, and a count Bloom filter. The process involves acquiring access data, searching the source address index structure and fingerprint index bucket structure based on the access data, performing duplicate data deletion based on the search results, searching the source address index structure based on the source address of the access data, and performing index entry migration management based on the search results within the source address index structure. Specifically, this includes: The frequent access source address index structure, temporary index cache, medium access source address index structure and cold access source address index structure are searched sequentially until the source address of the accessed data matches one of the index structures. When the source address hits the temporary index cache, the corresponding index entry is moved to the medium-access source address index structure; when the source address hits the cold-access source address index structure, the corresponding index entry is moved to the medium-access source address index structure; when the source address hits the medium-access source address index structure, the corresponding index entry is moved to the frequent-access source address index structure; when the source address hits the frequent-access source address index structure, the corresponding index entry is updated to the most frequently accessed position in the frequent-access source address index structure, and deduplication is skipped, and the data is directly written to the storage device. When a new index entry is added to the frequently accessed source address index structure and the space of the frequently accessed source address index structure is full, the index entry with the lowest access frequency in the frequently accessed source address index structure is moved to the temporary index cache; when a new index entry is added to the temporary index cache and the space of the temporary index cache is full, the index entry with the lowest access frequency count in the temporary index cache is moved to the cold accessed source address index structure; when a new index entry is added to the medium accessed source address index structure and the space of the medium accessed source address index structure is full, the index entry with the lowest access frequency count in the medium accessed source address index structure is moved to the temporary index cache.

2. The deduplication method for storage architecture according to claim 1, characterized in that, The process of searching the source address index structure and fingerprint index bucket structure based on the access data, and performing duplicate data deletion based on the search results, specifically includes: obtaining the source address and data content of the access data, and calculating fingerprint information based on the data content; searching the fingerprint information in the fingerprint index bucket structure, and performing index entry migration management in the fingerprint index bucket structure based on the search results; in response to determining that fingerprint information identical to the fingerprint information exists in the fingerprint index bucket structure, updating the source address information of the access data to the source address index structure, while not writing data; in response to determining that fingerprint information identical to the fingerprint information does not exist in the fingerprint index bucket structure, performing duplicate data deletion based on the access data... A new index entry is generated from the fingerprint information of the accessed data and added to the fingerprint index bucket structure. The source address of the accessed data is searched in the source address index structure, and the index entry is migrated and managed based on the search result. In response to determining that a source address identical to the source address exists in the source address index structure, the cache physical address corresponding to the source address in the source address index structure is updated to the cache physical address corresponding to the source address of the accessed data. In response to determining that no source address identical to the source address exists in the source address index structure, a new index entry is generated based on the source address of the accessed data and its corresponding cache physical address, and added to the source address index structure.

3. The deduplication method for storage architecture according to claim 2, characterized in that, The process of searching the fingerprint information in the fingerprint index bucket structure and managing the migration of index entries in the fingerprint index bucket structure based on the search results specifically includes: sequentially searching the fingerprint index structure, the historical fingerprint index structure, and the historical fingerprint Bloom filter until the fingerprint information matches one of the index structures; when the fingerprint information matches the fingerprint index structure, its corresponding index entry is moved to the frequently accessed source address index structure; when the fingerprint information matches the historical fingerprint index structure, its corresponding index entry is moved to the fingerprint index structure; when the fingerprint information matches the historical fingerprint Bloom filter, its corresponding index entry is moved to the fingerprint index structure; when a new index entry is added to the fingerprint index structure and the fingerprint index structure space is full, the index entry with the lowest heat count in the fingerprint index structure is moved to the historical fingerprint index structure; when a new index entry is added to the historical fingerprint index structure and the historical fingerprint index structure space is full, the index entry with the lowest heat count in the historical fingerprint index structure is moved to the historical fingerprint Bloom filter.

4. The deduplication method for storage architecture according to claim 1, characterized in that, The management method of using reverse index management coupled with fingerprint index bucket structure to perform dirty page data address index management specifically includes: maintaining a dirty page list for each fingerprint index structure, using reverse mapping from cached physical address to source address and a one-to-one index management method coupled with the dirty page list and fingerprint index structure to perform dirty page data address index management.

5. The deduplication method for storage architecture according to claim 1, characterized in that, Also includes: For each fingerprint index structure, a dirty page list is maintained. A reverse mapping from cached physical address to source address and a one-to-one index management method that couples the dirty page list with the fingerprint index structure are used to manage the dirty page data address index.

6. The deduplication method for storage architecture according to claim 5, characterized in that, The source address bucket structure uses the source address as the key and the cache physical address and version number as the value. The fingerprint bucket structure uses the fingerprint information as the key and the hot count, reference count, version number, and source address as the value. The temporary index cache, the medium-access source address index structure, and the frequent-access source address index structure are stored in memory. The fingerprint index structure, the historical fingerprint index structure, and the dirty page list are stored in memory. The cold-access source address index structure and the historical fingerprint Bloom filter structure are stored in the cache.

7. A deduplication device for storage architecture, characterized in that, include: The index structure construction module is configured to construct a source address index structure and a fingerprint index bucket structure that are coordinated with a memory cache. The source address index structure includes a temporary index cache with a source address bucket structure, a medium-access source address index structure, a cold-access source address index structure, and a frequently accessed source address index structure. The fingerprint index bucket structure includes a fingerprint index structure with a fingerprint bucket structure, a historical fingerprint index structure, a historical fingerprint Bloom filter structure, and a count Bloom filter. The deletion operation module is configured to acquire access data, search the source address index structure and fingerprint index bucket structure based on the access data, perform duplicate data deletion based on the search results, search the source address index structure based on the source address of the access data, and perform index entry migration management based on the search results in the source address index structure, specifically including: The frequent access source address index structure, temporary index cache, medium access source address index structure and cold access source address index structure are searched sequentially until the source address of the accessed data matches one of the index structures. When the source address hits the temporary index cache, the corresponding index entry is moved to the medium-access source address index structure; when the source address hits the cold-access source address index structure, the corresponding index entry is moved to the medium-access source address index structure; when the source address hits the medium-access source address index structure, the corresponding index entry is moved to the frequent-access source address index structure; when the source address hits the frequent-access source address index structure, the corresponding index entry is updated to the most frequently accessed position in the frequent-access source address index structure, and deduplication is skipped, and the data is directly written to the storage device. When a new index entry is added to the frequently accessed source address index structure and the space of the frequently accessed source address index structure is full, the index entry with the lowest access frequency in the frequently accessed source address index structure is moved to the temporary index cache; when a new index entry is added to the temporary index cache and the space of the temporary index cache is full, the index entry with the lowest access frequency count in the temporary index cache is moved to the cold accessed source address index structure; when a new index entry is added to the medium accessed source address index structure and the space of the medium accessed source address index structure is full, the index entry with the lowest access frequency count in the medium accessed source address index structure is moved to the temporary index cache.

8. An electronic device, comprising: One or more processors; A storage device for storing one or more programs, which, when executed by one or more processors, cause the one or more processors to perform the method as described in any one of claims 1-6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1-6.

Images