A block-level data backup management method and system for S3 object storage

Abstract

The application provides a block-level data backup management method and system for S3 object storage, relates to the technical field of data backup management, and comprises the following steps: calculating the content fingerprint of a current data block to be backed up; inputting the content fingerprint into a first Bloom filter to perform existence judgment of pre-indexing; when the first Bloom filter judges that the content fingerprint may exist and local hot indexing fails to hit, inputting the content fingerprint into a second Bloom filter to perform secondary existence judgment of extended indexing; when the second Bloom filter judges that the content fingerprint may exist, performing hierarchical duplicate checking by using the full indexing of an S3 storage bucket and the third Bloom filter file; when it is confirmed that the data block to be backed up is a new data block, inserting the content fingerprint into the first Bloom filter and the second Bloom filter, and synchronizing the updated second Bloom filter to the S3 storage bucket. The application can realize hierarchical retrieval of large-scale block fingerprint indexing, thereby reducing the indexing access overhead in the incremental backup process.

Description

Technical Field

[0001] This application relates to the field of data backup management technology, and more specifically, to a block-level data backup management method and system for S3 object storage. Background Technology

[0002] Data backup management technology is a crucial technology for ensuring data security, business continuity, and disaster recovery capabilities. It is mainly used for backup, deduplication, indexing management, and recovery scheduling of files, databases, object data, and block-level data in business systems. S3 is a data storage service model based on an object storage interface. It typically manages massive amounts of unstructured data using buckets and object keys, and features convenient capacity expansion, unified access interfaces, and suitability for long-term archiving. With the widespread application of cloud storage architecture, backup management for S3 object storage can provide fundamental support for backup and recovery in multi-client, large-scale data scenarios through object-oriented storage, version management, and incremental backup mechanisms.

[0003] In existing technologies, block-level data backup typically requires splitting the data to be backed up and using block fingerprints to determine whether a data block already exists in the backup repository. However, as backup versions accumulate, the number of backup clients increases, and the size of the source data expands, the block fingerprint index will continuously grow. If the incremental backup process performs a search within the entire historical index each time, it will lead to increased index loading time, more remote read requests, and a decrease in local cache hit rate. If the index is split into a large number of small index objects, a single backup task will need to search across multiple index objects, increasing the number of object accesses and index location overhead. In addition, the block fingerprint index usually associates object keys, object offsets, and backup version references. The expansion of the index size will further affect backup version commits, historical version cleanup, and recovery integrity verification. Therefore, how to achieve hierarchical retrieval of large-scale block fingerprint indexes to reduce the index access overhead during incremental backup has become a challenge for the industry. Summary of the Invention

[0004] This application provides a block-level data backup management method and system for S3 object storage, which can realize hierarchical retrieval of large-scale block fingerprint indexes, thereby reducing the index access overhead during incremental backup.

[0005] In a first aspect, this application provides a block-level data backup management method for S3 object storage, the block-level data backup management method comprising the following steps: Build a first-level Bloom filter and a second-level Bloom filter locally, and store the third-level Bloom filter file in an S3 bucket. At the same time, calculate the content fingerprint of the data block to be backed up. The content fingerprint is input into the first-level Bloom filter to determine the existence of the pre-index; When the first-level Bloom filter determines that the content fingerprint may exist and the query of the local hot index fails, the content fingerprint is input into the second-level Bloom filter for a second existence determination of the extended index. When the secondary Bloom filter determines that it may exist, a hierarchical deduplication lookup is performed using the full index of the S3 storage bucket and the tertiary Bloom filter file. Based on the lookup results, it is confirmed whether the data block to be backed up is a new data block. When the data block to be backed up is confirmed to be a new data block, the content fingerprint is inserted into the first-level Bloom filter and the second-level Bloom filter, and the updated second-level Bloom filter is synchronized to the S3 storage bucket.

[0006] In this embodiment, a first-level Bloom filter and a second-level Bloom filter are built locally, and a third-level Bloom filter file is stored in the S3 storage bucket. The calculation of the content fingerprint of the current data block to be backed up specifically includes: Allocate a first storage area and a second storage area in local memory for the first-level Bloom filter and the second-level Bloom filter, respectively; All bit slots in the first and second storage areas are set to logic zero to form the first sparse bitmap of the first-level Bloom filter and the second sparse bitmap of the second-level Bloom filter. The first sparse bitmap is configured to use a first hash map set for fingerprint mapping, and the second sparse bitmap is configured to use a second hash map set for fingerprint mapping. Send a directory probe request to the S3 storage bucket to query whether a three-level Bloom filter file exists in the S3 storage bucket; If it exists, the third-level Bloom filter file is pulled from the S3 storage bucket and cached in the local temporary storage area. If it does not exist, an initialized third-level Bloom filter file is created in the S3 storage bucket, and the metadata information of the third-level Bloom filter file is written back to the object metadata tag of the S3 storage bucket. The current data stream to be backed up is dynamically sliced ​​and divided into blocks, and each data block obtained after the division is used as the current data block to be backed up. It also performs a collision-resistant hash operation on the current data block to be backed up, generating the content fingerprint of the current data block to be backed up.

[0007] In this embodiment, the existence determination of inputting the content fingerprint into the first-level Bloom filter for pre-indexing specifically includes: The content fingerprint is input as a query key into the first-level Bloom filter, and the content fingerprint is hashed through the first hash mapping set to obtain multiple slot indices of the content fingerprint in the first sparse bitmap. Determine the front index factor of the content fingerprint in the first-level Bloom filter based on all slot indexes; The false positive confidence of the first-level Bloom filter in its current state is determined based on the current bitmap capacity and the number of inserted fingerprints. The content fingerprint is pre-verified using the pre-index factor and the false judgment confidence to obtain the pre-verification pass mark of the content fingerprint. If the pre-verification pass mark is not passed, it is determined that the content fingerprint does not exist in the first-level Bloom filter, the current data block to be backed up is confirmed as a new data block, and a jump instruction to insert the content fingerprint into the first-level Bloom filter and the second-level Bloom filter is triggered. If the pre-verification pass mark is passed, it is determined that the content fingerprint may exist in the first-level Bloom filter, and the local hot index is accessed using the content fingerprint as the query keyword.

[0008] In this embodiment, determining the front index factor of the content fingerprint in the first-level Bloom filter based on all slot indices specifically includes: The multiple slot indices are sorted according to the order of hash mapping to obtain the slot index sequence of the content fingerprint in the first sparse bitmap. According to the slot index sequence, the corresponding slots in the first sparse bitmap are accessed sequentially, the binary bit values ​​of each slot are read, and the bit values ​​are combined into the bit value state vector of the content fingerprint in the first sparse bitmap according to the reading order. Based on the continuous distribution interval of logical truth values ​​in the bit-value state vector, the distribution span feature of logical truth values ​​in the bit-value state vector is extracted, and the content fingerprint is generated as a pre-index factor in the first-level Bloom filter based on the distribution span feature.

[0009] In this embodiment, the pre-verification of the content fingerprint using the pre-index factor and the false positive confidence level to obtain the pre-verification pass mark of the content fingerprint specifically includes: The content fingerprint is subjected to a first-level check code extraction based on the pre-index factor to obtain the pre-check code of the content fingerprint; The confidence-gated transformation of the pre-check code is performed using the false positive confidence level to obtain the pre-confidence gate code; The verification flag corresponding to the content fingerprint is written to the pre-verification gate code to obtain the pre-verification pass mark of the content fingerprint.

[0010] In this embodiment, when the first-level Bloom filter determines that the content fingerprint may exist and the query on the local hot index fails, the process of inputting the content fingerprint into the second-level Bloom filter for a secondary existence determination of the expanded index specifically includes: When the first-level Bloom filter determines that the content fingerprint may exist and the query of the local hot index fails, the content fingerprint is input into the second-level Bloom filter and mapped through the second hash mapping set to obtain multiple slot indices of the content fingerprint in the second sparse bitmap. Construct the hit decision tree of the content fingerprint in the second-level Bloom filter based on all slot indexes; The content fingerprint is extended and verified using the hit decision tree to obtain the extended retrieval trigger value of the content fingerprint; When the extended retrieval trigger value is lower than a preset threshold, the current data block to be backed up is confirmed as a new data block, and a jump instruction to insert the content fingerprint into the first-level Bloom filter and the second-level Bloom filter is triggered. When the extended retrieval trigger value is not lower than a preset threshold, it is determined that the content fingerprint may exist in the secondary Bloom filter.

[0011] In this embodiment, the extended index verification of the content fingerprint through the hit decision tree to obtain the extended retrieval trigger value of the content fingerprint specifically includes: Based on the multiple slot indices of the content fingerprint in the second sparse bitmap, the tree nodes in the hit decision tree are matched with index landing points to obtain the extended index landing point set corresponding to the content fingerprint; Extract the binary bit values ​​of the corresponding leaf nodes along each decision path of the hit decision tree, and perform logical AND reduction on the bit values ​​of the leaf nodes on each decision path to obtain the path pass flag of each decision path. Based on the extended index landing set and all paths, the extended retrieval trigger value of the content fingerprint in the secondary Bloom filter is determined by flags.

[0012] In this embodiment, when the secondary Bloom filter determines that a new data block may exist, a hierarchical deduplication lookup is performed using the full index of the S3 storage bucket and the tertiary Bloom filter file. Confirming whether the data block to be backed up is a new data block based on the lookup results specifically includes: When the secondary Bloom filter is determined to be possible, a remote index verification is performed on the tertiary Bloom filter file of the S3 storage bucket to obtain the remote index verification status. When the remote index verification status is valid, the content fingerprint is mapped and calculated to obtain multiple remote slot indexes; The remote deduplication gate value of the content fingerprint is generated based on all remote slot indexes, and the remote deduplication gate value is tokenized to obtain the lookup access token. When the remote index verification status is invalid, skip the slot mapping process of the three-level Bloom filter file and configure the lookup access token corresponding to the content fingerprint as an admission token. When the access token is a disabled token, the current data block to be backed up is confirmed to be a new data block, and the current data block to be backed up is uploaded to the allocated storage path of the S3 storage bucket; When the lookup access token is an admission token, the full index of the S3 bucket is accessed using the content fingerprint as the query key value, and the content fingerprint is used for exact matching lookup to obtain exact matching lookup results. When the search result is not found, the current data block to be backed up is confirmed as a new data block, and the current data block to be backed up is uploaded to the allocated storage path of the S3 storage bucket; When the search result is a match, it is confirmed that the data block to be backed up is an existing data block.

[0013] In this embodiment, when the data block to be backed up is confirmed to be a new data block, inserting the content fingerprint into the first-level Bloom filter and the second-level Bloom filter, and synchronizing the updated second-level Bloom filter to the S3 storage bucket specifically includes: When it is confirmed that the current data block to be backed up is a new data block, the content fingerprint is mapped according to the first hash mapping set to obtain multiple first slot indices, and the bit values ​​corresponding to the multiple first slot indices in the first sparse bitmap are set to logical true. The content fingerprint is mapped according to the second hash mapping set to obtain multiple second slot indices, and the bit values ​​corresponding to the multiple second slot indices in the second sparse bitmap are set to logical true. A snapshot of the updated second sparse bitmap is extracted and version information is attached to generate a synchronization file. The synchronization file is then uploaded to the preset storage directory of the S3 storage bucket, overwriting the existing old version file.

[0014] Secondly, this application provides a block-level data backup management system for S3 object storage, used to execute a block-level data backup management method for S3 object storage, the block-level data backup management system comprising: The hierarchical index building module is used to build first-level and second-level Bloom filters locally, store third-level Bloom filter files in the S3 bucket, and calculate the content fingerprint of the data block to be backed up. The pre-index judgment module is used to input the content fingerprint into the first-level Bloom filter to judge the existence of the pre-index. The extended index determination module is used to input the content fingerprint into the second-level Bloom filter to perform a second existence determination of the extended index when the first-level Bloom filter determines that it may exist and the query of the local hot index is not hit. The remote hierarchical deduplication module is used to perform hierarchical deduplication lookup by using the full index of the S3 storage bucket and the tertiary Bloom filter file when the secondary Bloom filter determines that the data block to be backed up is a new data block. The index synchronization update module is used to insert the content fingerprint into the first-level Bloom filter and the second-level Bloom filter when it is confirmed that the data block to be backed up is a new data block, and to synchronize the updated second-level Bloom filter to the S3 storage bucket.

[0015] The technical solutions provided by the embodiments disclosed in this application have the following beneficial effects: A first-level Bloom filter and a second-level Bloom filter are built locally, and a third-level Bloom filter file is stored in the S3 bucket. Simultaneously, the content fingerprint of the data block to be backed up is calculated. The content fingerprint is input into the first-level Bloom filter for a pre-index existence check. If the first-level Bloom filter determines that the data block may exist but the query of the local hot index fails, the content fingerprint is input into the second-level Bloom filter for a second existence check of the extended index. If the second-level Bloom filter determines that the data block may exist, a hierarchical deduplication lookup is performed using the full index of the S3 bucket and the third-level Bloom filter file. Based on the lookup result, it is confirmed whether the data block to be backed up is a new data block. When the data block to be backed up is confirmed to be a new data block, the content fingerprint is inserted into the first-level Bloom filter and the second-level Bloom filter, and the updated second-level Bloom filter is synchronized to the S3 bucket.

[0016] Therefore, in this application, when the data block to be backed up is confirmed to be a new data block, the content fingerprint can be inserted into the first-level Bloom filter and the second-level Bloom filter, and the updated second-level Bloom filter can be synchronized to the S3 storage bucket. Firstly, by constructing the first-level and second-level Bloom filters locally and storing the third-level Bloom filter file in the S3 storage bucket, while simultaneously calculating the content fingerprint of the current data block to be backed up, the block fingerprint query process can be decomposed into a progressive retrieval chain of local fast judgment, local extended judgment, and remote precise deduplication, so that the incremental backup task does not need to directly access the complete historical index for each data block. Secondly, by inputting the content fingerprint into the first-level Bloom filter for the existence judgment of the pre-index, data blocks that have obviously not appeared can be quickly excluded locally, reducing the number of fingerprints entering the subsequent retrieval stage, thereby reducing... Lighten the load on index loading and cache matching; furthermore, when the first-level Bloom filter determines that a duplicate data block may exist and the query on the local hot index fails, a second-level Bloom filter is used to perform a second existence check on the extended index. This can further filter suspected duplicate data blocks, reducing the probability of prematurely triggering remote queries due to Bloom filter misjudgments or local index misses, and reducing cross-object index access and remote read requests. Finally, only when the second-level Bloom filter still determines that a duplicate data block may exist is a hierarchical deduplication lookup performed using the full index of the S3 bucket and the third-level Bloom filter file. After confirming the new data block, the local Bloom filter and the remote filter status are updated. This ensures accurate block-level deduplication and allows the fingerprint of newly added content to enter the subsequent backup retrieval chain in a timely manner, thereby reducing the index access burden during backup version submission, historical version cleanup, and restoration integrity verification.

[0017] In summary, the technical solution adopted in this application can realize hierarchical retrieval of large-scale block fingerprint indexes and reduce the index access overhead during incremental backup. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only for this embodiment of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is an exemplary flowchart of a block-level data backup management method for S3 object storage provided in this application; Figure 2 This is a time consumption relationship graph based on the index size provided in this application; Figure 3It is based on the bitmap occupancy rate access ratio diagram provided in this application; Figure 4 This is a module structure diagram of a block-level data backup and management system for S3 object storage provided in this application; Figure 5 This is an application scenario diagram of the block-level data backup management method for S3 object storage provided in this application. Detailed Implementation

[0020] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0021] This application provides a block-level data backup management method and system for S3 object storage. Its core is to construct a first-level Bloom filter and a second-level Bloom filter locally, and store a third-level Bloom filter file in the S3 storage bucket. Simultaneously, it calculates the content fingerprint of the data block to be backed up. The content fingerprint is input into the first-level Bloom filter for a pre-index existence check. When the first-level Bloom filter determines that the data block may exist but the query of the local hot index fails, the content fingerprint is input into the second-level Bloom filter for a second existence check of the extended index. When the second-level Bloom filter determines that the data block may exist, a hierarchical deduplication lookup is performed using the full index of the S3 storage bucket and the third-level Bloom filter file. Based on the lookup result, it is confirmed whether the data block to be backed up is a new data block. When the data block to be backed up is confirmed to be a new data block, the content fingerprint is inserted into the first-level Bloom filter and the second-level Bloom filter, and the updated second-level Bloom filter is synchronized to the S3 storage bucket.

[0022] Example 1 To better understand the above technical solutions, a detailed explanation will be provided below with reference to the accompanying drawings and specific implementation methods. Figure 1 As shown in the figure, this is an exemplary flowchart of a block-level data backup management method for S3 object storage according to this embodiment of the present application, including the following steps: In step S1, a first-level Bloom filter and a second-level Bloom filter are built locally, and a third-level Bloom filter file is stored in bucket S3. At the same time, the content fingerprint of the data block to be backed up is calculated.

[0023] In this embodiment, a first-level Bloom filter and a second-level Bloom filter are built locally, and a third-level Bloom filter file is stored in the S3 storage bucket. The calculation of the content fingerprint of the current data block to be backed up can be achieved using the following steps: Allocate a first storage area and a second storage area in local memory for the first-level Bloom filter and the second-level Bloom filter, respectively; All bit slots in the first and second storage areas are set to logic zero to form the first sparse bitmap of the first-level Bloom filter and the second sparse bitmap of the second-level Bloom filter. The first sparse bitmap is configured to use a first hash map set for fingerprint mapping, and the second sparse bitmap is configured to use a second hash map set for fingerprint mapping. Send a directory probe request to the S3 storage bucket to query whether a three-level Bloom filter file exists in the S3 storage bucket; If it exists, the third-level Bloom filter file is pulled from the S3 storage bucket and cached in the local temporary storage area. If it does not exist, an initialized third-level Bloom filter file is created in the S3 storage bucket, and the metadata information of the third-level Bloom filter file is written back to the object metadata tag of the S3 storage bucket. The current data stream to be backed up is dynamically sliced ​​and divided into blocks, and each data block obtained after the division is used as the current data block to be backed up. It also performs a collision-resistant hash operation on the current data block to be backed up, generating the content fingerprint of the current data block to be backed up.

[0024] It should be noted that, in this application, the first storage area represents a contiguous memory space in local memory specifically used to carry the bit data of the first-level Bloom filter; the second storage area represents a contiguous memory space in local memory specifically used to carry the bit data of the second-level Bloom filter; the first sparse bitmap represents a bit array structure in the first-level Bloom filter consisting of all bit slots, with all bits initially set to logical zeros; the second sparse bitmap represents a bit array structure in the second-level Bloom filter consisting of all bit slots, with all bits initially set to logical zeros; the first hash map set represents a set of independent hash functions applied to the first sparse bitmap to map the content fingerprint to the corresponding slot index; the second hash map set represents a set of independent hash functions applied to the second sparse bitmap to map the content fingerprint to the corresponding slot index; and the directory probe request represents a request sent to the S3 bucket for verification. The query request specifies whether an object exists; the three-level Bloom filter file represents a persistent Bloom filter file stored in an S3 bucket for full fingerprint existence pre-verification; the local temporary storage area represents temporary storage space on the local disk for temporarily caching files pulled from remote sources; the metadata information represents attribute information recording the version, capacity, and number of hash functions of the three-level Bloom filter file; the object metadata tag represents a key-value pair field built into the S3 object for storing custom attribute information; the dynamic slicing and block processing represents the process of performing fixed-length slicing of a continuous data stream according to a preset granularity; the data stream to be backed up represents the original continuous byte stream to be backed up at the block level; the collision-resistant hash operation represents a hash operation used to map data blocks of arbitrary length to fixed-length unique identifiers; the content fingerprint represents a fixed-length hash value that can uniquely identify the content of a data block.

[0025] In practice, firstly, standard library memory allocation functions are used to calculate the required number of contiguous bytes based on the preset total capacity of the Bloom filter. Two independent contiguous memory spaces are allocated in the local process heap memory. The smaller space corresponds to the capacity requirements of the first-level Bloom filter and is designated as the first storage area. The larger space corresponds to the capacity requirements of the second-level Bloom filter and is designated as the second storage area. The capacity ratio of the two storage areas is set to 1:4, based on the expected capacity of the hot fingerprint and extended fingerprint, ensuring the memory access efficiency of the hot query path. Secondly, a memory batch zeroing instruction is used to perform a bit-by-bit zeroing operation on all bytes in both storage areas, ensuring that all slots in both storage areas are in a logical zero state. The continuous bit array formed after clearing the first storage area is used as the first sparse bitmap, and the continuous bit array formed after clearing the second storage area is used as the second sparse bitmap. Next, two sets of independent unencrypted hash functions are selected: the first set contains 3 hash functions, and the second set contains 5 hash functions. The number of functions is set according to the classic false positive rate model of the Bloom filter, aiming to control the false positive rate to within one-thousandth under the expected insertion volume. The first set of hash functions is configured as the fingerprint mapping function set of the first sparse bitmap, serving as the first hash mapping set, and the second set of hash functions is configured as the fingerprint mapping function set of the second sparse bitmap, serving as the second hash mapping set. Subsequently, the HEAD request method of the S3 protocol is used to concatenate the preset objects of the three-level Bloom filter file. The system sends a query request to the S3 bucket using the key and the bucket server endpoint address. This request is treated as a directory probe request, and the existence of the corresponding file is determined based on the HTTP status code returned by the request. Next, when the status code is 200, a GET request using the S3 protocol is used to retrieve the complete three-level Bloom filter file and write it to the preset temporary storage directory on the local disk. The storage space corresponding to this temporary directory is used as the local temporary storage area. When the status code is 404, an array file with all zero bits matching the preset size is constructed by filling each byte with zero values ​​as the initialized three-level Bloom filter file. A PUT request using the S3 protocol is used to upload it to the preset storage path of the bucket, simultaneously recording the file size, number of hash functions, and initial version number. The attributes are written to the custom metadata fields of the S3 object, and these attribute information is used as metadata information. The custom fields carrying these attributes are used as object metadata tags. Then, a fixed-length block algorithm is used to sequentially divide the data stream to be backed up. The block size is set to 4MB. This value is based on the optimal granularity setting for block-level backup in object storage scenarios, which can balance data deduplication rate and metadata management overhead. The continuous data stream is divided sequentially according to byte offset, and each fixed-length data unit obtained after the division is used as the current data block to be backed up. Finally, the SHA-256 hash algorithm is used to hash all bytes of the current data block to be backed up, and a fixed-length hash value is output. The calculated hash value is used as the content fingerprint of the current data block to be backed up.

[0026] In step S2, the content fingerprint is input into the first-level Bloom filter to determine the existence of the pre-index.

[0027] In this embodiment, the existence determination of inputting the content fingerprint into the first-level Bloom filter for pre-indexing can be achieved through the following steps: The content fingerprint is input as a query key into the first-level Bloom filter, and the content fingerprint is hashed through the first hash mapping set to obtain multiple slot indices of the content fingerprint in the first sparse bitmap. Determine the front index factor of the content fingerprint in the first-level Bloom filter based on all slot indexes; The false positive confidence of the first-level Bloom filter in its current state is determined based on the current bitmap capacity and the number of inserted fingerprints. The content fingerprint is pre-verified using the pre-index factor and the false judgment confidence to obtain the pre-verification pass mark of the content fingerprint. If the pre-verification pass mark is not passed, it is determined that the content fingerprint does not exist in the first-level Bloom filter, the current data block to be backed up is confirmed as a new data block, and a jump instruction to insert the content fingerprint into the first-level Bloom filter and the second-level Bloom filter is triggered. If the pre-verification pass mark is passed, it is determined that the content fingerprint may exist in the first-level Bloom filter, and the local hot index is accessed using the content fingerprint as the query keyword.

[0028] It should be noted that, in this application, the query key value represents the input identifier value used to trigger hash mapping and slot location when inputting to the first-level Bloom filter; the slot index represents the slot position number corresponding to the content fingerprint in the sparse bitmap after hash mapping; the bitmap capacity represents the total number of slots contained in the sparse bitmap of the Bloom filter; the number of inserted fingerprints represents the total number of content fingerprints currently written to the Bloom filter; the false positive confidence represents a quantitative value reflecting the probability of false positives in the current state of the Bloom filter; the jump instruction represents the control instruction that triggers the fingerprint insertion operation; the query keyword represents the identifier value used for matching and retrieval when accessing the local hot index; and the local hot index represents a fast retrieval index stored locally that records the data block information corresponding to frequently accessed fingerprints.

[0029] In specific implementation, firstly, each hash function in the first hash map set is called sequentially to perform hash calculation on the complete byte sequence of the content fingerprint. The integer value output by each hash function is moduloed by the total number of slots in the first sparse bitmap to obtain the slot position number corresponding to each hash function. All position numbers are used as slot indices. Based on all slot indices, the pre-index factor of the content fingerprint in the first-level Bloom filter is determined. Then, the false positive confidence of the first-level Bloom filter in the current state is calculated using the following formula:

[0030] in, This indicates the false positive confidence level of the first-stage Bloom filter in the current state. This indicates the number of hash functions in the first hash map set. This indicates the number of inserted content fingerprints in the first-level Bloom filter. This represents the bitmap capacity of the first-stage Bloom filter. The value represents a natural constant, and the subscript 1 indicates that the corresponding parameter applies to the first-level Bloom filter. Then, the content fingerprint is pre-verified using the pre-index factor and the false positive confidence level to obtain a pre-verification pass flag for the content fingerprint. Next, if the pre-verification pass flag is in a failed state, it is directly determined that the corresponding content fingerprint is not recorded in the first-level Bloom filter, confirming that the current data block to be backed up is a new data block, and outputting a control signal to trigger the fingerprint insertion process, which is used as a jump instruction. Finally, if the pre-verification pass flag is in a passed state, it is determined that the corresponding content fingerprint may exist in the first-level Bloom filter, and the content fingerprint is used as the input identifier for retrieval matching, i.e., the query keyword, to initiate a retrieval access to the locally stored hot index table.

[0031] In this embodiment, determining the front index factor of the content fingerprint in the first-level Bloom filter based on all slot indices can be achieved using the following steps: The multiple slot indices are sorted according to the order of hash mapping to obtain the slot index sequence of the content fingerprint in the first sparse bitmap. According to the slot index sequence, the corresponding slots in the first sparse bitmap are accessed sequentially, the binary bit values ​​of each slot are read, and the bit values ​​are combined into the bit value state vector of the content fingerprint in the first sparse bitmap according to the reading order. Based on the continuous distribution interval of logical truth values ​​in the bit-value state vector, the distribution span feature of logical truth values ​​in the bit-value state vector is extracted, and the content fingerprint is generated as a pre-index factor in the first-level Bloom filter based on the distribution span feature.

[0032] It should be noted that, in this application, the slot index sequence refers to an ordered numbering sequence formed by arranging multiple slot indices in the order of hash mapping; the slot refers to the smallest storage unit in a sparse bitmap used to store a single binary bit value; the binary bit value refers to the binary state value of logical zero or logical true stored in the slot; the bit value state vector refers to a binary sequence vector formed by sequentially combining the binary bit values ​​of each slot; the logical true value refers to the binary state of the corresponding slot that has been marked by fingerprint mapping; the continuous distribution interval refers to the range of positions in the bit value state vector where logical true values ​​appear consecutively; the distribution span feature refers to a quantitative feature reflecting the size of the continuous distribution range of logical true values; and the pre-index factor refers to a feature value reflecting the continuous distribution state of the bit values ​​of the mapped slots in the first-level Bloom filter.

[0033] In specific implementation, firstly, according to the order of hash function calls in the hash mapping set, multiple slot indices are sequentially arranged to obtain a number sequence arranged according to the mapping order, and this ordered number sequence is used as the slot index sequence; secondly, according to the number order in the slot index sequence, the corresponding storage unit in the first sparse bitmap is accessed bit by bit, and the binary state stored in each unit is read. All the read binary states are concatenated in the access order to form an ordered binary sequence, and this binary sequence is used as the bit value state vector; next, all bits of the bit value state vector are traversed, and the start and end positions of the logical truth values ​​are marked bit by bit. The bit length of each continuous logical truth value segment is counted, and the maximum bit length value among all continuous segments is selected. This maximum bit length value is used as the distribution span feature. The value of the distribution span feature is divided by the total number of bits contained in the bit value state vector to obtain the length ratio of continuous logical truth values, and the value of this ratio is used as the pre-index factor.

[0034] In this embodiment, the pre-verification of the content fingerprint using the pre-index factor and the false positive confidence level to obtain the pre-verification pass mark of the content fingerprint can be achieved through the following steps: The content fingerprint is subjected to a first-level check code extraction based on the pre-index factor to obtain the pre-check code of the content fingerprint; The confidence-gated transformation of the pre-check code is performed using the false positive confidence level to obtain the pre-confidence gate code; The verification flag corresponding to the content fingerprint is written to the pre-verification gate code to obtain the pre-verification pass mark of the content fingerprint.

[0035] It should be noted that, in this application, the pre-verification code represents the feature encoding value used for pre-existence verification; the pre-confidence gate code represents the gate encoding value that controls the output of the verification result; the verification flag bit represents the binary storage bit that stores the status of the verification result of a single content fingerprint; and the pre-verification pass flag represents the status flag that indicates whether the pre-verification result of the content fingerprint has passed.

[0036] In specific implementation, firstly, a binary mask truncation method is used to map the value of the preceding index factor to a fixed-length binary mask. The mask length is set to 8 bits according to the verification accuracy requirements, which can balance the verification discrimination and computational overhead. A bitwise AND operation is performed between the mask and the binary value of the content fingerprint, and the result is used as the preceding verification code. Secondly, a threshold comparison gating method is used to compare the false positive confidence value with a preset confidence threshold. The confidence threshold is set to 0.001 based on the false positive tolerance of the preceding verification, ensuring that within a certain range... When the Bloom filter is under high load, the verification conditions are tightened. If the false positive confidence level is less than or equal to the confidence threshold, an 8-bit gating mask of all 1s is output. If it is greater than the confidence threshold, an 8-bit gating mask of all 0s is output. The gating mask is then bitwise ANDed with the pre-verification code, and the result is used as the pre-verification confidence gating code. Finally, the value of the pre-verification confidence gating code is checked for non-zero values. If it is non-zero, the verification flag is written to the logical true value. If it is zero, the verification flag is written to the logical zero value. The status value of the verification flag is used as the pre-verification pass mark.

[0037] In step S3, when the first-level Bloom filter determines that the content fingerprint may exist and the query of the local hot index fails, the content fingerprint is input into the second-level Bloom filter for a second existence determination of the extended index.

[0038] In this embodiment, when the first-level Bloom filter determines that the content fingerprint may exist and the query on the local hot index fails, the secondary existence determination of the expanded index by inputting the content fingerprint into the second-level Bloom filter can be achieved by the following steps: When the first-level Bloom filter determines that the content fingerprint may exist and the query of the local hot index fails, the content fingerprint is input into the second-level Bloom filter and mapped through the second hash mapping set to obtain multiple slot indices of the content fingerprint in the second sparse bitmap. Construct the hit decision tree of the content fingerprint in the second-level Bloom filter based on all slot indexes; The content fingerprint is extended and verified using the hit decision tree to obtain the extended retrieval trigger value of the content fingerprint; When the extended retrieval trigger value is lower than a preset threshold, the current data block to be backed up is confirmed as a new data block, and a jump instruction to insert the content fingerprint into the first-level Bloom filter and the second-level Bloom filter is triggered. When the extended retrieval trigger value is not lower than a preset threshold, it is determined that the content fingerprint may exist in the secondary Bloom filter.

[0039] It should be noted that, in this application, the slot index refers to the slot position number corresponding to the content fingerprint in the sparse bitmap after hash mapping; the preset threshold refers to a pre-set boundary value used to divide the secondary existence determination results; and the jump instruction refers to a control instruction that triggers the fingerprint insertion operation.

[0040] In specific implementation, firstly, each hash function in the second hash map set is called sequentially to perform hash calculation on the complete byte sequence of the content fingerprint. The integer value output by each hash function is modulo the total number of slots in the second sparse bitmap to obtain the slot position number corresponding to each hash function. All position numbers are used as slot indices. Secondly, a hierarchical binary tree construction method is adopted, with the total number of functions contained in the second hash map set as the total number of layers in the decision tree. Each layer corresponds to the slot index of a hash function. Each intermediate node stores the branch decision rules for the corresponding slot value, and the leaf nodes store the combined decision results of all slot values. The constructed hierarchical tree decision structure is then formed. As a hit decision tree, the content fingerprint is extended and indexed for verification to obtain an extended retrieval trigger value. Then, the extended retrieval trigger value is compared with a preset threshold. The preset threshold is set to 0.8 based on the false positive control target of the second-level Bloom filter, which can filter out most false positive scenarios while ensuring verification accuracy. If the extended retrieval trigger value is lower than the preset threshold, the current data block to be backed up is confirmed as a new data block, and a control signal is output to trigger the fingerprint insertion process, which is then used as a jump instruction. Finally, if the extended retrieval trigger value is not lower than the preset threshold, it is determined that the content fingerprint may exist in the second-level Bloom filter.

[0041] In this embodiment, the extended retrieval trigger value of the content fingerprint obtained by performing extended index verification on the content fingerprint through the hit decision tree can be achieved through the following steps: Based on the multiple slot indices of the content fingerprint in the second sparse bitmap, the tree nodes in the hit decision tree are matched with index landing points to obtain the extended index landing point set corresponding to the content fingerprint; Extract the binary bit values ​​of the corresponding leaf nodes along each decision path of the hit decision tree, and perform logical AND reduction on the bit values ​​of the leaf nodes on each decision path to obtain the path pass flag of each decision path. Based on the extended index landing set and all paths, the extended retrieval trigger value of the content fingerprint in the secondary Bloom filter is determined by flags.

[0042] It should be noted that, in this application, the tree-level nodes refer to decision nodes distributed at various levels in the decision tree that carry slot index matching rules; the index landing point matching refers to the process of locating and matching the slot index corresponding to the content fingerprint with the nodes at each level of the decision tree; the extended index landing point set refers to the set of nodes composed of all successfully matched decision tree nodes obtained after index landing point matching; the leaf node refers to the node that hits the last level of the decision tree and stores the final judgment result of the bit value; the path is represented by a flag indicating whether all bit values ​​on a single decision path are hit; and the extended retrieval trigger value is a quantitative value used to determine whether to start a remote retrieval.

[0043] In specific implementation, firstly, following the hierarchical arrangement of the hit decision tree from top to bottom, each slot index is sequentially matched and verified with all tree nodes of the corresponding level. Nodes that match are marked as valid landing nodes. All valid landing nodes at all levels are collected, and the set of nodes formed by these nodes is used as the extended index landing set. Secondly, all decision paths from the root node to each leaf node in the hit decision tree are traversed. The binary bit values ​​stored in the corresponding landing nodes are extracted sequentially along each decision path. A logical AND operation is performed on all bit values ​​within the same decision path. If the result is logically true, the corresponding path is marked as passed; otherwise, it is marked as failed. The status marking result of each decision path is used as the path passing flag for the corresponding decision path. Finally, the extended retrieval trigger value of the content fingerprint in the second-level Bloom filter is calculated using the following formula:

[0044] in, This indicates the extended retrieval trigger value of the content fingerprint in the second-level Bloom filter; the subscript 2 indicates that the extended retrieval trigger value applies to the second-level Bloom filter. This represents the total number of decision paths that were hit in the decision tree. This indicates the first [number] hit in the decision tree. Decision-making path; Indicates the first The number of valid landing points belonging to the extended index landing point set in each decision path; Indicates the first The path of each decision-making path is marked; Indicates the first The first decision path The binary bit value corresponding to each valid landing node; This represents a very small positive number used to avoid a denominator of zero.

[0045] In step S4, when the secondary Bloom filter determines that it may exist, a hierarchical deduplication lookup is performed using the full index of the S3 storage bucket and the tertiary Bloom filter file. Based on the lookup results, it is confirmed whether the data block to be backed up is a new data block.

[0046] In this embodiment, when the secondary Bloom filter determines that a new data block may exist, a hierarchical deduplication lookup is performed using the full index of the S3 storage bucket and the tertiary Bloom filter file. The following steps can be used to confirm whether the data block to be backed up is a new data block based on the lookup results: When the secondary Bloom filter is determined to be possible, a remote index verification is performed on the tertiary Bloom filter file of the S3 storage bucket to obtain the remote index verification status. When the remote index verification status is valid, the content fingerprint is mapped and calculated to obtain multiple remote slot indexes; The remote deduplication gate value of the content fingerprint is generated based on all remote slot indexes, and the remote deduplication gate value is tokenized to obtain the lookup access token. When the remote index verification status is invalid, skip the slot mapping process of the three-level Bloom filter file and configure the lookup access token corresponding to the content fingerprint as an admission token. When the access token is a disabled token, the current data block to be backed up is confirmed to be a new data block, and the current data block to be backed up is uploaded to the allocated storage path of the S3 storage bucket; When the lookup access token is an admission token, the full index of the S3 bucket is accessed using the content fingerprint as the query key value, and the content fingerprint is used for exact matching lookup to obtain exact matching lookup results. When the search result is not found, the current data block to be backed up is confirmed as a new data block, and the current data block to be backed up is uploaded to the allocated storage path of the S3 storage bucket; When the search result is a match, it is confirmed that the data block to be backed up is an existing data block.

[0047] It should be noted that, in this application, the remote index verification status refers to the status identifier obtained after validating the file in the S3 bucket using a three-level Bloom filter; the remote slot index refers to the remote slot position number obtained after hash mapping of the content fingerprint using a three-level Bloom filter; the remote deduplication gate value refers to the gate value used to control the deduplication process; the lookup access token refers to the permission token used to control whether to start a full index exact lookup; the disable token refers to the permission token indicating that a full index lookup is not required; the admission token refers to the permission token indicating that a full index lookup is allowed; the allocated storage path refers to the storage path pre-allocated in the S3 bucket for storing newly added data blocks; the full index refers to the complete index set stored in the S3 bucket that records the content fingerprints of all stored data blocks; and the lookup result refers to the hit or miss status result obtained after a full index exact match.

[0048] In practical implementation, firstly, the HEAD request method of the S3 protocol is used to send a metadata retrieval request for the three-level Bloom filter file to the S3 storage bucket. The file's version identifier and integrity check value are verified. If the verification passes, it is marked as valid; otherwise, it is marked as invalid. This status is used as the remote index verification status. Secondly, when the remote index verification status is valid, the set of hash functions corresponding to the three-level Bloom filter is called. The number of hash functions is set to seven based on the expected deduplication rate and false positive control requirements of the full data, which can control the false positive rate of remote pre-verification to within one ten-thousandth. The complete byte sequence of the content fingerprint is then processed sequentially. The system performs hash calculations, taking the modulo of the total number of slots in the three-level Bloom filter for each hash result to obtain the position number corresponding to each hash function. All position numbers are then used as remote slot indices. Next, during system initialization, a binary key-value mapping table is pre-built, clearly defining the one-to-one mapping relationship between logical true values ​​and access tokens, and logical false values ​​and disabling tokens. This mapping table is then permanently stored in the local configuration storage area, forming a preset token mapping rule. The system reads the binary bit values ​​corresponding to all remote slot indices from the three-level Bloom filter file, performs a logical AND operation on all bit values, and uses the resulting binary value as the remote deduplication gate value. The system then calls the locally stored... The storage's preset token mapping rules match and transform the remote deduplication gate value. If the gate value is logically true, an admission token is output; if the gate value is logically false, a disabling token is output. The matched token is used as the lookup access token. Subsequently, when the remote index verification status is invalid, the slot mapping and bit value reading process of the three-level Bloom filter file is skipped, and the lookup access token corresponding to the content fingerprint is directly configured as the admission token. Then, when the lookup access token is a disabling token, the current data block to be backed up is confirmed as a new data block. According to the S3 storage bucket's preset path allocation rules, the current data block to be backed up is uploaded to the corresponding storage location. The storage location is used as the allocated storage path. Then, when the access token matches the admission token, the content fingerprint is used as the query matching key to call the full index retrieval interface of the S3 bucket, performing a completely consistent matching retrieval operation to obtain a search hit or miss status result, which is then used as the search result. Next, if the search result is a miss, the current data block to be backed up is confirmed as a new data block, and according to the S3 bucket's preset path allocation rules, the current data block to be backed up is uploaded to the corresponding storage location, which is then used as the allocated storage path. Finally, if the search result is a hit, the current data block to be backed up is confirmed as an existing data block.

[0049] For example, to verify the effectiveness of the hierarchical deduplication mechanism in suppressing search time under large-scale block fingerprint indexing conditions, a control test can be set up under the same object storage access path, the same block granularity, and the same content fingerprint generation method. In the test, the data blocks to be backed up are divided into 4MB fixed-length blocks, and the content fingerprints are generated using SHA-256. Index environments with 100 million, 300 million, 600 million, and 1 billion existing fingerprints are constructed respectively, and the single-block search time of full exact match, remote Bloom preflight, and hierarchical deduplication mechanism are statistically analyzed. (Reference) Figure 2 As shown in the figure, this is a graph illustrating the relationship between index size and search time provided in this application. As the index size increases from 100 million to 1 billion records, the single-block search time for full exact matching increases from 1.86 ms / block to 15.28 ms / block, the single-block search time for remote Bloom preflight increases from 1.04 ms / block to 3.19 ms / block, while the single-block search time for the hierarchical deduplication mechanism only increases from 0.58 ms / block to 1.29 ms / block. This demonstrates that relying solely on full index exact matching leads to a continuous increase in remote index reading and matching overhead as the index size expands. While remote Bloom preflight can filter out some non-existent fingerprints, it still requires frequent remote bitmap readings. The hierarchical deduplication mechanism, through the progressive convergence of the front index, extended index, and remote deduplication gating, truncates a large number of non-existent content fingerprints before they enter the full index, thus significantly reducing the rate at which search time increases with index size.

[0050] For example, to further verify the effect of the three-level Bloom filter file and lookup access token on reducing the number of S3 full index accesses, the test sample can be fixed at 10,000 data blocks to be backed up, and the occupancy rate of the three-level Bloom bitmap can be set to 25%, 45%, 65%, and 85% respectively. The proportion of full index access triggered under different deduplication methods can be counted. (Reference) Figure 3 As shown in the figure, this is a graph showing the bitmap occupancy rate and access ratio provided in this application. When the occupancy rate of the three-level Bloom bitmap is 25%, the full index access ratio for full exact match remains at 100%, the full index access ratio for remote Bloom pre-check is 39%, and the full index access ratio for the hierarchical deduplication mechanism is 16%. When the occupancy rate of the three-level Bloom bitmap increases to 85%, the full index access ratio for remote Bloom pre-check rises to 79%, while the hierarchical deduplication mechanism remains at 47%. The above data shows that as the occupancy rate of the three-level Bloom bitmap increases, the chance of false positives from the Bloom filter increases accordingly, and the ability of single-level remote pre-check to reduce full index access gradually decreases. The hierarchical deduplication mechanism completes local pre-check and extended index verification before remote bitmap verification, and controls whether to allow full index exact match by looking up the access token during the remote deduplication stage. Therefore, it can continue to reduce unnecessary S3 read requests when the bitmap load increases, keeping the remote index access overhead during incremental backup at a low level.

[0051] In step S5, when it is confirmed that the data block to be backed up is a new data block, the content fingerprint is inserted into the first-level Bloom filter and the second-level Bloom filter, and the updated second-level Bloom filter is synchronized to the S3 storage bucket.

[0052] In this embodiment, when the data block to be backed up is confirmed to be a new data block, inserting the content fingerprint into the first-level Bloom filter and the second-level Bloom filter, and synchronizing the updated second-level Bloom filter to the S3 storage bucket can be achieved through the following steps: When it is confirmed that the current data block to be backed up is a new data block, the content fingerprint is mapped according to the first hash mapping set to obtain multiple first slot indices, and the bit values ​​corresponding to the multiple first slot indices in the first sparse bitmap are set to logical true. The content fingerprint is mapped according to the second hash mapping set to obtain multiple second slot indices, and the bit values ​​corresponding to the multiple second slot indices in the second sparse bitmap are set to logical true. A snapshot of the updated second sparse bitmap is extracted and version information is attached to generate a synchronization file. The synchronization file is then uploaded to the preset storage directory of the S3 storage bucket, overwriting the existing old version file.

[0053] It should be noted that, in this application, the first slot index represents the slot position number corresponding to the content fingerprint in the first sparse bitmap after being mapped by the first hash mapping set; the second slot index represents the slot position number corresponding to the content fingerprint in the second sparse bitmap after being mapped by the second hash mapping set; the version information represents attribute information used to identify the iteration order and generation time of the synchronization file; the synchronization file represents the file to be uploaded that encapsulates the full data of the second sparse bitmap and the corresponding version information; and the preset storage directory represents the storage path pre-defined in the S3 storage bucket for storing the synchronization files of the second-level Bloom filter.

[0054] In practice, firstly, when it is confirmed that the data block to be backed up is a new data block, all hash functions in the first hash map set are called sequentially to perform hash calculations on the complete byte sequence of the content fingerprint. The integer value output by each hash function is modulo the total number of slots in the first sparse bitmap to obtain the corresponding position number. All position numbers are used as the first slot index, and write logical operations are performed on the storage units corresponding to the first slot index in the first sparse bitmap one by one to complete the fingerprint insertion of the first-level Bloom filter. Secondly, all hash functions in the second hash map set are called sequentially to perform hash calculations on the complete byte sequence of the content fingerprint. The integer value output by each hash function is modulo the total number of slots in the second sparse bitmap to obtain the corresponding position number. All position numbers are used as the second slot index, and write logical operations are performed on the storage units corresponding to the second slot index in the second sparse bitmap one by one to complete the fingerprint insertion. The storage unit performs a write logic operation to complete the fingerprint insertion of the second-level Bloom filter. Finally, a full memory copy method is used to perform snapshot extraction on the updated second sparse bitmap, copying the complete bit array data. The full copy method is set according to the design feature of the fixed capacity of the second sparse bitmap, which is simple to implement and can ensure that the remote and local data are completely consistent. At the same time, an identifier information combining an incrementing version number and the current timestamp is generated. This identifier information is used as version information. The snapshot data of the second sparse bitmap and the version information are encapsulated into a single binary file, which is used as a synchronization file. The PUT upload interface of the S3 protocol is called to upload the synchronization file to the fixed storage path of the S3 storage bucket. This fixed storage path is used as the preset storage directory. After the upload is completed, the old version file in the same path is overwritten, completing the remote synchronization of the second-level Bloom filter.

[0055] Therefore, in this application, when the data block to be backed up is confirmed to be a new data block, the content fingerprint can be inserted into the first-level Bloom filter and the second-level Bloom filter, and the updated second-level Bloom filter can be synchronized to the S3 storage bucket. Firstly, by constructing the first-level and second-level Bloom filters locally and storing the third-level Bloom filter file in the S3 storage bucket, while simultaneously calculating the content fingerprint of the current data block to be backed up, the block fingerprint query process can be decomposed into a progressive retrieval chain of local fast judgment, local extended judgment, and remote precise deduplication, so that the incremental backup task does not need to directly access the complete historical index for each data block. Secondly, by inputting the content fingerprint into the first-level Bloom filter for the existence judgment of the pre-index, data blocks that have obviously not appeared can be quickly excluded locally, reducing the number of fingerprints entering the subsequent retrieval stage, thereby reducing... Lighten the load on index loading and cache matching; furthermore, when the first-level Bloom filter determines that a duplicate data block may exist and the query on the local hot index fails, a second-level Bloom filter is used to perform a second existence check on the extended index. This can further filter suspected duplicate data blocks, reducing the probability of prematurely triggering remote queries due to Bloom filter misjudgments or local index misses, and reducing cross-object index access and remote read requests. Finally, only when the second-level Bloom filter still determines that a duplicate data block may exist is a hierarchical deduplication lookup performed using the full index of the S3 bucket and the third-level Bloom filter file. After confirming the new data block, the local Bloom filter and the remote filter status are updated. This ensures accurate block-level deduplication and allows the fingerprint of newly added content to enter the subsequent backup retrieval chain in a timely manner, thereby reducing the index access burden during backup version submission, historical version cleanup, and restoration integrity verification.

[0056] In summary, the technical solution adopted in this application can realize hierarchical retrieval of large-scale block fingerprint indexes and reduce the index access overhead during incremental backup.

[0057] Example 2 This application provides a block-level data backup and management system for S3 object storage, referencing... Figure 4 As shown in the figure, this is a module structure diagram of a block-level data backup management system for S3 object storage according to this embodiment of the present application. The block-level data backup management system includes: The hierarchical index building module 100 is used to build the first-level Bloom filter and the second-level Bloom filter locally, and store the third-level Bloom filter file in the S3 storage bucket, while calculating the content fingerprint of the data block to be backed up. The pre-index judgment module 200 is used to input the content fingerprint into the first-level Bloom filter to judge the existence of the pre-index. The extended index determination module 300 is used to input the content fingerprint into the second-level Bloom filter to perform a second existence determination of the extended index when the first-level Bloom filter determines that it may exist and the query of the local hot index fails. The remote hierarchical deduplication module 400 is used to perform hierarchical deduplication lookup by using the full index of the S3 storage bucket and the tertiary Bloom filter file when the secondary Bloom filter determines that it may exist, and to confirm whether the data block to be backed up is a new data block based on the lookup result. The index synchronization update module 500 is used to insert the content fingerprint into the first-level Bloom filter and the second-level Bloom filter when it is confirmed that the data block to be backed up is a new data block, and to synchronize the updated second-level Bloom filter to the S3 storage bucket.

[0058] It should also be noted that the reference Figure 5 As shown in the figure, this is an application scenario diagram of the block-level data backup management method for S3 object storage provided in this application. In this figure, the data source side includes a business server, a database, and a file system, which are used to provide the raw data to be backed up at the block level. The backup management node, as the local processing entity, is used to receive the data stream to be backed up and sequentially perform block processing and content fingerprint calculation to obtain the content fingerprint of the current data block to be backed up. Subsequently, the content fingerprint is first input into a first-level Bloom filter to determine the existence of the front index. If it is determined that it may exist, the local hot index is accessed. If the local hot index is not found, it is then input into a second-level Bloom filter to determine the existence of the extended index. For content fingerprints that may still exist, the backup management node accesses the full index and the third-level Bloom filter file in the S3 object storage through remote deduplication control to complete the hierarchical deduplication search, and confirms whether the current data block to be backed up is a new data block or an existing data block based on the search results. If it is confirmed to be a new data block, the data block is written to the data block storage area, and the relevant content fingerprint is written to the corresponding Bloom filter to realize the collaborative update of the local index and the remote storage.

[0059] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0060] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, including read-only memory (ROM), random access memory (RAM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), one-time programmable read-only memory (OTPROM), electrically-Erasable Programmable Read-Only Memory (EEPROM), compactdisc read-only memory (CD-ROM) or other optical disc storage, disk storage, magnetic tape storage, or any other computer-readable medium capable of carrying or storing data.

[0061] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

Claims

1. A block-level data backup management method for S3 object storage, characterized in that, Includes the following steps: Build a first-level Bloom filter and a second-level Bloom filter locally, and store the third-level Bloom filter file in an S3 bucket. At the same time, calculate the content fingerprint of the data block to be backed up. The content fingerprint is input into the first-level Bloom filter to determine the existence of the pre-index; When the first-level Bloom filter determines that the content fingerprint may exist and the query of the local hot index fails, the content fingerprint is input into the second-level Bloom filter for a second existence determination of the extended index. When the secondary Bloom filter determines that it may exist, a hierarchical deduplication lookup is performed using the full index of the S3 storage bucket and the tertiary Bloom filter file. Based on the lookup results, it is confirmed whether the data block to be backed up is a new data block. When the data block to be backed up is confirmed to be a new data block, the content fingerprint is inserted into the first-level Bloom filter and the second-level Bloom filter, and the updated second-level Bloom filter is synchronized to the S3 storage bucket.

2. The block-level data backup management method for S3 object storage as described in claim 1, characterized in that, A first-level and second-level Bloom filter is built locally, and a third-level Bloom filter file is stored in an S3 bucket. Simultaneously, the content fingerprint of the data block to be backed up is calculated, specifically including: Allocate a first storage area and a second storage area in local memory for the first-level Bloom filter and the second-level Bloom filter, respectively; All bit slots in the first and second storage areas are set to logic zero to form the first sparse bitmap of the first-level Bloom filter and the second sparse bitmap of the second-level Bloom filter. The first sparse bitmap is configured to use a first hash map set for fingerprint mapping, and the second sparse bitmap is configured to use a second hash map set for fingerprint mapping. Send a directory probe request to the S3 storage bucket to query whether a three-level Bloom filter file exists in the S3 storage bucket; If it exists, the third-level Bloom filter file is pulled from the S3 storage bucket and cached in the local temporary storage area. If it does not exist, an initialized third-level Bloom filter file is created in the S3 storage bucket, and the metadata information of the third-level Bloom filter file is written back to the object metadata tag of the S3 storage bucket. The current data stream to be backed up is dynamically sliced ​​and divided into blocks, and each data block obtained after the division is used as the current data block to be backed up. It also performs a collision-resistant hash operation on the current data block to be backed up, generating the content fingerprint of the current data block to be backed up.

3. The block-level data backup management method for S3 object storage as described in claim 1, characterized in that, The existence determination of inputting the content fingerprint into the first-level Bloom filter for pre-indexing specifically includes: The content fingerprint is input as a query key into the first-level Bloom filter, and the content fingerprint is hashed through the first hash mapping set to obtain multiple slot indices of the content fingerprint in the first sparse bitmap. Determine the front index factor of the content fingerprint in the first-level Bloom filter based on all slot indexes; The false positive confidence of the first-level Bloom filter in its current state is determined based on the current bitmap capacity and the number of inserted fingerprints. The content fingerprint is pre-verified using the pre-index factor and the false judgment confidence to obtain the pre-verification pass mark of the content fingerprint. If the pre-verification pass mark is not passed, it is determined that the content fingerprint does not exist in the first-level Bloom filter, the current data block to be backed up is confirmed as a new data block, and a jump instruction to insert the content fingerprint into the first-level Bloom filter and the second-level Bloom filter is triggered. If the pre-verification pass mark is passed, it is determined that the content fingerprint may exist in the first-level Bloom filter, and the local hot index is accessed using the content fingerprint as the query keyword.

4. The block-level data backup management method for S3 object storage as described in claim 3, characterized in that, Determining the front index factor of the content fingerprint in the first-level Bloom filter based on all slot indices specifically includes: The multiple slot indices are sorted according to the order of hash mapping to obtain the slot index sequence of the content fingerprint in the first sparse bitmap. According to the slot index sequence, the corresponding slots in the first sparse bitmap are accessed sequentially, the binary bit values ​​of each slot are read, and the bit values ​​are combined into the bit value state vector of the content fingerprint in the first sparse bitmap according to the reading order. Based on the continuous distribution interval of logical truth values ​​in the bit-value state vector, the distribution span feature of logical truth values ​​in the bit-value state vector is extracted, and the content fingerprint is generated as a pre-index factor in the first-level Bloom filter based on the distribution span feature.

5. The block-level data backup management method for S3 object storage as described in claim 3, characterized in that, Pre-verification of the content fingerprint using the pre-index factor and the false positive confidence level specifically includes obtaining the pre-verification pass mark of the content fingerprint: The content fingerprint is subjected to a first-level check code extraction based on the pre-index factor to obtain the pre-check code of the content fingerprint; The confidence-gated transformation of the pre-check code is performed using the false positive confidence level to obtain the pre-confidence gate code; The verification flag corresponding to the content fingerprint is written to the pre-verification gate code to obtain the pre-verification pass mark of the content fingerprint.

6. The block-level data backup management method for S3 object storage as described in claim 1, characterized in that, When the first-level Bloom filter determines that the content fingerprint may exist but the query on the local hot index fails, the second-level Bloom filter is used to perform a secondary existence check on the expanded index. This specifically includes: When the first-level Bloom filter determines that the content fingerprint may exist and the query of the local hot index fails, the content fingerprint is input into the second-level Bloom filter and mapped through the second hash mapping set to obtain multiple slot indices of the content fingerprint in the second sparse bitmap. Construct the hit decision tree of the content fingerprint in the second-level Bloom filter based on all slot indexes; The content fingerprint is extended and verified using the hit decision tree to obtain the extended retrieval trigger value of the content fingerprint; When the extended retrieval trigger value is lower than the preset threshold, the current data block to be backed up is confirmed as a new data block, and a jump instruction to insert the content fingerprint into the first-level Bloom filter and the second-level Bloom filter is triggered. When the extended retrieval trigger value is not lower than a preset threshold, it is determined that the content fingerprint may exist in the secondary Bloom filter.

7. A block-level data backup management method for S3 object storage as described in claim 6, characterized in that, The extended retrieval trigger value of the content fingerprint obtained by performing extended index verification on the content fingerprint through the hit decision tree specifically includes: Based on the multiple slot indices of the content fingerprint in the second sparse bitmap, the tree nodes in the hit decision tree are matched with index landing points to obtain the extended index landing point set corresponding to the content fingerprint; Extract the binary bit values ​​of the corresponding leaf nodes along each decision path of the hit decision tree, and perform logical AND reduction on the bit values ​​of the leaf nodes on each decision path to obtain the path pass flag of each decision path. Based on the extended index landing set and all paths, the extended retrieval trigger value of the content fingerprint in the secondary Bloom filter is determined by flags.

8. A block-level data backup management method for S3 object storage as described in claim 1, characterized in that, When the secondary Bloom filter determines that a new data block may exist, a hierarchical deduplication lookup is performed using the full index of the S3 storage bucket and the tertiary Bloom filter file. The lookup results are used to confirm whether the data block to be backed up is a new data block. Specifically, this includes: When the secondary Bloom filter is determined to be possible, a remote index verification is performed on the tertiary Bloom filter file of the S3 storage bucket to obtain the remote index verification status. When the remote index verification status is valid, the content fingerprint is mapped and calculated to obtain multiple remote slot indexes; The remote deduplication gate value of the content fingerprint is generated based on all remote slot indexes, and the remote deduplication gate value is tokenized to obtain the lookup access token. When the remote index verification status is invalid, skip the slot mapping process of the three-level Bloom filter file and configure the lookup access token corresponding to the content fingerprint as an admission token. When the access token is a disabled token, the current data block to be backed up is confirmed to be a new data block, and the current data block to be backed up is uploaded to the allocated storage path of the S3 storage bucket; When the lookup access token is an admission token, the full index of the S3 bucket is accessed using the content fingerprint as the query key value, and the content fingerprint is used for exact matching lookup to obtain exact matching lookup results. When the search result is not found, the current data block to be backed up is confirmed as a new data block, and the current data block to be backed up is uploaded to the allocated storage path of the S3 storage bucket; When the search result is a match, it is confirmed that the data block to be backed up is an existing data block.

9. A block-level data backup management method for S3 object storage as described in claim 1, characterized in that, When the data block to be backed up is confirmed to be a new data block, the content fingerprint is inserted into the first-level Bloom filter and the second-level Bloom filter, and the updated second-level Bloom filter is synchronized to the S3 storage bucket. Specifically, this includes: When it is confirmed that the current data block to be backed up is a new data block, the content fingerprint is mapped according to the first hash mapping set to obtain multiple first slot indices, and the bit values ​​corresponding to the multiple first slot indices in the first sparse bitmap are set to logical true. The content fingerprint is mapped according to the second hash mapping set to obtain multiple second slot indices, and the bit values ​​corresponding to the multiple second slot indices in the second sparse bitmap are set to logical true. A snapshot of the updated second sparse bitmap is extracted and version information is attached to generate a synchronization file. The synchronization file is then uploaded to the preset storage directory of the S3 storage bucket, overwriting the existing old version file.

10. A block-level data backup management system for S3 object storage, used to execute a block-level data backup management method for S3 object storage as described in any one of claims 1 to 9, characterized in that, The block-level data backup management system includes: The hierarchical index building module is used to build first-level and second-level Bloom filters locally, store third-level Bloom filter files in the S3 bucket, and calculate the content fingerprint of the data block to be backed up. The pre-index judgment module is used to input the content fingerprint into the first-level Bloom filter to judge the existence of the pre-index. The extended index determination module is used to input the content fingerprint into the second-level Bloom filter to perform a second existence determination of the extended index when the first-level Bloom filter determines that it may exist and the query of the local hot index is not hit. The remote hierarchical deduplication module is used to perform hierarchical deduplication lookup by using the full index of the S3 storage bucket and the tertiary Bloom filter file when the secondary Bloom filter determines that the data block to be backed up is a new data block. The index synchronization update module is used to insert the content fingerprint into the first-level Bloom filter and the second-level Bloom filter when it is confirmed that the data block to be backed up is a new data block, and to synchronize the updated second-level Bloom filter to the S3 storage bucket.

Images