Fingerprint library reconstruction method, device, equipment, readable storage medium and program product

By reading the header of the data file and obtaining the header fingerprint and metadata of the data block in parallel, the problem of low fingerprint database reconstruction efficiency in the existing technology is solved, and more efficient fingerprint database reconstruction is achieved.

CN120973313BActive Publication Date: 2026-08-04广州鼎甲计算机科技有限公司
View PDF 1 Cites 0 Cited by

Patent Information

Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
广州鼎甲计算机科技有限公司
Filing Date
2025-08-19
Publication Date
2026-08-04

AI Technical Summary

Technical Problem

In public cloud environments, existing technologies require waiting for the RTT (Real-Time To Request) when reading the header of a data file using the serial request method, resulting in low fingerprint database reconstruction efficiency and increased object storage processing time.

Method used

By reading the header of the data file, the header offset of the data block is obtained based on the bit set carried in the header. The header of the data block is read in parallel to obtain the data block fingerprint and metadata, and then inserted into the fingerprint database.

Benefits of technology

It reduces the reading of invalid data, saves traffic and bandwidth consumption, avoids bandwidth becoming a bottleneck for rebuilding the fingerprint database, saves RTT and object storage processing time for read requests, and improves the efficiency of fingerprint database reconstruction.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN120973313B_ABST
    Figure CN120973313B_ABST
Patent Text Reader

Abstract

The application relates to a fingerprint library reconstruction method and device, equipment, a readable storage medium and a program product, and relates to the technical field of data recovery backup. The method comprises the following steps: reading the head part of a data file; acquiring the head part offset of each data block in the data file according to a bit set carried in the head part of the data file; reading the head part of each data block in parallel according to the head part offset of each data block, and obtaining the data block fingerprint and metadata carried in the head part of each data block; and inserting the data block fingerprint and metadata corresponding to each data block into a fingerprint library. The method can reduce the reading of invalid data and read the head part of each data block in parallel, so as to improve the fingerprint library reconstruction efficiency.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This application relates to the field of data recovery and backup technology, and in particular to a fingerprint database reconstruction method, apparatus, device, readable storage medium, and program product. Background Technology

[0002] In the field of data recovery and backup, rebuilding the fingerprint database of an object deduplication pool typically requires reading data files from object storage and identifying data block headers to extract fingerprints and metadata. Traditionally, reading data files involves a full read, meaning reading the entire file content. However, most of the data file content consists of data block data or invalid data used for alignment padding, with data block headers accounting for a very small percentage. Therefore, most bandwidth and traffic are consumed on non-target data, resulting in low fingerprint database reconstruction efficiency. Currently, a serial request method is commonly used, adding a Range header to the GetObject request. This allows reading only data block headers, reducing the reading of invalid data.

[0003] However, in a public cloud environment, if the serial request method is used to read data files, after each data block header is read, it is necessary to wait for one round of RTT (Round Trip Time) to know the offset of the next data block header, and then initiate a read request for the next data block. Therefore, the time for object storage to process requests is increased in the overall database reconstruction process, which affects the efficiency of fingerprint database reconstruction. Summary of the Invention

[0004] Therefore, it is necessary to provide a fingerprint database reconstruction method, apparatus, device, readable storage medium, and program product that can improve the efficiency of fingerprint database reconstruction in response to the above-mentioned technical problems.

[0005] Firstly, this application provides a fingerprint database reconstruction method, including:

[0006] Read the header of the data file;

[0007] Based on the set of bits carried in the header of the data file, obtain the header offset of each data block in the data file;

[0008] Based on the header offset of each data block, the headers of each data block are read in parallel to obtain the data block fingerprint and metadata carried in the header of each data block.

[0009] Insert the data block fingerprint and metadata corresponding to each data block into the fingerprint database.

[0010] In one embodiment, the step of reading the headers of each data block in parallel based on the header offset of each data block to obtain the data block fingerprint and metadata carried in the header of each data block includes:

[0011] Based on the header offset of each data block, obtain the data block header read request corresponding to each data block;

[0012] Initiate parallel read requests for the headers of each data block to the object storage to obtain the data block fingerprint and metadata carried in the header of each data block;

[0013] The data block header read request corresponding to the data block is used by the object storage to read the header of the data block according to the header offset of each data block, and return the data block fingerprint and metadata carried in the header of the data block.

[0014] In one embodiment, the step of reading the headers of each data block in parallel based on the header offset of each data block to obtain the data block fingerprint and metadata carried in the header of each data block includes:

[0015] The header offset of each data block is filled into a byte range request header to obtain a byte range request header indicating that the data filling is complete.

[0016] Based on the byte range request header for filling the offset, the corresponding data block header read request is obtained.

[0017] Initiate a data block header read request to the object storage to obtain the data block fingerprint and metadata carried in the header of each data block;

[0018] The data block header read request is used by the object storage to read the header of each data block according to the header offset of each data block, and return the data block fingerprint and metadata carried in the header of each data block.

[0019] In one embodiment, the method further includes:

[0020] In response to a data file write request, retrieve multiple data blocks to be written;

[0021] The data to be written corresponding to each of the data blocks to be written is sequentially written to the writable fragment, and the corresponding bit flag is set in the bit set according to the number of alignment blocks occupied by each of the data blocks to be written; the writable fragment is stored in object storage;

[0022] When the target termination condition is met, the file header data to be written is obtained based on the current bit set and the data already written, and the file header data to be written is written into the header fragment; the header fragment is stored in the object storage;

[0023] The header fragments that have completed data writing and the writable fragments that have completed data writing are merged to obtain the data file corresponding to the data file write request.

[0024] In one embodiment, the step of sequentially writing the data to be written corresponding to each of the data blocks to be written into the writable fragment, and setting the corresponding bit flag in the bit set according to the number of alignment blocks occupied by each of the data blocks to be written, includes:

[0025] According to the order of the data blocks to be written, the data to be written corresponding to each data block to be written is written to the writable fragment in sequence;

[0026] Before writing the data to be written to the current writable fragment corresponding to the current data block to be written, it is determined whether the fragment switching conditions are met. The fragment switching conditions include: no transmission session is currently open, or the sum of the size of the written data of the current writable fragment and the size of the data to be written corresponding to the current data block to be written is greater than a target threshold. The transmission session is used to interact with the object storage.

[0027] If so, complete the data writing for the current writable fragment, create a writable fragment in the transmission session, open the transmission session, write the data to be written corresponding to the current data block to be written to the created writable fragment, and set the corresponding bit flag in the bit set according to the number of alignment blocks occupied by the data to be written corresponding to the current data block to be written.

[0028] If not, the data to be written corresponding to the current data block to be written is written to the current writable fragment, and the corresponding bit flag is set in the bit set according to the number of alignment blocks occupied by the data to be written corresponding to the current data block to be written.

[0029] In one embodiment, before sequentially writing the data to be written corresponding to each of the data blocks to be written to the writable fragment, the method further includes:

[0030] If the target termination condition is not met, the data block data of the current data block to be written is obtained. Based on the data block data of the current data block to be written, the data block header and data block alignment padding data of the current data block to be written are obtained. The data block header, data block data and data block alignment padding data of the current data block to be written are concatenated to obtain the data to be written corresponding to the current data block to be written.

[0031] Secondly, this application also provides a fingerprint database reconstruction apparatus, comprising:

[0032] The read module is used to read the header of the data file;

[0033] The acquisition module is used to acquire the header offset of each data block in the data file based on the bit set carried in the header of the data file.

[0034] The parallel reading module is used to read the header of each data block in parallel according to the header offset of each data block, so as to obtain the data block fingerprint and metadata carried in the header of each data block.

[0035] An insertion module is used to insert the data block fingerprint and the metadata corresponding to each data block into the fingerprint database.

[0036] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:

[0037] Read the header of the data file;

[0038] Based on the set of bits carried in the header of the data file, obtain the header offset of each data block in the data file;

[0039] Based on the header offset of each data block, the headers of each data block are read in parallel to obtain the data block fingerprint and metadata carried in the header of each data block.

[0040] Insert the data block fingerprint and metadata corresponding to each data block into the fingerprint database.

[0041] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, performs the following steps:

[0042] Read the header of the data file;

[0043] Based on the set of bits carried in the header of the data file, obtain the header offset of each data block in the data file;

[0044] Based on the header offset of each data block, the headers of each data block are read in parallel to obtain the data block fingerprint and metadata carried in the header of each data block.

[0045] Insert the data block fingerprint and metadata corresponding to each data block into the fingerprint database.

[0046] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, performs the following steps:

[0047] Read the header of the data file;

[0048] Based on the set of bits carried in the header of the data file, obtain the header offset of each data block in the data file;

[0049] Based on the header offset of each data block, the headers of each data block are read in parallel to obtain the data block fingerprint and metadata carried in the header of each data block.

[0050] Insert the data block fingerprint and metadata corresponding to each data block into the fingerprint database.

[0051] The aforementioned fingerprint database reconstruction method, apparatus, device, readable storage medium, and program product read the header of a data file, obtain the header offset of each data block in the data file based on the bit set carried in the header of the data file, and read the headers of each data block in parallel based on the header offset of each data block to obtain the data block fingerprint and metadata carried in the header of each data block. The fingerprints and metadata corresponding to each data block are then inserted into the fingerprint database. Therefore, this scheme can obtain the data block fingerprints and metadata used for fingerprint database reconstruction simply by reading the header of the data file. This reduces the reading of invalid data, saves traffic and bandwidth consumption, avoids bandwidth becoming a bottleneck for fingerprint database reconstruction, and avoids affecting other communication activities on the same node. Moreover, by obtaining the header offset of each data block in the data file based on the bit set carried in the header of the data file, the position of each data block in the data file can be determined. This allows for parallel reading of the headers of each data block during data file scanning, thereby saving RTT and object storage processing time for read requests and improving fingerprint database reconstruction efficiency. In addition, the bit set is built into the data file and can be deleted or overwritten as the data file is deleted, so there is no need to consider consistency issues. Attached Figure Description

[0052] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0053] Figure 1 This is a flowchart illustrating a fingerprint database reconstruction method in one embodiment;

[0054] Figure 2 This is a schematic diagram illustrating the mapping between intermediate sets and data blocks in one embodiment;

[0055] Figure 3 This is a flowchart illustrating the data file writing step in one embodiment;

[0056] Figure 4 This is a schematic diagram illustrating the distribution of internal data in an unmerged fragment in one embodiment;

[0057] Figure 5 This is a flowchart illustrating the fingerprint database reconstruction method in another embodiment;

[0058] Figure 6 This is a structural block diagram of a fingerprint database reconstruction device in one embodiment;

[0059] Figure 7 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0060] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0061] It should be noted that the terms "comprising" and "having," and any variations thereof, as used in this application, are intended to cover non-exclusive inclusion. The term "multiple" as used in this application refers to two or more. The term "and / or" as used in this application refers to one of the solutions, or any combination of multiple solutions.

[0062] The following is an explanation of some of the terms and technical abbreviations used in this application.

[0063] 1. Bitset: A data structure that uses 8 bits to compress and store 8 Boolean type variables. This solution uses a dynamically expandable bitset.

[0064] 2. GetObject Request: A widely implemented request by various object storage vendors, used to download the metadata and data of a specified object from the object storage. A Range HTTP header can be appended to the request to indicate the range of data to be downloaded; if the storage service implementation allows, a Range HTTP header containing multiple ranges can also be appended to download multiple data ranges simultaneously in a single request. HTTP (Hypertext Transfer Protocol) is an application-layer protocol used to transfer hypertext between clients (such as browsers) and servers.

[0065] 3. MultipartUpload Request: A type of request widely implemented by various object storage vendors, used to upload multiple parts of an object in chunks, ultimately merging them into a single complete object. The process of writing an object primarily involves three requests: CreateMultipartUpload (initialize chunked upload), UploadPart (upload chunks), and CompleteMultipartUpload (complete chunked upload). CreateMultipartUpload creates a chunked upload task and returns a task ID (Identifier); UploadPart uploads the specified chunks and returns the chunk ETag; and CompleteMultipartUpload completes the specified chunked upload task. Uploaded chunks are merged into the final object sequentially according to their respective sequence numbers during CompleteMultipartUpload. UploadPart allows for flexible specification of the sequence number of each uploaded chunk, making object uploads flexible. The sequence number range for UploadPart is typically 1 to 10000.

[0066] 4. RTT: The time between sending a network request and receiving a response, used to measure network latency and network congestion.

[0067] 5. QPS (Query Per Second): The number of network requests per second, used to measure the concurrent performance of communication between the client and the server.

[0068] 6. Key-value databases: Compared to relational databases, key-value databases use unique keys to index the values ​​to be stored.

[0069] 7. Alignment Blocks: Data files need to be aligned to a specific length; this specific length of block is called an alignment block. Data files are composed of alignment blocks. When the length of a data block is not an integer multiple of the length of the alignment block, padding data needs to be appended after the data block to meet the alignment requirements.

[0070] 8. Data Block: A data block is composed of a data block header and data block data concatenated in sequence. The data block header contains fields describing the data block data, including fingerprint, offset, and data length.

[0071] 9. ETag (Entity Tag): An identifier defined in HTTP used to identify different versions of the same resource. When the content of the resource changes, the ETag also changes.

[0072] To enable those skilled in the art to better understand this application, the relevant technologies are first introduced below.

[0073] In the backup field, backup systems typically pool the storage they use to abstract the storage and simplify its use and management. A deduplication pool (hereinafter referred to as a deduplication pool) is a commonly used storage pool, characterized by its ability to save storage space and accelerate write speeds in multiple backups of the same target, especially full backups. The principle of a deduplication pool is to divide the data to be written into blocks, calculate the fingerprint of each data block, and then select data blocks that have not been stored in the deduplication pool for writing based on the fingerprints.

[0074] Deduplication pools typically use a "fingerprint database-index file-data file" model: the fingerprint database is implemented based on a key-value database, where the key is the fingerprint and the value is the metadata of the data block. The metadata includes information such as the data file where the data block is located, its offset, and the data block length. The fingerprint database provides fingerprint-based data block addressing services. The index file only records the fingerprint and does not need to know the actual location of the data block. This design increases the flexibility of data block storage. The index file corresponds one-to-one with the files that the deduplication pool user wants to store, and it records the fingerprint of each data block and the metadata of the remaining data blocks in sequence. The data file stores the data block header, which includes the fingerprint and other metadata, as well as the data block data. The data file is usually aligned to a certain size to improve read and write performance. Among them, the key-value database used by the fingerprint database has high requirements for disk random read and write performance, so it is usually stored on solid-state drives (SSDs); the index file and data file are mostly read and written sequentially and have a large amount of data, so they are usually stored on high-capacity hard disk arrays with redundancy mechanisms.

[0075] An object storage-based deduplication pool (hereinafter referred to as an object deduplication pool) is a deduplication pool that utilizes the high scalability and high reliability of object storage to store index files and data files. Since object storage does not provide random write services, data files in the object deduplication pool are only written to during creation and are not modified subsequently.

[0076] To meet performance and consistency requirements, the object deduplication pool's fingerprint database is stored locally on a solid-state drive (SSD) and not synchronized to object storage. Compared to object storage, SSDs are less reliable; if they fail, the fingerprint database must be rebuilt for the object deduplication pool to continue providing services. Rebuilding the fingerprint database requires reading the fingerprints and metadata recorded in the data files and rewriting them into the database. Since scanning the data files is necessary to ensure the existence of the data blocks referenced by the fingerprints, rebuilding the fingerprint database typically involves scanning the data files, not the index files. Furthermore, the data block fingerprints and metadata used for rebuilding the fingerprint database are stored in the headers of each data block in the data files. Additionally, the object deduplication pool remains locked and unavailable until the fingerprint database rebuild is complete; therefore, the fingerprint database rebuilding process can be considered free from other business competition.

[0077] Based on this, this application provides a fingerprint database reconstruction method, apparatus, device, readable storage medium, and program product, which can reduce the reading of invalid data and save the time of object storage processing read requests, thereby improving the efficiency of fingerprint database reconstruction. In an exemplary embodiment, such as Figure 1 As shown, a fingerprint database reconstruction method is provided. This embodiment illustrates the method applied to a terminal, but it is understood that the method can also be applied to a server, or to a system including both a terminal and a server, and implemented through interaction between the terminal and the server. In this embodiment, the method includes steps 101 to 104. Wherein:

[0078] Step 101: Read the header of the data file.

[0079] The header of the data file includes a bit set and the remaining fields (including the bit set field). The total size of the data file header and its alignment padding is a fixed value. This fixed value is the total size of the data file header and its alignment padding when the total size of the data blocks and their alignment padding reaches a set maximum value. Therefore, when the total size of the data blocks and their alignment padding is small, the alignment padding in the data file header may occupy more than one alignment block to maintain a constant total size.

[0080] For example, the header of the data file is read by initiating a GetObject request with the range being the header of the data file to the object storage. Since the length of the bit set is variable, the length of the data file header is also variable. Therefore, when reading the header of the data file, the length of the data file header needs to be calculated based on the size of the data file to fill the Range header of the GetObject request. In this embodiment, the length of the data file header can be calculated as follows: Length of data file header = ceil(size of data file / length of alignment padding data in the data file header / number of bits per byte in the bit set) + length of the remaining fields in the data file header. Here, ceil is a round-up function used to round the input value up to the nearest integer (e.g., if the input is 3.2, then 4 is obtained).

[0081] Step 102: Obtain the header offset of each data block in the data file based on the bit set carried in the header of the data file.

[0082] Specifically, a bit set is added to the header of the data file. This bit set records the alignment block where the header of each data block is located. Based on the alignment block where the header of each data block is located, the header offset of each data block can be determined, providing the conditions for parallel reading of the headers of each data block. Based on the header offset of each data block, the header position of each data block can be determined, thereby accurately locating and reading the header of each data block. In this embodiment, since the total size of the data file header and the alignment padding data of the data file header is a fixed value, the header offset of the data block can be: Header offset of data block = Total size of the data file header and the alignment padding data of the data file header + Alignment block length × Alignment block number.

[0083] For example, the data file is aligned to 512 bytes, and the mapping between bit sets and data blocks is as follows: Figure 2 As shown. In this embodiment, the bit set is 11001101. In the bit set, a bit set to 1 indicates that the corresponding alignment block contains a data block header, and a bit set to 0 indicates that the corresponding alignment block does not contain a data block header. Since each new data block starts from the next alignment block, the beginning of the alignment block corresponding to a bit set to 1 stores the data block header. In summary, based on this bit set, we can know that n×512, (n+1)×512, (n+4)×512, (n+5)×512, and (n+7)×512 are the starting positions of the data block header; where n is the order of the alignment block corresponding to data block 1 in the data file. Furthermore, the data file header and its alignment padding data are not recorded in the bit set; that is, the first bit in the bit set corresponds to the first alignment block after the data file header and its alignment padding data.

[0084] Step 103: Based on the header offset of each of the above data blocks, read the header of each of the above data blocks in parallel to obtain the data block fingerprint and metadata carried in the header of each of the above data blocks.

[0085] The header offset of each data block is used to determine the start position of the header of each data block. Combined with the length of the header of each data block, the reading range of the header of each data block can be determined. Parallel reading of the headers of each data block refers to using multiple threads to read the headers of each data block simultaneously, which can reduce the overall reading time of the data file. The data block fingerprint and metadata carried in the header of each data block are the fingerprints and metadata recorded in the data file, used to reconstruct the fingerprint database.

[0086] For example, a GetObject request with the range as the header of a data block is initiated to the object storage to read the header of the data block. Specifically, the start position of the header of each data block is obtained based on the bit set, and the content to be filled in the Range header of the GetObject request is determined based on the start position and length of the header of each data block. Step 104: The fingerprint of each data block and the metadata corresponding to each of the aforementioned data blocks are inserted into the fingerprint database.

[0087] In the aforementioned fingerprint database reconstruction method, the header of the data file is read. Based on the bit set carried in the header, the header offset of each data block in the data file is obtained. Based on the header offset of each data block, the headers of each data block are read in parallel to obtain the data block fingerprint and metadata carried in the header of each data block. The corresponding data block fingerprint and metadata are then inserted into the fingerprint database. Therefore, this scheme can obtain the data block fingerprints and metadata used for fingerprint database reconstruction simply by reading the data file header. This reduces the reading of invalid data, saves traffic and bandwidth consumption, avoids bandwidth becoming a bottleneck for fingerprint database reconstruction, and avoids affecting other communication activities on the same node. Furthermore, by obtaining the header offset of each data block in the data file based on the bit set carried in the header, the position of the header of each data block in the data file can be determined. This allows for parallel reading of the headers of each data block during data file scanning, thereby saving RTT and object storage processing time for read requests and improving fingerprint database reconstruction efficiency. In addition, the bit set is built into the data file and can be used when the data file is deleted or overwritten, thus eliminating the need to consider consistency issues.

[0088] In an exemplary embodiment, in step 103 above, the headers of each data block are read in parallel according to the header offset of each data block to obtain the data block fingerprint and metadata carried in the header of each data block, including:

[0089] Based on the header offset of each of the aforementioned data blocks, a data block header read request corresponding to each of the aforementioned data blocks is obtained; the data block header read request corresponding to each of the aforementioned data blocks is initiated in parallel to the object storage to obtain the data block fingerprint and metadata carried in the header of each of the aforementioned data blocks; wherein, the data block header read request corresponding to each of the aforementioned data blocks is used by the object storage to read the header of the aforementioned data block based on the header offset of each of the aforementioned data blocks, and return the data block fingerprint and metadata carried in the header of the aforementioned data block.

[0090] It should be noted that since the object deduplication pool uses object storage as its large-capacity storage, users usually do not configure local large-capacity storage for the object deduplication pool. Therefore, data files are written directly to object storage when they are created, and the fingerprint database needs to be read from object storage in a streaming manner when it is rebuilt. That is, local storage cannot be used for temporary storage when reading and writing data files.

[0091] For example, for each data block, the byte range of the data block header is determined based on the header offset of the data block and the length of the data block header. Based on the byte range, a GetObject request (i.e., a data block header read request) with the range of the data block header is initiated to the object storage to read the header of the data block; wherein, the byte range is used to fill the Range header of the GetObject request.

[0092] In this embodiment, based on the header offset recorded in the data block header read record and the bit set, it can be further determined whether there is a missed header offset in the bit set (i.e., the offset of the header of an unread data block). If so, a data block header read request corresponding to the data block can be obtained based on the header offset, and the data block header read request can be initiated to the object storage to read the data block fingerprint and metadata carried in the data block header from the object storage. In an exemplary embodiment, in step 103 above, based on the header offset of each of the above data blocks, the headers of each of the above data blocks are read in parallel to obtain the data block fingerprint and metadata carried in the header of each of the above data blocks, including:

[0093] The header offset of each of the aforementioned data blocks is filled into a byte range request header; based on the byte range request header with the offset filled in, the corresponding data block header read request is obtained; the aforementioned data block header read request is sent to the object storage to obtain the data block fingerprint and metadata carried in the header of each of the aforementioned data blocks; wherein, the aforementioned data block header read request is used by the object storage to read the header of each of the aforementioned data blocks based on the header offset of each of the aforementioned data blocks, and return the data block fingerprint and metadata carried in the header of each of the aforementioned data blocks.

[0094] The Byte Range Request Header is the header of a data block header read request, used to limit the range of bytes to be read from object storage.

[0095] For example, based on the header offset of each data block and the length of each data block, the byte range of the header of each data block is determined. The byte range of the header of each data block is then filled into the Range header (i.e., the byte range request header) of a GetObject request, thereby obtaining the corresponding GetObject request. This allows the headers of each data block of the data file to be read in batches using a single GetObject request, thereby reducing the overhead of multiple RTTs in each worker thread.

[0096] In one exemplary embodiment, such as Figure 3 As shown, the method in this embodiment further includes a data file writing step, which includes steps 301 to 304. Wherein:

[0097] Step 301: In response to the data file write request, obtain multiple data blocks to be written.

[0098] For example, in response to a data file write request, multiple data blocks to be written are obtained. Based on the key of the allocated data file object, a CreateMultipartUpload request is initiated to the object storage to create a fragmented upload task for the data file object. The ID of the fragmented upload task is obtained in the response. For the fragmented upload task, the initialization bit set is empty, and the sequence number of the current writable fragment is initialized to 2 (the fragment with sequence number 1 refers to the header fragment).

[0099] The key of an object in a data file is a unique identifier for that object in the bucket, which can be used to represent the storage path of the object in the object storage.

[0100] Step 302: Write the data to be written corresponding to each of the above-mentioned data blocks to be written to the writable fragments in sequence, and set the corresponding bit flags in the bit set according to the number of alignment blocks occupied by each of the above-mentioned data blocks to be written; the above-mentioned writable fragments are stored in object storage.

[0101] The bit markers in the bit set can be used to record the alignment block where the header of each data block of the data file is located. Based on the alignment block where the header of each data block of the data file is located, the offset of the header of each data block of the data file can be obtained, and then the position of the header of each data block can be obtained based on the offset of the header of each data block. Since writable fragments are one of the components of the data file, after writing the data to be written corresponding to each data block to be written to the writable fragment, the specific position of the header of each data block to be written in the data file can be known. In this embodiment, since the total size of the header of the data file and the alignment padding data of the header of the data file is a fixed value, the header offset of the data block can be: header offset of the data block = total size of the header of the data file and the alignment padding data of the header of the data file + alignment block length × alignment block number.

[0102] For example, before writing the data to be written to each data block to be written to the writable fragment, it is determined whether the target end condition is met. If not, the writable fragment corresponding to the current data block to be written is generated based on the data block data of the current data block to be written (i.e., the data carried in the current data block to be written). Based on the number of alignment blocks occupied by the current data to be written, the corresponding bit flag is set in the bit set.

[0103] In one embodiment, the header of the current data block to be written is in the first aligned block of the data file, so the first bit of the bit set is set to 1 and the other bits are set to 0.

[0104] Step 303: When the target termination condition is met, obtain the file header data to be written based on the current bit set and the data already written, and write the file header data to be written into the header fragment; the header fragment is stored in object storage.

[0105] The target termination condition refers to the condition that must be met when writing data blocks ends. "Written data" refers to data that has already been written to writable data blocks. In this embodiment, the target termination condition can be: the size of the written data blocks reaches a set value, or there are no more data blocks to write. A written data block is one whose data block data has been written to a writable data block. Data block data refers to the data carried within a data block.

[0106] For example, when the target termination condition is met, based on the current bit set and the data already written, the remaining fields of the data file header, the bit set, and the alignment padding data for the data file header are generated, such as... Figure 4As shown, following the order of "remaining fields of the data file header - bit set - alignment padding data of the data file header," the remaining fields of the data file header, the bit set, and the alignment padding data of the data file header are concatenated to form the file header data to be written. Based on the object key of the data file, the ID of the fragmented upload task, and the sequence number of the header fragment, an UploadPart request is initiated to the object storage to write the file header data to be written into the header fragment. For example, Figure 4 As shown, the remaining fields of the data file header include the bit set length. The total size of the data file header and its alignment padding data is a fixed value. The data file header includes the remaining fields and the bit set. In this embodiment, the total size of the data file header and its alignment padding data can be: Total size of the data file header and its alignment padding data = Length of the remaining fields of the data file header + Maximum value of the total size of the written data blocks and their alignment padding data / Size of the written data blocks' alignment padding data / Bits per byte.

[0107] Step 304: Merge the header fragments that have completed data writing and the writable fragments that have completed data writing to obtain the data file corresponding to the above data file write request.

[0108] For example, after creating a chunked upload task for a data file, the following initialization is performed for the chunked upload task: the initialization bit set is empty, the sequence number of the current writable chunk is initialized to 2, and the chunk with sequence number 1 is used as the header chunk (i.e., the current writable chunk is chunk 2, and the header chunk is chunk 1). Based on the above initialization, the chunked upload task is performed. When the target termination condition is met, the writable chunks that have completed data writing include chunks 2 to 6. Based on the key of the data file object, the ID of the chunked upload task, the sequence numbers 1 to 6 (i.e., the sequence numbers of all chunks that have completed data writing), and ETag, a CompleteMultipartUpload request is initiated to the object storage to synthesize the current chunks 1 to 6 to obtain the data file corresponding to the above data file write request.

[0109] In an exemplary embodiment, in step 302 above, the data to be written corresponding to each of the aforementioned data blocks to be written is sequentially written to the writable fragment, and a corresponding bit flag is set in the bit set according to the number of alignment blocks occupied by each of the aforementioned data blocks to be written, including:

[0110] According to the order of the aforementioned data blocks to be written, the data to be written corresponding to each of the aforementioned data blocks to be written is sequentially written to the writable fragment; before writing the data to be written corresponding to the current data block to be written to the current writable fragment, it is determined whether the fragment switching conditions are met; wherein, the fragment switching conditions include: the current transmission session is not open, or, the sum of the size of the written data of the current writable fragment and the size of the data to be written corresponding to the current data block to be written is greater than a target threshold; the aforementioned transmission session is used to interact with the aforementioned object storage; if yes, then the data writing to the current writable fragment is completed, a writable fragment is created in the aforementioned transmission session, the aforementioned transmission session is opened, the data to be written corresponding to the current data block to be written is written to the created writable fragment, and the corresponding bit flag is set in the bit set according to the number of alignment blocks occupied by the data to be written corresponding to the current data block to be written; if no, then the data to be written corresponding to the current data block to be written is written to the current writable fragment, and the corresponding bit flag is set in the bit set according to the number of alignment blocks occupied by the data to be written corresponding to the current data block to be written.

[0111] In one embodiment, the current writable fragment number is 2 (i.e., the current writable fragment is fragment 2), the current data block to be written is data block 3, and the target termination condition is not met (indicating that data blocks need to be written). Since the transmission session is currently open, and the sum of the size of the data already written in fragment 2 and the size of the data to be written corresponding to data block 3 is not greater than the target threshold, it is not necessary to create a new writable fragment. At this time, the data to be written corresponding to data block 3 is written to fragment 2, and the corresponding bit flag is set in the bit set according to the number of alignment blocks occupied by the data to be written corresponding to data block 3.

[0112] In another embodiment, the current writable fragment sequence number is 3 (i.e., the current writable fragment is fragment 3), the current data block to be written is data block 8, and the target termination condition is not met (indicating that data blocks need to be written). Since the sum of the size of the data already written in fragment 3 and the size of the data to be written corresponding to data block 8 is greater than the target threshold, a new writable fragment needs to be created. At this time, the open transmission session is closed, and an UploadPart request is initiated to the object storage according to the object key of the data file, the ID of the fragment upload task, and the current writable fragment sequence number to create a writable fragment in the transmission session. The transmission session is then opened, that is, an HTTP connection is established between the terminal and the object storage. The sequence number of the writable fragment is recorded as 4 (i.e., the writable fragment is fragment 4). The terminal sends an HTTP message body carrying the data to be written corresponding to data block 8 to the object storage through the HTTP connection, and writes the data to be written corresponding to data block 8 to fragment 4. According to the number of alignment blocks occupied by the data to be written corresponding to data block 8, the corresponding bit flag is set in the bit set.

[0113] In an exemplary embodiment, the method of this application embodiment further includes a data acquisition step before step 302 above, which includes:

[0114] If the above target termination condition is not met, then obtain the data block data of the current data block to be written, obtain the data block header and data block alignment padding data of the current data block to be written based on the data block data of the current data block to be written, and concatenate the above data block header, the above data block data and the above data block alignment padding data of the current data block to be written to obtain the data to be written corresponding to the current data block to be written.

[0115] Since the header of the data file and the total size of the alignment padding data in the header of the data file are fixed values, the offset field in the header of the data block (i.e., the header offset of the data block in the data file after it is written) is determinable, and therefore the header of the data block is also determinable.

[0116] For example, such as Figure 4 As shown, following the order of "the header of the current data block to be written - the data block data of the current data block to be written - the data block alignment padding data of the current data block to be written", the header of the current data block to be written, the data block data, and the data block alignment padding data are concatenated to form the data to be written corresponding to the current data block to be written, which facilitates the subsequent writing of the data to be written corresponding to the current data block to be written to the writable fragment.

[0117] In one exemplary embodiment, such as Figure 5As shown, a database reconstruction method is also provided, which includes steps 501 to 509. Wherein:

[0118] Step 501: In response to the data file write request, obtain multiple data blocks to be written.

[0119] Step 502: Write the data to be written to each data block to be written to the writable fragment in sequence, and set the corresponding bit flag in the bit set according to the number of alignment blocks occupied by each data block to be written.

[0120] Step 502 above may include: sequentially writing the data to be written corresponding to each data block to be written to the writable fragment according to the order of each data block to be written; before writing the data to be written corresponding to the current data block to be written to the current writable fragment, determining whether the fragment switching condition is met; wherein the fragment switching condition includes: the current transmission session is not open, or the sum of the size of the written data of the current writable fragment and the size of the data to be written corresponding to the current data block to be written is greater than a first target threshold; the transmission session is used to interact with the object storage; if yes, then the data writing of the current writable fragment is completed, a writable fragment is created in the transmission session, the transmission session is opened, the data to be written corresponding to the current data block to be written is written to the created writable fragment, and the corresponding bit flag is set in the bit set according to the number of alignment blocks occupied by the data to be written corresponding to the current data block to be written; if no, then the data to be written corresponding to the current data block to be written is written to the current writable fragment, and the corresponding bit flag is set in the bit set according to the number of alignment blocks occupied by the data to be written corresponding to the current data block to be written.

[0121] Step 503: If the target termination condition is not met, obtain the data block data of the current data block to be written, obtain the data block header and data block alignment padding data of the current data block to be written based on the data block data of the current data block to be written, and concatenate the data block header, data block data and data block alignment padding data of the current data block to be written to obtain the data to be written corresponding to the current data block to be written.

[0122] Step 504: When the target termination condition is met, obtain the file header data to be written based on the current bit set and the data already written, and write the file header data to be written into the header fragment.

[0123] Step 505: Merge the header fragments that have completed data writing and the writable fragments that have completed data writing to obtain the data file corresponding to the data file write request.

[0124] Step 506: Read the header of the data file.

[0125] Step 507: Obtain the header offset of each data block in the data file based on the bit set carried in the header of the data file.

[0126] Step 508: Based on the header offset of each data block, synchronously read the header of each data block to obtain the data block fingerprint and metadata carried in the header of each data block.

[0127] Step 508 may include: obtaining a data block header read request corresponding to each data block based on the header offset of each data block; initiating a parallel data block header read request corresponding to each data block to the object storage to obtain the data block fingerprint and metadata carried in the header of each data block; wherein, the data block header read request corresponding to each data block is used by the object storage to read the header of the data block based on the header offset of each data block and return the data block fingerprint and metadata carried in the header of the data block.

[0128] Step 508 may further include: filling the header offset of each data block into a byte range request header to obtain a byte range request header for completed data filling; obtaining the corresponding data block header read request based on the byte range request header for completed offset filling; initiating a data block header read request to the object storage to obtain the data block fingerprint and metadata carried in the header of each data block; wherein, the data block header read request is used by the object storage to read the header of each data block based on the header offset of each data block and return the data block fingerprint and metadata carried in the header of each data block.

[0129] Step 509: Insert the data block fingerprint and metadata corresponding to each data block into the fingerprint database.

[0130] In this embodiment, when writing data to be written, including data block data and data block headers, to a data file, a corresponding bit flag is set in the bit set according to the number of alignment blocks occupied by the data to be written, and the bit set is placed at the beginning of the data file. By reading the bit set carried in the header of the data file, the header offset of each data block written to the data file can be obtained, thereby determining the specific position of the header of each data block in the data file. Based on the header offset of each data block, parallel data block header read requests corresponding to each data block can be initiated to the object storage, or the header offset of each data block can be filled into the byte range of a data block header read request. The system requests headers and initiates read requests for the headers of data blocks to object storage, thus parallelizing the reading of data block headers. Based on this, each data block header read request no longer needs to wait for the RTT of other requests, and the RTTs of concurrent requests overlap. Therefore, the bottleneck of fingerprint database reconstruction shifts from RTT to the ability of the local node to initiate requests and the QPS limitation imposed by the object storage provider, significantly improving the QPS ceiling. Furthermore, by batch reading multiple data block headers with a single data block header read request containing the header offsets of each data block, corresponding RTT and object storage request processing time can be saved for each worker thread. Additionally, the bit set is built into the header of the data file and can be deleted or overwritten with the data file, eliminating the need to consider the consistency between the bit set used to obtain the header offsets and the data file during read and write operations.

[0131] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps. It is understood that the steps in different embodiments can be freely combined as needed, and all non-contradictory solutions formed by such combinations are within the scope of protection of this application.

[0132] Based on the same inventive concept, this application also provides a fingerprint database reconstruction apparatus for implementing the fingerprint database reconstruction method described above. The solution provided by this apparatus is similar to the implementation described in the above method; therefore, the specific limitations in one or more fingerprint database reconstruction apparatus embodiments provided below can be found in the limitations of the fingerprint database reconstruction method described above, and will not be repeated here.

[0133] In one exemplary embodiment, such as Figure 6 As shown, a fingerprint database reconstruction device is provided, comprising: a reading module 601, an acquisition module 602, a parallel reading module 603, and an insertion module 604, wherein:

[0134] Reading module 601 is used to read the header of the data file;

[0135] The acquisition module 602 is used to acquire the header offset of each data block in the data file based on the bit set carried in the header of the data file.

[0136] The parallel reading module 603 is used to read the header of each data block in parallel according to the header offset of each data block, so as to obtain the data block fingerprint and metadata carried in the header of each data block.

[0137] The insertion module 604 is used to insert the data block fingerprint and the metadata corresponding to each data block into the fingerprint database.

[0138] In an exemplary embodiment, the parallel reading module 603 is configured to obtain a data block header read request corresponding to each data block based on the header offset of each data block; and to initiate parallel data block header read requests corresponding to each data block to the object storage to obtain the data block fingerprint and metadata carried in the header of each data block; wherein, the data block header read request corresponding to each data block is used by the object storage to read the header of the data block based on the header offset of each data block, and return the data block fingerprint and metadata carried in the header of the data block.

[0139] In an exemplary embodiment, the parallel reading module 603 is configured to fill the header offset of each data block into a byte range request header; obtain the corresponding data block header read request based on the byte range request header with the completed offset filling; initiate the data block header read request to the object storage to obtain the data block fingerprint and metadata carried in the header of each data block; wherein, the data block header read request is used by the object storage to read the header of each data block according to the header offset of each data block, and return the data block fingerprint and metadata carried in the header of each data block.

[0140] In an exemplary embodiment, the above-described apparatus further includes a writing module. This writing module is configured to, in response to a data file write request, acquire multiple data blocks to be written; sequentially write the data to be written corresponding to each of the data blocks to be written into a writable fragment; set corresponding bit flags in a bit set according to the number of alignment blocks occupied by each of the data blocks to be written; store the writable fragments in object storage; when a target termination condition is met, obtain the file header data to be written based on the current bit set and the written data, and write the file header data to be written into a header fragment; store the header fragments in the object storage; merge the header fragments that have completed data writing and each writable fragment that has completed data writing to obtain the data file corresponding to the data file write request.

[0141] In an exemplary embodiment, the writing module is configured to sequentially write the data to be written corresponding to each of the data blocks to be written into a writable fragment according to the order of the data blocks to be written; before writing the data to be written corresponding to the current data block to be written into the current writable fragment, it is determined whether the fragment switching condition is met; wherein, the fragment switching condition includes: no transmission session is currently open, or, the sum of the size of the already written data of the current writable fragment and the size of the data to be written corresponding to the current data block to be written is greater than a target threshold; the transmission session is used to manage the header fragmentation. And each of the writable fragments; if yes, then complete the data writing to the current writable fragment, open the transmission session, create a writable fragment in the transmission session, write the data to be written corresponding to the current data block to be written to the created writable fragment, and set the corresponding bit flag in the bit set according to the number of alignment blocks occupied by the data to be written corresponding to the current data block to be written; if no, then write the data to be written corresponding to the current data block to be written to the current writable fragment, and set the corresponding bit flag in the bit set according to the number of alignment blocks occupied by the data to be written corresponding to the current data block to be written.

[0142] In an exemplary embodiment, the above-described apparatus further includes a splicing module. This splicing module is configured to, if the target termination condition is not currently met, acquire the data block data of the current data block to be written, obtain the data block header and data block alignment padding data of the current data block to be written based on the current data block data, and splice the data block header, the data block data, and the data block alignment padding data of the current data block to be written to obtain the data to be written corresponding to the current data block to be written.

[0143] Each module in the aforementioned fingerprint database reconstruction device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the processor of a computer device in hardware form or independent of it, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0144] In one exemplary embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 7 As shown, the computer device includes a processor, memory, input / output interfaces, a communication interface, a display unit, and an input device. The processor, memory, and input / output interfaces are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The input / output interfaces are used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, Near Field Communication (NFC), or other technologies. When executed by the processor, the computer program implements a fingerprint database reconstruction method. The display unit is used to form a visually visible image and can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input device of the computer device can be a touch layer covering the display screen, or buttons, trackballs, or touchpads set on the casing of the computer device, or external keyboards, touchpads, or mice, etc.

[0145] Those skilled in the art will understand that Figure 7 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0146] In one exemplary embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the fingerprint database reconstruction method described above.

[0147] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the fingerprint database reconstruction method described above.

[0148] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the fingerprint database reconstruction method described above.

[0149] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0150] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0151] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A method of fingerprint library reconstruction, characterized by, The method includes: In response to a data file write request, retrieve multiple data blocks to be written; The data to be written corresponding to each of the data blocks to be written is sequentially written to the writable fragment, and the corresponding bit flag is set in the bit set according to the number of alignment blocks occupied by each of the data blocks to be written; the writable fragment is stored in object storage; When the target termination condition is met, the file header data to be written is obtained based on the current bit set and the data already written, and the file header data to be written is written into the header fragment; the header fragment is stored in the object storage; The header fragments that have completed data writing and each writable fragment that has completed data writing are merged to obtain the data file corresponding to the data file write request; Read the header of the data file; Based on the set of bits carried in the header of the data file, obtain the header offset of each data block in the data file; Based on the header offset of each data block, the headers of each data block are read in parallel to obtain the data block fingerprint and metadata carried in the header of each data block. Insert the data block fingerprint and metadata corresponding to each data block into the fingerprint database.

2. The method of claim 1, wherein, The step of reading the headers of each data block in parallel based on the header offset of each data block to obtain the data block fingerprint and metadata carried in the header of each data block includes: Based on the header offset of each data block, obtain the data block header read request corresponding to each data block; Initiate parallel read requests for the headers of each data block to the object storage to obtain the data block fingerprint and metadata carried in the header of each data block; The data block header read request corresponding to the data block is used by the object storage to read the header of the data block according to the header offset of each data block, and return the data block fingerprint and metadata carried in the header of the data block.

3. The method of claim 1, wherein, The step of reading the headers of each data block in parallel based on the header offset of each data block to obtain the data block fingerprint and metadata carried in the header of each data block includes: Fill the header offset of each of the data blocks into a one-byte range request header; Based on the byte range request header for the completed offset filling, the corresponding data block header read request is obtained; Initiate a data block header read request to the object storage to obtain the data block fingerprint and metadata carried in the header of each data block; The data block header read request is used by the object storage to read the header of each data block according to the header offset of each data block, and return the data block fingerprint and metadata carried in the header of each data block.

4. The method of claim 1, wherein, The step of sequentially writing the data to be written to each of the data blocks to be written into the writable fragment, and setting the corresponding bit flag in the bit set according to the number of alignment blocks occupied by each of the data blocks to be written, includes: According to the order of the data blocks to be written, the data to be written corresponding to each data block to be written is written to the writable fragment in sequence; Before writing the data to be written to the current writable fragment corresponding to the current data block to be written, it is determined whether the fragment switching conditions are met. The fragment switching conditions include: no transmission session is currently open, or the sum of the size of the written data of the current writable fragment and the size of the data to be written corresponding to the current data block to be written is greater than a target threshold. The transmission session is used to interact with the object storage. If so, complete the data writing for the current writable fragment, create a writable fragment in the transmission session, open the transmission session, write the data to be written corresponding to the current data block to be written to the created writable fragment, and set the corresponding bit flag in the bit set according to the number of alignment blocks occupied by the data to be written corresponding to the current data block to be written. If not, the data to be written corresponding to the current data block to be written is written to the current writable fragment, and the corresponding bit flag is set in the bit set according to the number of alignment blocks occupied by the data to be written corresponding to the current data block to be written.

5. The method according to claim 1, characterized in that, Before sequentially writing the data to be written to each of the data blocks to be written to the writable fragment, the method further includes: If the target termination condition is not met, the data block data of the current data block to be written is obtained. Based on the data block data of the current data block to be written, the data block header and data block alignment padding data of the current data block to be written are obtained. The data block header, data block data and data block alignment padding data of the current data block to be written are concatenated to obtain the data to be written corresponding to the current data block to be written.

6. A fingerprint library reconstruction apparatus characterized by comprising: The device includes: The writing module is used to respond to a data file write request by acquiring multiple data blocks to be written; sequentially writing the data to be written corresponding to each data block to a writable fragment, and setting corresponding bit flags in the bit set according to the number of alignment blocks occupied by each data block to be written; the writable fragments are stored in object storage; when the target end condition is met, the file header data to be written is obtained according to the current bit set and the written data, and the file header data to be written is written to the header fragment; the header fragments are stored in the object storage; the header fragments that have completed data writing and each writable fragment that has completed data writing are merged to obtain the data file corresponding to the data file write request; The read module is used to read the header of the data file; The acquisition module is used to acquire the header offset of each data block in the data file based on the bit set carried in the header of the data file. The parallel reading module is used to read the header of each data block in parallel according to the header offset of each data block, so as to obtain the data block fingerprint and metadata carried in the header of each data block. An insertion module is used to insert the data block fingerprint and the metadata corresponding to each data block into the fingerprint database. 7.A computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the computer device is configured to perform the method according to any one of claims 1-6 when the computer program is executed by the processor. When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 5.

8. A computer-readable storage medium having stored thereon a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 5.

9. A computer program product comprising a computer program, characterized in that, The computer program, which is executed by a processor, implements the steps of the method according to any one of claims 1 to 5.