Backup chain data processing method

Abstract

The invention provides a backup chain data processing method. According to the method, the metadata of the backup chain is analyzed in parallel, the data range information is extracted, and the global sparse mapping table is constructed in combination with the scanning line algorithm, so that traditional dynamic chain backtracking can be converted into direct query; in a data access stage, binary search is realized based on a mapping table to position a physical address or a hole region, and redundant data interaction operation is avoided, so that random access performance and large-scale recovery efficiency are remarkably improved; and the query performance and the backup chain length are decoupled through preprocessing, so that the expandability of the system is ensured. And finally, high-performance and low-delay backup chain data access is realized, and the utilization rate of storage resources is optimized.

Description

Technical Field

[0001] This application relates to the field of backup chain data management, and in particular to a backup chain data processing method. Background Technology

[0002] In the field of data backup and recovery technology, a strategy of "full backup + multiple incremental / differential backups" can be adopted to balance storage costs and backup efficiency. This strategy forms a backup chain, which includes a base full backup file and a series of subsequent incremental or differential backup files. These files collectively record the complete state of a virtual disk at a specific point in time, with data blocks in later files overwriting data blocks at the same logical addresses in earlier files. During data recovery, whether restoring the entire virtual disk or a portion of files, it is necessary to accurately read data from this logically unified but physically distributed backup chain. The core challenge lies in how to efficiently parse and process the data overwrite relationships between the backup files to quickly and accurately reconstruct the final virtual disk data view and provide high-performance read services.

[0003] In practice, the main challenge in handling backup chains lies in efficiently mapping the logical addresses of virtual disks to specific data blocks scattered across multiple physical backup files in real time. For example, one feasible implementation uses dynamic chained backtracking for data recovery. Upon receiving a read request, without preprocessing, it directly starts from the latest file in the backup chain and searches back through the older files. For each requested logical data block, it checks if the current file contains it; if so, it reads the data; if not, it continues searching in the previous file in the chain until the data is found or the entire backup chain has been traversed. This dynamic, on-demand parsing model results in high data access latency, poor system scalability, and performance degradation as the data volume increases, leading to inefficient data access and poor system performance during data recovery. Therefore, a backup chain data processing method is needed to address these issues. Summary of the Invention

[0004] The purpose of this application is to at least address one of the aforementioned technical deficiencies, particularly the technical deficiencies of low data access efficiency and poor system performance during the data recovery process in the prior art.

[0005] In a first aspect, this application provides a backup chain data processing method, the method comprising:

[0006] Preprocessing stage: Parallel parsing of metadata for all files in the backup chain, extracting data range information for the backup files corresponding to each piece of metadata;

[0007] The data range information includes the address range and physical address corresponding to the logical address, and the physical location is used to indicate the actual storage location of the backup file.

[0008] The data range information is processed based on the scanline algorithm to determine the mapping relationship between the logical address and the physical address;

[0009] Data access phase:

[0010] For a read request, locate the mapping entry corresponding to the request address in the sparse mapping table, and read the target data based on the physical address corresponding to the mapping entry;

[0011] For addressing requests, the location of the data area or the location of the void area are located based on the sparse mapping table.

[0012] As an optional implementation, the step of processing the data range information based on the scanline algorithm to determine the mapping relationship between the logical address and the physical address includes:

[0013] The data range information is converted into start event points and end event points, and all start event points and end event points are globally sorted by logical address;

[0014] Using scan lines, each start event point and each end event point are scanned sequentially according to the logical address order;

[0015] Based on the index value corresponding to each of the data range information, determine the valid data source in the address range corresponding to each of the logical addresses;

[0016] Based on each of the valid data sources, mapping entries are generated, and a sparse mapping table from the logical address to the physical address is constructed using each of the mapping entries.

[0017] As an optional implementation, the method further includes:

[0018] If any of the mapping entries has both a logical address and a physical address that are consecutive with the adjacent mapping entries, then they are merged to generate a new merged mapping entry.

[0019] As an optional implementation, the step of locating the mapping entry corresponding to the request address in the sparse mapping table for a read request, and reading the target data based on the physical address corresponding to the mapping entry, includes:

[0020] In the sparse mapping table, perform a binary search to find the mapping entry corresponding to the request address;

[0021] If the corresponding mapping entry is matched, the target data is read directly based on the physical file path and offset recorded in the mapping entry.

[0022] If the corresponding mapping entry is not hit, it is determined that a hole region has been hit, and zero-value data is filled into the return buffer as the target data;

[0023] Furthermore, for requests spanning multiple mapping entries or the void regions, data reading operations are performed in segments and cyclically.

[0024] As an optional implementation, the step of locating the data area or hole location based on the sparse mapping table for an addressing request includes:

[0025] For data area location, based on the sparse mapping table, query the first mapping entry whose starting address is greater than or equal to the requested address from the starting address, and return the starting address corresponding to the mapping entry;

[0026] For locating void areas, the system queries the sparse mapping table for mapping entries that contain or immediately follow the requested address and returns the end address corresponding to that mapping entry. If no corresponding mapping entry is found, the system returns the disk end address.

[0027] As an optional implementation, the parallel parsing of metadata for all files in the backup chain, and the extraction of data range information for the backup files corresponding to each piece of metadata, includes:

[0028] Create independent data exchange channels for each backup file in the backup chain;

[0029] Concurrently execute metadata parsing tasks to extract the data range information of the backup files corresponding to each metadata;

[0030] In addition, during the extraction process, the data range information of adjacent logical addresses and physical addresses is merged, specifically including:

[0031] If the logical end address of the first data range is continuous with the logical start address of the second data range, and the physical end address of the first data range is continuous with the physical start address of the second data range, then the first data range and the second data range are merged into a single target data range.

[0032] Secondly, this application provides a backup chain data processing apparatus, comprising:

[0033] The processing module is used to parse the metadata of all files in the backup chain in parallel during the preprocessing stage and extract the data range information of the backup file corresponding to each metadata.

[0034] The data range information includes the address range and physical address corresponding to the logical address, and the physical location is used to indicate the actual storage location of the backup file.

[0035] The data range information is processed based on the scanline algorithm to determine the mapping relationship between the logical address and the physical address;

[0036] The processing module is further configured to, during the data access phase, locate the mapping entry corresponding to the request address in the sparse mapping table for a read request, and read the target data based on the physical address corresponding to the mapping entry; and for an addressing request, locate the data area location or the void area location based on the sparse mapping table.

[0037] As an optional implementation, the processing module processes the data range information based on the scanline algorithm to determine the specific method of the mapping relationship between the logical address and the physical address, including:

[0038] The data range information is converted into start event points and end event points, and all start event points and end event points are globally sorted by logical address;

[0039] Using scan lines, each start event point and each end event point are scanned sequentially according to the logical address order;

[0040] Based on the index value corresponding to each of the data range information, determine the valid data source in the address range corresponding to each of the logical addresses;

[0041] Based on each of the valid data sources, mapping entries are generated, and a sparse mapping table from the logical address to the physical address is constructed using each of the mapping entries.

[0042] Thirdly, this application provides a computer device including one or more processors and a memory storing computer-readable instructions that, when executed by the one or more processors, perform the steps of the method described in the first aspect.

[0043] Fourthly, this application provides a storage medium storing computer-readable instructions that, when executed by one or more processors, cause the one or more processors to perform the steps of the method described in the first aspect.

[0044] As can be seen from the above technical solutions, the embodiments of this application have the following advantages:

[0045] Based on any of the above embodiments, the method provided in this application, in the preprocessing stage, extracts data range information containing logical address ranges and physical locations by parsing the metadata of all files in the backup chain in parallel, providing complete input for global mapping construction; based on the scanline algorithm, it converts the data range into event points and sorts them globally, and generates mapping entries by dynamically tracking the active range and selecting the latest data version, which can efficiently parse the overlay relationship between backup files and generate a sparse mapping table from logical address to physical address; in the data access stage, for read requests, it directly locates physical data or fills holes by searching the mapping table in binary search, and for addressing requests, it quickly locates the boundary of the data area or hole area by searching the mapping table, thereby avoiding the chain-like search of traditional dynamic backtracking. This implementation transforms runtime parsing into static mapping table lookup through one-time preprocessing, optimizing the random access complexity from the linear dependency of traditional methods to logarithmic level; it identifies hole areas and plans continuous data operations through a global view, reducing fragmented operations and improving the throughput of large-scale recovery; and it ensures initialization speed and system scalability through the efficiency of parallel parsing and the scanline algorithm. Finally, it achieves high-performance access to backup chain data, solving the core problems of high latency, low efficiency, and poor scalability in traditional technologies. Attached Figure Description

[0046] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0047] Figure 1 A schematic flowchart illustrating a backup chain data processing method provided in one embodiment of this application;

[0048] Figure 2 This is an internal structural diagram of a computer device provided in an embodiment of this application. Detailed Implementation

[0049] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0050] The following is a brief explanation of the important terms that may be involved:

[0051] An image chain is a sequence of files consisting of a full backup file and one or more subsequent incremental / differential backup files. This sequence together constitutes a complete virtual disk image at a specific point in time.

[0052] The sweep-line algorithm is an algorithmic pattern in computational geometry. It imagines a straight line (sweep-line) sweeping across the entire plane (in this scheme, a one-dimensional logical address space), and processes the data when the sweep-line encounters specific event points (such as the start or end of an interval), thereby solving problems related to geometric intervals such as coverage and intersection.

[0053] In this application, a data range refers to a logically contiguous area of ​​data blocks. It is typically defined by a logical starting block number, the number of blocks, and its storage offset within a physical backup file.

[0054] A hole is a region in the logical address space of a virtual disk that has not been allocated data and does not contain any valid information. In a file system, this typically corresponds to disk space that has never been written to.

[0055] Positional Read, or pread, is a system call interface for file reading operations. It allows a program to read a specified amount of data from a specified offset in a file without changing the current position of the file pointer, making it ideal for concurrent reading in a multi-threaded environment.

[0056] Reposition read / write file offset, lseek, is a system call interface used to move the file read / write pointer position. In this application, it specifically refers to its extended functions SEEK_DATA and SEEK_HOLE.

[0057] SEEK_DATA: Starting from the specified offset, find and locate the beginning of the next region containing data.

[0058] SEEK_HOLE: Starting from the specified offset, find and locate the starting position of the next hole region.

[0059] I / O (Input / Output) refers to the data exchange process between a computer and external devices (such as disks and networks).

[0060] O(N log N) is a notation for the time complexity of an algorithm. O(N log N) indicates that the execution time of an algorithm is proportional to the logarithm of the input size N multiplied by N. This is generally considered a very efficient time complexity for sorting and processing algorithms.

[0061] O(log M) is a notation for the time complexity of an algorithm. O(log M) indicates that the execution time of the algorithm is proportional to the logarithm of the input size M. This is typically based on the complexity of efficient search algorithms such as binary search, meaning that even with massive datasets, the search speed is extremely fast.

[0062] This application specifically relates to the field of data backup and recovery, and in particular to efficient data access and recovery for backup chains consisting of full and incremental / differential backup files.

[0063] In data protection practices, to balance storage costs and backup efficiency, a common strategy is "full backup + multiple incremental / differential backups." This strategy forms a backup chain, which includes a base full backup file and a series of subsequent incremental or differential backup files. These files collectively record the complete state of a virtual disk at a specific point in time, with data blocks in later files overwriting data blocks at the same logical addresses in earlier files.

[0064] When performing data recovery, whether restoring the entire virtual disk or only a portion of files, it is necessary to accurately read data from this logically unified but physically distributed backup chain. The core challenge lies in how to efficiently parse and process the data overwrite relationships between the backup files in order to quickly and accurately reconstruct the final virtual disk data view and provide high-performance read services to the outside world.

[0065] One feasible implementation is dynamic chained backtracking. Upon receiving a read request, without preprocessing, it directly starts from the latest file in the backup chain and searches backtracks to older files level by level. For each requested logical data block, it checks whether the current file contains the data block; if it does, the data is read; if it does not, it continues searching in the previous file in the chain until the data is found or the entire backup chain has been traversed.

[0066] Therefore, the following limitations may exist: Firstly, read performance is poor and latency is high, and the dynamic backtracking mechanism leads to excessively long I / O paths. Reading a data block located early in the backup chain requires multiple invalid metadata queries and file location operations. This complex overwrite relationship processing logic makes the system unable to support low-latency random access. For large-scale sequential reads (such as full disk recovery), this block-by-block search method will evolve into a series of independent, slow search operations, severely impacting overall recovery efficiency. Secondly, redundant operations exist in the recovery process. Different incremental files in the backup chain may contain multiple updates to the same logical address, forming data redundancy. When searching for the final valid data, the dynamic backtracking technology needs to check these intermediate versions one by one, increasing unnecessary metadata I / O. More importantly, due to the lack of a global view of the final disk layout, it cannot efficiently plan the recovery process, such as identifying and skipping large unallocated holes, which may lead to the reading and writing of a large amount of useless zero data during full disk recovery, causing data amplification and prolonged recovery time.

[0067] Therefore, the core technical challenge for these feasible implementations in handling backup chains lies in how to efficiently map the logical addresses of virtual disks to specific data blocks scattered across multiple physical backup files in real time. Furthermore, this dynamic, on-demand resolution model leads to the following aspects that urgently need to be addressed:

[0068] The high-latency random access performance is the most significant limitation. When reading single or small amounts of non-contiguous data (e.g., recovering a single file or performing a database query), the dynamic chained backtracking method requires a serial metadata lookup starting from the latest backup file. This lookup process involves long and unpredictable I / O paths, especially when the required data is located in an earlier file in the backup chain. This results in significant redundant I / O and computational overhead, leading to extremely high read latency and making it unsuitable for applications requiring low latency.

[0069] Large-scale recovery suffers from low I / O efficiency. During large-scale sequential reads such as full disk recovery, while current technologies only recover valid data, the recovery process is inefficient. Due to the lack of global knowledge of the final disk layout, an optimized overall read plan cannot be generated. A logically contiguous large block read request may be broken down into a series of physically fragmented small block read operations on different backup files. This **I / O fragmentation** leads to a large number of random seeks in the storage backend, failing to leverage the performance advantages of sequential reads, thus severely reducing overall recovery throughput and efficiency.

[0070] The system suffers from poor scalability, and its performance deteriorates as the backup chain grows. The performance of dynamic backtracking is strongly correlated with the length of the backup chain. As the number of incremental backups increases, the backup chain grows longer, and the average search path also increases, leading to a linear decline in read performance. This makes the performance of this technology increasingly unacceptable in scenarios requiring long-term data retention and involving very long backup chains.

[0071] Therefore, the main purpose of this application is to realize how to construct a final mapping view from logical address to unique physical location through efficient one-time preprocessing, thereby transforming complex and dynamic chain lookup into simple and deterministic direct query, so as to achieve high-performance, low-latency random access and high-throughput sequential reading of backup chain data.

[0072] Therefore, the main creative point is that a backup chain preprocessing method based on parallel metadata parsing and the sweep-line algorithm is proposed. Before the read operation begins, this method integrates the data block information scattered in multiple backup files into a globally unique and optimized mapping table from logical blocks to physical storage locations through a one-time, efficient computation.

[0073] Specifically, this scheme first reads and parses the metadata of all files in the backup chain in parallel, extracting the data block ranges (Ranges) contained in each file. Then, using the scanline algorithm, it efficiently processes the coverage relationships of these data ranges in O(N log N) time complexity (where N is the total number of data block ranges in all files). This algorithm can accurately calculate which backup file (i.e., the latest file) should ultimately provide each data block in the logical address space of the entire virtual disk. Finally, a concise and ordered mapping table is generated, where each entry directly points to a unique and valid data source. This avoids the inefficient and repetitive dynamic backtracking lookup process. Any subsequent data read request, whether random or sequential, can be directly located to the physical file and offset of the data by performing a fast binary search on this mapping table in O(log M) time complexity (where M is the total number of data ranges after merging), thus achieving near-optimal read performance.

[0074] In summary, the technical concept of this application lies in the following: The method provided in this application, during the preprocessing stage, extracts data range information containing logical address intervals and physical locations by parsing the metadata of all files in the backup chain in parallel, providing complete input for global mapping construction. Based on the scanline algorithm, the data range is converted into event points and globally sorted. By dynamically tracking the active range and selecting the latest data version to generate mapping entries, the overlay relationship between backup files can be efficiently resolved, generating a sparse mapping table from logical addresses to physical addresses. During the data access stage, for read requests, the physical data or gaps are directly located using a binary search of the mapping table; for addressing requests, the boundaries of data areas or gap areas are quickly located using the mapping table, thus avoiding the chain-like search of traditional dynamic backtracking. This implementation transforms runtime parsing into static mapping table lookup through one-time preprocessing, optimizing the random access complexity from the linear dependency of traditional methods to logarithmic levels. It identifies gap areas and plans continuous data operations through a global view, reducing fragmented operations and improving large-scale recovery throughput. The efficiency of parallel parsing and the scanline algorithm ensures initialization speed and system scalability. Ultimately, this enables high-performance access to backup chain data, solving the core problems of high latency, low efficiency, and poor scalability in traditional technologies.

[0075] The method provided in this application will be described in detail below based on corresponding implementation methods in some practical application scenarios. This application constructs a global data mapping through a preprocessing stage, thereby achieving high performance in the subsequent data reading and addressing stages.

[0076] To implement the method provided in any of the embodiments, this application provides a corresponding system architecture.

[0077] The system mainly consists of the following modules:

[0078] The central control module (Image_Chain_Reader) is responsible for the initialization, coordination, and task assignment of the entire system. It receives backup chain information from external sources, manages the lifecycle of other modules, and ultimately provides a unified data access interface, including data reading (pread) and data addressing (lseek).

[0079] Concurrent I / O Module (Client_Pool_Manager): Provides and manages a set of independent I / O channels (client connections) for each physical backup file in the backup chain. This module supports high-concurrency parallel access to multiple backup files through connection pooling technology, and effectively reduces the overhead of establishing and tearing down I / O operations.

[0080] The metadata parsing module (Metadata_Parser_Callback & Image_Parser) is responsible for reading and parsing the metadata of a single backup file. This module can identify the data layout description within the file and extract the "data range" information. A "data range" defines a logically contiguous block of data and its storage location within the physical file.

[0081] Mapping Module (run_sweep_line): The core processing unit of this invention. It receives "data range" information from all backup files and calculates the final, unique data source for each logical data block on the virtual disk using a deterministic overlay relationship processing algorithm, ultimately generating a global, optimized "logical-physical address mapping table".

[0082] Data access module (pread, lseek): Responsible for executing external data access requests.

[0083] The data reading submodule (pread) uses a "logical-physical address mapping table" to directly convert a logical read request for a virtual disk into one or more physical read operations for a specific physical backup file, or to identify empty areas and fill them with zero data.

[0084] The data addressing submodule (lseek) utilizes a "logical-physical address mapping table" to enable efficient querying of virtual disk data layout, especially supporting the search for the starting position of the next data area (SEEK_DATA) or the next hole area (SEEK_HOLE) from a specified location.

[0085] Figure 1 This is a flowchart illustrating a backup chain data processing method provided in one embodiment of this application, as shown below. Figure 1 As shown, this application provides a backup chain data processing method, the method comprising:

[0086] S101, Preprocessing stage: Parallel parsing of metadata of all files in the backup chain, extracting data range information of the backup file corresponding to each metadata;

[0087] The data range information includes the address range and physical address corresponding to the logical address, and the physical location is used to indicate the actual storage location of the backup file.

[0088] The data range information is processed based on the scanline algorithm to determine the mapping relationship between the logical address and the physical address;

[0089] S102, Data Access Stage: For a read request, locate the mapping entry corresponding to the request address in the sparse mapping table, and read the target data based on the physical address corresponding to the mapping entry; for an addressing request, locate the data area or void area based on the sparse mapping table.

[0090] This implementation method efficiently constructs the basic framework of the dataset by parsing the metadata of all files in the backup chain in parallel, extracting data range information including logical address intervals and physical locations. It processes the data range information using a scanline algorithm to determine the mapping relationship between logical and physical addresses, integrating scattered backup file data into a globally ordered mapping view. During the data access phase, read requests directly locate the physical address to read data using a sparse mapping table, while addressing requests quickly distinguish between data areas and empty areas using the mapping table, thus avoiding redundant calculations in dynamic backtracking. By generating a global mapping table through one-time preprocessing, traditional chained lookups are transformed into deterministic queries, significantly reducing data access latency, improving the throughput efficiency of large-scale recovery, and ensuring decoupling of system performance from backup chain length.

[0091] As an optional implementation, the step of processing the data range information based on the scanline algorithm to determine the mapping relationship between the logical address and the physical address includes:

[0092] The data range information is converted into start event points and end event points, and all start event points and end event points are globally sorted by logical address;

[0093] Using scan lines, each start event point and each end event point are scanned sequentially according to the logical address order;

[0094] Based on the index value corresponding to each of the data range information, determine the valid data source in the address range corresponding to each of the logical addresses;

[0095] Based on each of the valid data sources, mapping entries are generated, and a sparse mapping table from the logical address to the physical address is constructed using each of the mapping entries.

[0096] This implementation converts data range information into start and end event points and sorts them globally by logical address, enabling unified processing of scattered data intervals. By scanning event points sequentially with scan lines and dynamically tracking active ranges, and combining this with data range index values ​​to determine valid data sources, it can accurately identify the latest version of data. Finally, it generates a sparse mapping table from logical address to physical address, thereby efficiently resolving the overlay relationship between backup files, avoiding redundant operations of dynamic backtracking at runtime, and improving the accuracy and efficiency of mapping table construction.

[0097] As an optional implementation, the method further includes:

[0098] If any of the mapping entries has both a logical address and a physical address that are consecutive with the adjacent mapping entries, then they are merged to generate a new merged mapping entry.

[0099] This implementation reduces the number of redundant entries in the sparse mapping table by merging adjacent mapping entries with both logical and physical addresses, thereby optimizing the storage structure and improving query efficiency. The merging operation reduces the size of the mapping table, alleviating the computational load in subsequent data access processes, shortening the binary search time, and further improving the response speed of random access and addressing operations.

[0100] As an optional implementation, the step of locating the mapping entry corresponding to the request address in the sparse mapping table for a read request, and reading the target data based on the physical address corresponding to the mapping entry, includes:

[0101] In the sparse mapping table, perform a binary search to find the mapping entry corresponding to the request address;

[0102] If the corresponding mapping entry is matched, the target data is read directly based on the physical file path and offset recorded in the mapping entry.

[0103] If the corresponding mapping entry is not hit, it is determined that a hole region has been hit, and zero-value data is filled into the return buffer as the target data;

[0104] Furthermore, for requests spanning multiple mapping entries or the void regions, data reading operations are performed in segments and cyclically.

[0105] This implementation uses binary search to quickly locate the entry corresponding to the requested address in the sparse mapping table, enabling deterministic queries. When a match is found, the physical file is read directly, and when a match is missed, zero-value data is filled in, which can efficiently handle empty areas. For requests spanning multiple entries, the read operation is performed in segments, thereby avoiding fragmented data interaction of traditional chained backtracking, improving the throughput efficiency of random reads, and reducing the latency caused by invalid data access.

[0106] As an optional implementation, the step of locating the data area or hole location based on the sparse mapping table for an addressing request includes:

[0107] For data area location, based on the sparse mapping table, query the first mapping entry whose starting address is greater than or equal to the requested address from the starting address, and return the starting address corresponding to the mapping entry;

[0108] For locating void areas, the system queries the sparse mapping table for mapping entries that contain or immediately follow the requested address and returns the end address corresponding to that mapping entry. If no corresponding mapping entry is found, the system returns the disk end address.

[0109] This implementation uses a sparse mapping table to quickly locate data areas and hole areas: for data areas, the query returns the starting address of the first mapping entry that is greater than or equal to the requested address; for hole areas, the query returns the ending address of the mapping entry or the disk end address, which can directly skip invalid areas. The global view of the mapping table accurately locates the boundaries, avoiding redundant operations of scanning the backup chain step by step, significantly improving addressing efficiency, especially reducing hole processing overhead during full disk recovery.

[0110] As an optional implementation, the parallel parsing of metadata for all files in the backup chain, and the extraction of data range information for the backup files corresponding to each piece of metadata, includes:

[0111] Create independent data exchange channels for each backup file in the backup chain;

[0112] Concurrently execute metadata parsing tasks to extract the data range information of the backup files corresponding to each metadata;

[0113] In addition, during the extraction process, the data range information of adjacent logical addresses and physical addresses is merged, specifically including:

[0114] If the logical end address of the first data range is continuous with the logical start address of the second data range, and the physical end address of the first data range is continuous with the physical start address of the second data range, then the first data range and the second data range are merged into a single target data range.

[0115] This implementation method creates an independent data interaction channel for backup files and concurrently parses metadata, enabling parallel extraction of data range information and accelerating the preprocessing stage. During the extraction process, merging data ranges where logical and physical addresses are adjacent reduces the number of redundant entries and optimizes the input data scale of the scanline algorithm. By initially merging, the total number of event points is reduced, shortening the time consumed by global sorting and dynamic coverage processing, thereby improving the overall efficiency of mapping table construction.

[0116] The various embodiments of this application can be implemented in combination. As mentioned above, the workflow of this application mainly includes two stages:

[0117] Mapping construction phase and data access phase.

[0118] The mapping construction phase is completed in one go before the first access to the data, and the specific steps are as follows:

[0119] Parallel parsing: The central control module starts and allocates an independent I / O channel for each physical backup file in the backup chain through the concurrent I / O module, and initiates a metadata parsing task. The parsing of all files is performed in parallel, which greatly reduces processing time.

[0120] Data Range Extraction: Each metadata parsing task independently scans its corresponding backup file. It identifies and extracts all data range descriptions from the file, each description containing the logical starting address, length, and physical offset of the range within the current file. Simultaneously, during the extraction process, logically and physically adjacent smaller ranges are initially merged into a larger range to reduce the complexity of subsequent processing.

[0121] Coverage relationship resolution (based on scanline algorithm):

[0122] a. Event Point Transformation: The mapping module collects all data ranges extracted from all backup files. For each data range [Start, End), it generates two key events: a "range start" event point, marked at logical address Start; and a "range end" event point, marked at logical address End. Each event point is accompanied by information about its source file (specifically, the file's index in the backup chain).

[0123] b. Global Sorting: Sort all event points globally according to their logical addresses in ascending order. If addresses are the same, the "End of Range" event is processed before the "Start of Range" event to ensure correct boundary handling.

[0124] c. Scanning and merging: The system simulates a scan line, starting from logical address 0 and scanning each event point in sequence.

[0125] i. Within the address range between any two adjacent event points, the source of all active data ranges (i.e., the ranges where the scan line has swept through its "start" event but not its "end" event) is fixed.

[0126] ii. Among these active ranges, the system selects the range with the largest index value (i.e., the most recent) as the unique valid data source for that address range, based on the order of the files' age.

[0127] iii. Based on this unique and valid data source, the system generates a new mapping entry, recording the direct mapping relationship between the logical address segment and its physical location.

[0128] iv. If a newly generated mapping entry is logically and physically contiguous with the previous entry, they are merged to further optimize the mapping table.

[0129] d. Mapping Generation: After the scan line traverses all event points, a complete, conflict-free, merged and optimized logical-physical address mapping table is constructed. This table is an ordered sparse mapping describing all allocated data regions of the virtual disk.

[0130] When the system receives a data access request, it performs the following steps:

[0131] For data read (pread) requests:

[0132] Quick lookup: In the "Logical-Physical Address Mapping Table", the mapping entry containing the starting address of the request can be quickly located using an efficient search algorithm (such as binary search).

[0133] Direct physical read: If the requested address falls within a valid mapping entry, the system initiates a read operation on the physical file directly through the concurrent I / O module based on the physical file information and offset recorded in the entry.

[0134] Hole handling: If the requested address is not in any entry in the mapping table, the system determines that the region is a hole and directly fills it with zeros in the return buffer.

[0135] Segmentation and Looping: For read requests that span multiple mapping entries or holes, the system breaks them down and processes them in a loop.

[0136] For data seeking (lseek) requests (taking SEEK_DATA and SEEK_HOLE as examples):

[0137] SEEK_DATA (Search for the next data area):

[0138] Given a starting logical address, the data addressing submodule performs a lookup in an ordered logical-physical address mapping table.

[0139] It will find the first mapping entry whose starting address is greater than or equal to the given address.

[0140] The starting address of this entry is the starting position of the next data area, and the system directly returns this address.

[0141] SEEK_HOLE (Find the next hole):

[0142] Given a starting logical address, the data addressing submodule finds the mapping entry that contains that address or immediately following it.

[0143] The end address of this entry (i.e., the start address + length) is the starting position of the next empty region.

[0144] The system returns the address directly. If it is the last data area, it returns the address of the end of the disk.

[0145] Since the mapping table is pre-computed and sorted, these addressing operations can also be completed using efficient lookup algorithms with extremely low time complexity.

[0146] Therefore, this application effectively overcomes the shortcomings of the prior art by employing a series of technical means such as parallel metadata parsing and scanning line algorithm to construct a global mapping table, and achieves the following effects:

[0147] It achieves high-performance, low-latency random data access. Through one-time preprocessing, the complex and time-consuming dynamic chained backtracking lookup in traditional techniques is transformed into a simple, deterministic table lookup operation with a time complexity of O(log M). Any random read request can be quickly located to its unique physical data source, completely eliminating the high latency caused by the uncertainty of backup chain length and data location. This results in an order-of-magnitude improvement in the efficiency of restoring a single file from the backup chain or performing fine-grained database-level recovery.

[0148] This solution significantly improves the throughput and efficiency of large-scale data recovery. Thanks to a pre-built global logical-physical address mapping table, it enables efficient hole skipping for large-scale recovery (such as full disk recovery). Through the lseek interface (supporting SEEK_DATA operations), the recovery program can quickly jump directly from one valid data area to the next in the logical view of the virtual disk, completely skipping large, unallocated holes in between. This avoids invalid I / O checks or writing zero data to hole areas, greatly reducing the total amount of data to be processed and significantly shortening the overall recovery time. Simultaneously, it optimizes the I / O pattern. With a thorough understanding of the global data layout, the system can plan logically consecutive large-block read requests into physically consecutive large-block I / O operations, effectively avoiding I / O fragmentation. This ability to aggregate scattered random reads into sequential reads fully utilizes the performance of the underlying storage device, resulting in extremely high recovery throughput.

[0149] This approach ensures stable and predictable system performance with excellent scalability. After the mapping table is constructed, the system performance is independent of the backup chain length. Whether the backup chain contains 10 or 100 incremental files, subsequent data reading and addressing performance remains consistently high. This characteristic solves the problem of linear performance degradation with the growth of the backup chain in existing technologies, providing strong technical support for scenarios requiring long-term data retention and managing complex backup chains. The mapping construction process itself employs parallel design and an efficient O(N log N) algorithm, ensuring rapid completion even during the initialization phase.

[0150] This application also provides a backup chain data processing device, including:

[0151] The processing module is used to parse the metadata of all files in the backup chain in parallel during the preprocessing stage and extract the data range information of the backup file corresponding to each metadata.

[0152] The data range information includes the address range and physical address corresponding to the logical address, and the physical location is used to indicate the actual storage location of the backup file.

[0153] The data range information is processed based on the scanline algorithm to determine the mapping relationship between the logical address and the physical address;

[0154] The processing module is further configured to, during the data access phase, locate the mapping entry corresponding to the request address in the sparse mapping table for a read request, and read the target data based on the physical address corresponding to the mapping entry; and for an addressing request, locate the data area location or the void area location based on the sparse mapping table.

[0155] This implementation method efficiently constructs the basic framework of the dataset by parsing the metadata of all files in the backup chain in parallel, extracting data range information including logical address intervals and physical locations. It processes the data range information using a scanline algorithm to determine the mapping relationship between logical and physical addresses, integrating scattered backup file data into a globally ordered mapping view. During the data access phase, read requests directly locate the physical address to read data using a sparse mapping table, while addressing requests quickly distinguish between data areas and empty areas using the mapping table, thus avoiding redundant calculations in dynamic backtracking. By generating a global mapping table through one-time preprocessing, traditional chained lookups are transformed into deterministic queries, significantly reducing data access latency, improving the throughput efficiency of large-scale recovery, and ensuring decoupling of system performance from backup chain length.

[0156] As an optional implementation, the processing module processes the data range information based on the scanline algorithm to determine the specific method of the mapping relationship between the logical address and the physical address, including:

[0157] The data range information is converted into start event points and end event points, and all start event points and end event points are globally sorted by logical address;

[0158] Using scan lines, each start event point and each end event point are scanned sequentially according to the logical address order;

[0159] Based on the index value corresponding to each of the data range information, determine the valid data source in the address range corresponding to each of the logical addresses;

[0160] Based on each of the valid data sources, mapping entries are generated, and a sparse mapping table from the logical address to the physical address is constructed using each of the mapping entries.

[0161] This implementation converts data range information into start and end event points and sorts them globally by logical address, enabling unified processing of scattered data intervals. By scanning event points sequentially with scan lines and dynamically tracking active ranges, and combining this with data range index values ​​to determine valid data sources, it can accurately identify the latest version of data. Finally, it generates a sparse mapping table from logical address to physical address, thereby efficiently resolving the overlay relationship between backup files, avoiding redundant operations of dynamic backtracking at runtime, and improving the accuracy and efficiency of mapping table construction.

[0162] As an optional implementation, the processing module is further configured to:

[0163] If any of the mapping entries has both a logical address and a physical address that are consecutive with the adjacent mapping entries, then they are merged to generate a new merged mapping entry.

[0164] This implementation reduces the number of redundant entries in the sparse mapping table by merging adjacent mapping entries with both logical and physical addresses, thereby optimizing the storage structure and improving query efficiency. The merging operation reduces the size of the mapping table, alleviating the computational load in subsequent data access processes, shortening the binary search time, and further improving the response speed of random access and addressing operations.

[0165] As an optional implementation, the step of locating the mapping entry corresponding to the request address in the sparse mapping table for a read request, and reading the target data based on the physical address corresponding to the mapping entry, includes:

[0166] In the sparse mapping table, perform a binary search to find the mapping entry corresponding to the request address;

[0167] If the corresponding mapping entry is matched, the target data is read directly based on the physical file path and offset recorded in the mapping entry.

[0168] If the corresponding mapping entry is not hit, it is determined that a hole region has been hit, and zero-value data is filled into the return buffer as the target data;

[0169] Furthermore, for requests spanning multiple mapping entries or the void regions, data reading operations are performed in segments and cyclically.

[0170] This implementation uses binary search to quickly locate the entry corresponding to the requested address in the sparse mapping table, enabling deterministic queries. When a match is found, the physical file is read directly, and when a match is missed, zero-value data is filled in, which can efficiently handle empty areas. For requests spanning multiple entries, the read operation is performed in segments, thereby avoiding fragmented data interaction of traditional chained backtracking, improving the throughput efficiency of random reads, and reducing the latency caused by invalid data access.

[0171] As an optional implementation, the specific method by which the processing module locates the data area or the hole area based on the sparse mapping table for an addressing request includes:

[0172] For data area location, based on the sparse mapping table, query the first mapping entry whose starting address is greater than or equal to the requested address from the starting address, and return the starting address corresponding to the mapping entry;

[0173] For locating void areas, the system queries the sparse mapping table for mapping entries that contain or immediately follow the requested address and returns the end address corresponding to that mapping entry. If no corresponding mapping entry is found, the system returns the disk end address.

[0174] This implementation uses a sparse mapping table to quickly locate data areas and hole areas: for data areas, the query returns the starting address of the first mapping entry that is greater than or equal to the requested address; for hole areas, the query returns the ending address of the mapping entry or the disk end address, which can directly skip invalid areas. The global view of the mapping table accurately locates the boundaries, avoiding redundant operations of scanning the backup chain step by step, significantly improving addressing efficiency, especially reducing hole processing overhead during full disk recovery.

[0175] As an optional implementation, the specific method by which the processing module parses the metadata of all files in the backup chain in parallel and extracts the data range information of the backup file corresponding to each metadata includes:

[0176] Create independent data exchange channels for each backup file in the backup chain;

[0177] Concurrently execute metadata parsing tasks to extract the data range information of the backup files corresponding to each metadata;

[0178] In addition, during the extraction process, the data range information of adjacent logical addresses and physical addresses is merged, specifically including:

[0179] If the logical end address of the first data range is continuous with the logical start address of the second data range, and the physical end address of the first data range is continuous with the physical start address of the second data range, then the first data range and the second data range are merged into a single target data range.

[0180] This implementation method creates an independent data interaction channel for backup files and concurrently parses metadata, enabling parallel extraction of data range information and accelerating the preprocessing stage. During the extraction process, merging data ranges where logical and physical addresses are adjacent reduces the number of redundant entries and optimizes the input data scale of the scanline algorithm. By initially merging, the total number of event points is reduced, shortening the time consumed by global sorting and dynamic coverage processing, thereby improving the overall efficiency of mapping table construction.

[0181] It should be noted that the division of the various modules in the above device is merely a logical functional division. In actual implementation, they can be fully or partially integrated into a single physical entity, or they can be physically separated. Furthermore, these modules can be implemented entirely in software via processing element calls; they can be fully implemented in hardware; or some modules can be implemented by processing element calls to software, while others are implemented in hardware. For example, a processing module can be a separate processing element, or it can be integrated into a chip within the device. Alternatively, it can be stored as program code in the device's memory, and its functions can be called and executed by a processing element. The implementation of other modules is similar. Moreover, these modules can be fully or partially integrated together, or they can be implemented independently. The processing element here can be an integrated circuit with signal processing capabilities. During implementation, each step of the above method or each of the above modules can be completed through integrated logic circuits in the hardware of the processor element or through software instructions.

[0182] Indicatively, such as Figure 2 As shown, Figure 2 This is a schematic diagram of the internal structure of a computer device 300 provided in an embodiment of this application. The computer device 300 can be provided as a server. (Refer to...) Figure 2 The computer device 300 includes a processing component 302, which further includes one or more processors, and memory resources represented by memory 301 for storing instructions, such as application programs, that can be executed by the processing component 302. The application programs stored in memory 301 may include one or more modules, each corresponding to a set of instructions. Furthermore, the processing component 302 is configured to execute instructions to perform the methods of any of the embodiments described above.

[0183] The computer device 300 may also include a power supply component 303 configured to perform power management of the computer device 300, a wired or wireless network interface 304 configured to connect the computer device 300 to a network, and an input / output (I / O) interface 305. The computer device 300 may operate on an operating system stored in memory 301, such as Windows Server™, Mac OS X™, Unix™, Linux™, Free BSD™, or similar.

[0184] Those skilled in the art will understand that Figure 2 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0185] This application provides a storage medium storing computer-readable instructions that, when executed by one or more processors, cause the one or more processors to perform the method provided in any embodiment.

[0186] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0187] The various embodiments in this specification are described in a progressive manner. Each embodiment focuses on the differences from other embodiments. The various embodiments can be combined as needed, and the same or similar parts can be referred to each other.

[0188] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A backup chain data processing method, characterized in that, The method includes: Preprocessing stage: Parallel parsing of metadata for all files in the backup chain, extracting data range information for the backup files corresponding to each piece of metadata; The data range information includes the address range and physical address corresponding to the logical address, and the physical location is used to indicate the actual storage location of the backup file. The data range information is processed based on the scanline algorithm to determine the mapping relationship between the logical address and the physical address; Data access phase: For a read request, locate the mapping entry corresponding to the request address in the sparse mapping table, and read the target data based on the physical address corresponding to the mapping entry; For addressing requests, the location of the data area or the location of the void area are located based on the sparse mapping table.

2. The method according to claim 1, characterized in that, The step of processing the data range information based on the scan-line algorithm to determine the mapping relationship between the logical address and the physical address includes: The data range information is converted into start event points and end event points, and all start event points and end event points are globally sorted by logical address; Using scan lines, each start event point and each end event point are scanned sequentially according to the logical address order; Based on the index value corresponding to each of the data range information, determine the valid data source in the address range corresponding to each of the logical addresses; Based on each of the valid data sources, mapping entries are generated, and a sparse mapping table from the logical address to the physical address is constructed using each of the mapping entries.

3. The method according to claim 2, characterized in that, The method further includes: If any of the mapping entries has both a logical address and a physical address that are consecutive with the adjacent mapping entries, then they are merged to generate a new merged mapping entry.

4. The method according to any one of claims 1-3, characterized in that, For a read request, locating the mapping entry corresponding to the request address in the sparse mapping table, and reading the target data based on the physical address corresponding to the mapping entry, includes: In the sparse mapping table, perform a binary search to find the mapping entry corresponding to the request address; If the corresponding mapping entry is matched, the target data is read directly based on the physical file path and offset recorded in the mapping entry. If the corresponding mapping entry is not hit, it is determined that a hole region has been hit, and zero-value data is filled into the return buffer as the target data; Furthermore, for requests spanning multiple mapping entries or the void regions, data reading operations are performed in segments and cyclically.

5. The method according to any one of claims 1-3, characterized in that, The step of locating the data region or hole region based on the sparse mapping table for an addressing request includes: For data area location, based on the sparse mapping table, query the first mapping entry whose starting address is greater than or equal to the requested address from the starting address, and return the starting address corresponding to the mapping entry; For locating void areas, the system queries the sparse mapping table for mapping entries that contain or immediately follow the requested address and returns the end address corresponding to that mapping entry. If no corresponding mapping entry is found, the system returns the disk end address.

6. The method according to claim 1, characterized in that, The parallel parsing of metadata for all files in the backup chain extracts the data range information of the backup file corresponding to each piece of metadata, including: Create independent data exchange channels for each backup file in the backup chain; Concurrently execute metadata parsing tasks to extract the data range information of the backup files corresponding to each metadata; In addition, during the extraction process, the data range information of adjacent logical addresses and physical addresses is merged, specifically including: If the logical end address of the first data range is continuous with the logical start address of the second data range, and the physical end address of the first data range is continuous with the physical start address of the second data range, then the first data range and the second data range are merged into a single target data range.

7. A backup chain data processing device, characterized in that, include: The processing module is used to parse the metadata of all files in the backup chain in parallel during the preprocessing stage and extract the data range information of the backup file corresponding to each metadata. The data range information includes the address range and physical address corresponding to the logical address, and the physical location is used to indicate the actual storage location of the backup file. The data range information is processed based on the scanline algorithm to determine the mapping relationship between the logical address and the physical address; The processing module is further configured to, during the data access phase, locate the mapping entry corresponding to the request address in the sparse mapping table for a read request, and read the target data based on the physical address corresponding to the mapping entry; and for an addressing request, locate the data area location or the void area location based on the sparse mapping table.

8. The apparatus according to claim 7, characterized in that, The processing module processes the data range information based on the scan-line algorithm to determine the specific method of the mapping relationship between the logical address and the physical address, including: The data range information is converted into start event points and end event points, and all start event points and end event points are globally sorted by logical address; Using scan lines, each start event point and each end event point are scanned sequentially according to the logical address order; Based on the index value corresponding to each of the data range information, determine the valid data source in the address range corresponding to each of the logical addresses; Based on each of the valid data sources, mapping entries are generated, and a sparse mapping table from the logical address to the physical address is constructed using each of the mapping entries.

9. A computer device, characterized in that, The method includes one or more processors and a memory storing computer-readable instructions that, when executed by the one or more processors, perform the steps of the method as described in any one of claims 1-6.

10. A storage medium, characterized in that, The storage medium stores computer-readable instructions that, when executed by one or more processors, cause the one or more processors to perform the steps of the method as described in any one of claims 1-6.

Images