A batch migration method for large file sets

Abstract

This invention belongs to the field of computer storage technology, specifically relating to a batch migration method for large-scale file sets. The method includes: directly loading and reading metadata information of the source file set from the source device; batch extracting metadata information of each file in the target file set; based on the metadata information of each valid file, obtaining the mapping relationship between all file logical blocks and their corresponding logical block addresses for that valid file, uniformly storing this mapping relationship in a data structure used for global data block indexing; sorting and merging all logical block addresses in the data structure in ascending order to obtain a new sequence of logical block addresses, with each logical block address corresponding to a contiguous read region; performing n disk reads through n direct I / O operations to obtain data from the n contiguous read regions; and sending all disk read data, metadata information of all valid files, and the data structure to the target device for reconstruction. This invention avoids the problem of frequent random I / O.

Description

Technical Field

[0001] This invention belongs to the field of computer storage technology, and more specifically, relates to a method for batch migration of large-scale file sets. Background Technology

[0002] The emergence of various data-intensive applications in the era of big data is rapidly changing the field of data storage. Simultaneously, with the continuous development of computer storage technology, the demand for accessing and processing large sets of files is increasing daily: enterprises need to perform high-frequency backups and migrations of massive files to ensure data availability; applications in various big data analytics systems need to read massive amounts of files to support parallel computing and simultaneously generate new files to store intermediate or final processing results. Furthermore, these massive files (usually small files) are mostly organized, stored, and accessed in batches.

[0003] Existing file systems mostly access files by calling the standard POSIX interface sequentially. While this approach effectively hides the complex internal implementation of the file system from applications, this single-file access method requires repeatedly traversing the entire I / O software stack when accessing a batch of files, and frequently reading and writing metadata and data at different locations on the underlying device, resulting in many discontinuous and random I / O operations. Therefore, this method leads to very poor access performance when handling a large number of small files.

[0004] Both local and remote batch file migrations are common scenarios; however, the single-file access method described above further degrades overall migration performance. Generally, local file migration requires reading a file from the source location, writing it to the destination location, and then processing the next file. This process necessitates calling the single-file access method once for each file to read and write metadata and data from different locations. This causes the access overhead of all files to accumulate continuously, significantly reducing file migration performance. Remote file migration also increases network transmission overhead, which further amplifies the overall cost of file migration.

[0005] In summary, the migration of massive files requires traversing the entire I / O software stack multiple times. Existing technologies suffer from redundant system calls, frequent random I / O, and a lack of structure awareness, leading to a blind migration process and a lack of fragment avoidance mechanisms on the target end, which fails to fully utilize the I / O performance of the underlying devices. Summary of the Invention

[0006] In view of the above-mentioned defects or improvement needs of the existing technology, the present invention provides a batch migration method for large-scale file sets, the purpose of which is to avoid the problem of frequent random I / O when performing large-scale file migration tasks.

[0007] To achieve the above objectives, according to one aspect of the present invention, a method for batch migration of large-scale file sets is provided, comprising: The metadata information of the source file set is directly loaded and read from the source device. The metadata information of each file in the target file set is extracted in batches from the metadata information. The validity of the file is determined based on the metadata information of each file. Based on the data mapping structure in the metadata information of each valid file, the mapping relationship between all file logical blocks and their corresponding logical block addresses of the valid file is obtained. The mapping relationship corresponding to all valid files is uniformly stored in a data structure for global data block indexing. All logical block addresses in the data structure are sorted in ascending order. The sorted logical block address sequence is traversed, and a region merging operation is performed on adjacent logical block addresses with consecutive addresses to obtain a new logical block address sequence. Each logical block address in the new logical block address sequence is recorded as a continuous read region, and there are n such regions. The data of the n continuous read regions is obtained by performing n disk reads in batches through n direct I / O operations. All disk read data, metadata information of all valid files, and the data structure are sent to the target end to perform the following operations: allocate a contiguous logical block address area on the target end with the same size as all received disk read data, write the disk read data corresponding to each valid file sequentially to achieve sequential layout storage, and reconstruct the metadata information of all valid files based on the layout order, metadata information of all valid files, and the data structure, and write the metadata area of ​​the target end file system in batches.

[0008] Furthermore, the implementation method for directly loading and reading metadata information from the source file set is as follows: Determine the starting address and length of the metadata area based on the type of the source device's file system; Based on the starting address and length of the metadata region, the metadata information of the source file set is loaded and read using the raw block access interface or by bypassing the system page cache.

[0009] Furthermore, when storing all the mapping relationships corresponding to all valid files into a data structure for global data block indexing, the method also includes: constructing a backfill mapping table to record the mapping relationship between logical block addresses and corresponding files; and sending the backfill mapping table to the target end when performing the sending. The method for reconstructing the metadata information of all valid files is as follows: Based on the sequential layout storage, backfill mapping table, and data structure used for global data block indexing, the new logical block address corresponding to each file logical address of each valid file on the target end is determined to construct the data block mapping structure required by the target file system. Based on this data block mapping structure, the metadata information of all valid files is changed in batches at once to obtain the reconstruction of the metadata information of all valid files.

[0010] Furthermore, the method for sending all disk read data, metadata information of all valid files, and the aforementioned data structure to the target end is as follows: After all disk read data is transmitted to the target end, metadata information is transmitted to the target end only after receiving an acknowledgment signal from the target end. The data structure is then transmitted only after receiving an acknowledgment signal from the target end.

[0011] According to another aspect of the present invention, a method for batch migration of a large-scale file set is provided, comprising: Receive the information sent by the source end by the batch migration method described above, including all disk read data, metadata information of all valid files, and data structures for global data block indexing; Allocate a contiguous logical block address region on the target end that is consistent with the size of all received disk read data, and write the disk read data corresponding to each valid file sequentially to achieve sequential layout storage. Based on the layout order, the metadata information of all valid files and the data structure, reconstruct the metadata information of all valid files and write it in batches to the target end file system metadata region.

[0012] Furthermore, when the information sent by the source also includes backfilling the mapping table, the method for reconstructing the metadata information of all valid files is as follows: Based on the sequential layout storage, backfill mapping table, and data structure used for global data block indexing, the new logical block address and file ownership corresponding to each file logical address of each valid file on the target end are determined to construct the data block mapping structure required by the target file system. Based on this data block mapping structure, the metadata information of all valid files is changed in batches at once to obtain the reconstruction of the metadata information of all valid files.

[0013] According to another aspect of the invention, an electronic device is provided, including a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement the steps of one or both of the methods described above.

[0014] According to another aspect of the invention, a computer-readable storage medium is provided, the computer-readable storage medium including a stored computer program, wherein, when the computer program is executed by a processor, it controls the device where the storage medium is located to perform the steps of one or both of the methods described above.

[0015] According to another aspect of the invention, a computer program product is provided, comprising a computer program or instructions that, when executed by a processor, implement the steps of one or both of the methods described above.

[0016] According to another aspect of the present invention, a batch migration system for large-scale file sets is provided, including a source end and a target end.

[0017] In summary, compared with the prior art, the technical solutions conceived by this invention have the following main advantages: 1. This invention provides a batch migration method for large-scale file sets. From the perspective of the source execution end, it first directly loads and reads the metadata information of the source file set from the source device, bypassing the complex operations of the file system and avoiding the complex kernel stack overhead. It then batch extracts the metadata information of each file in the target file set, avoiding single-file operations. Furthermore, it uniformly stores the mapping relationship between the file logical blocks of all valid files and their corresponding logical block addresses into a data structure used for global data block indexing. It sorts all logical block addresses in this data structure in ascending order, traverses the sorted logical block address sequence, and performs region merging operations on adjacent logical block addresses with consecutive addresses to obtain a new logical block address sequence. Each logical block address in this new logical block address sequence corresponds to a continuous read region, reducing the number of I / O operations and the complexity of spatial order. Finally, it sends all disk read data, metadata information of all valid files, and the data structure to the target end for reconstruction. Therefore, this invention is a data migration path based on structural parsing. Without heavily relying on the system kernel stack, it directly accesses the underlying structure information of the file system, extracts metadata and data block mappings in batches, avoids traversal overhead, significantly reduces system calls, context switching and control interactions, and achieves fast, sequential and highly consistent migration of tens of millions of small files.

[0018] 2. Regarding the transmission of all disk read data, metadata information of all valid files, and the data structure to the target end, the preferred implementation of this invention is as follows: after transmitting all disk read data to the target end, the metadata information is transmitted to the target end only after receiving an acknowledgment signal from the target end; and the data structure is transmitted only after receiving an acknowledgment signal from the target end. This is a transmission strategy of "data content first, metadata second, and other structures last," ensuring that the target end has a clear reconstruction context and order dependency guarantee during the reception process.

[0019] 3. This invention also proposes another method for batch migration of large-scale file sets. From the perspective of the target end's execution, after receiving the data sent by the source end, a contiguous logical block address region of the target end with the same size as all the received disk read data is allocated. The disk read data corresponding to each valid file is written sequentially to achieve sequential layout storage and improve storage throughput and prefetch efficiency. Then, based on the layout order, the metadata information of all valid files, and the data structure, the metadata information of all valid files is reconstructed and written in batches to the target end's file system metadata region to improve reconstruction efficiency.

[0020] 4. This invention also proposes that when the information sent by the source end includes a backfill mapping table, the method for reconstructing the metadata information of all valid files is as follows: based on the sequential layout storage, the backfill mapping table, and the data structure used for global data block indexing, determine the new logical block address and file ownership corresponding to each file logical address of each valid file on the target end, so as to construct the data block mapping structure required by the target end file system. Based on this data block mapping structure, the metadata information of all valid files is changed in batches at one time to obtain the reconstruction of the metadata information of all valid files. Attached Figure Description

[0021] Figure 1 A flowchart illustrating a batch migration method for a large-scale file set provided in an embodiment of the present invention; Figure 2 This is a schematic diagram illustrating an example of a file set migration process provided in an embodiment of the present invention; Figure 3 A schematic diagram illustrating the migration process of large-scale files provided in an embodiment of the present invention; Figure 4 This is a framework diagram of a batch migration method for a large-scale file set provided in an embodiment of the present invention. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0023] Example 1 A batch migration method for large-scale file sets, used for file migration between persistent storage devices, such as... Figure 1 As shown, it includes: The metadata information of the source file set is directly loaded and read from the source device. The metadata information of each file in the target file set is extracted in batches from the metadata information. The validity of the file is determined based on the metadata information of each file. Based on the data mapping structure in the metadata information of each valid file, the mapping relationship between all file logical blocks and their corresponding logical block addresses of the valid file is obtained. The mapping relationship corresponding to all valid files is uniformly stored in a data structure for global data block indexing. All logical block addresses in the data structure are sorted in ascending order. The sorted logical block address sequence is traversed, and a region merging operation is performed on adjacent logical block addresses with consecutive addresses to obtain a new logical block address sequence. Each logical block address in the new logical block address sequence is recorded as a continuous read region, and there are n such regions. The data of the n continuous read regions is obtained by performing n disk reads in batches through n direct I / O operations. All disk read data, metadata information of all valid files, and the data structure are sent to the target end to perform the following operations: allocate a contiguous logical block address area on the target end with the same size as all received disk read data, write the disk read data corresponding to each valid file sequentially to achieve sequential layout storage, and reconstruct the metadata information of all valid files based on the layout order, metadata information of all valid files, and the data structure, and write the metadata area of ​​the target end file system in batches.

[0024] This embodiment presents an efficient batch migration method for large-scale files. It can achieve unified extraction of metadata and data content of massive files, high sequential I / O, efficient migration and reconstruction of the target end sequential layout without relying on traditional file access paths. This greatly improves the data throughput and control efficiency of large-scale file systems during the migration process.

[0025] In general, the method of this embodiment can be divided into the following stages: (1) metadata extraction and valid file identification stage; (2) file data block mapping parsing and sequential construction stage; (3) multi-stage migration stage; (4) target end reception and sequential layout reconstruction stage.

[0026] (1) In the metadata extraction and valid file identification stage, the file system type used on the source device is first identified. Based on the file system specification, information on the key metadata stored on the physical medium by the file system is extracted.

[0027] Preferredly, the system can directly access core areas such as the superblock, block group descriptor, inode bitmap, block bitmap, inode table, and extent tree index via physical address location. During the read process, raw device read or an interface bypassing the page cache is used to directly load the required metadata blocks into memory from the physical offset address. This step does not require mounting a file system or relying on the kernel VFS path, making it suitable for cold migration scenarios such as static image analysis and snapshot reading.

[0028] More specific implementations of extracting metadata information include: using the file system superblock to extract information describing the parameters of the entire file system set, including key parameters such as data block size, inode size, block group size, and the boundary between the data area and the metadata area; obtaining the alignment, arrangement order, and overall distribution strategy of the stored file metadata based on this information; using the superblock to locate the starting position of the inode table, inode bitmap, and block bitmap; and using file metadata to obtain the physical offset (i.e., logical block address) of key structures such as the root node of the extent index tree.

[0029] Secondly, the metadata information of each file in the target file set is extracted in batches from the metadata information. Specifically, this includes: based on the starting address of each extracted inode (metadata) table (information recorded in the metadata area of ​​each device), the inode size of each valid file, and the inode arrangement rules in the inode table, batch traversal of the metadata of all files in the metadata area. When traversing the metadata of files in the file set in the metadata area, the corresponding metadata can be located by calculating offsets or reading bitmap indexes, without the need for path traversal or directory scanning.

[0030] For the metadata (inode) information of all files in the target file set, the validity of each extracted inode is judged according to the inode validity identifier corresponding to the file system type. Metadata marked as invalid, released, or reserved is skipped. For each valid metadata, information such as culture type, logical file size, owner user and group identifier, permission control information, timestamp information, and data mapping structure is extracted. At the same time, according to the data mapping structure, the starting position of the logical block address of the data mapping structure field of each file is located, providing an input interface for the data block mapping parsing in the next stage.

[0031] Finally, valid file identification is performed. Specifically, the structure access engine traverses all inode structure records and determines whether the file is validly allocated based on the validity flag bit corresponding to each record.

[0032] (2) The file data block mapping parsing and sequential construction stage follows the valid file meta-attribute extraction stage. Its purpose is to deeply parse the data block structure of each valid file, obtain the mapping relationship between multiple file logical blocks and their addresses, and, combined with the user's file reading behavior and the obtained logical block addresses, sort and merge the logical block addresses to construct a highly sequential storage address region that can be efficiently read in batches. The main operations in this stage include: mapping structure type identification and field location; data block mapping item expansion and standardization; and global physical address sorting and merging.

[0033] In the mapping structure type identification and field location operation, based on the data mapping structure field pointed to by the metadata information (inode) extracted in the previous stage, the mapping structure type used by that field is identified according to the file system category. For example, the pointer array structure used by Ext2 / Ext3, the extent tree structure used by Ext4 and XFS, etc. Based on the structure type, the corresponding structure parsing logic is used to recursively traverse all mapping entries, parsing each file logical block and its corresponding logical block address range, thus obtaining the mapping relationship between all file logical blocks and logical block addresses for each valid file.

[0034] Furthermore, in the global logical block address sorting and merging, each mapping entry is parsed into a standard format: file logical block number - logical block address (physical block) number - length. The file logical starting block number identifies the logical offset starting point of the data segment within the file, the starting logical block address identifies the physical starting position of the data segment, and the length identifies the number of consecutive blocks or bytes. All valid file mapping results (the mapping relationship from file logical blocks to logical block addresses) are uniformly stored in a specific data structure used for global data block indexing. Each entry in this data structure is sorted in ascending order according to its physical starting address, forming a complete sequential access view.

[0035] After sorting, the system performs a merging operation on adjacent block regions. This merging operation minimizes the amount of I / O required on the physical device, ultimately resulting in I / O covering all data blocks contained in the file set. The sorted and merged regions are organized into logical block address ranges or physical range tables to be read. Simultaneously, a corresponding backfill mapping table is generated for reconstruction operations on the target end. Each merged segment maintains the attribution of its contained logical segments to ensure no information is lost during subsequent recovery on the target end.

[0036] Through the above processing, the original fragmented data block layout is transformed into an optimal sequential read path and integrated into the minimum number of I / O operations. Then, Direct I / O can be used to read all file data of the file set using raw devices.

[0037] During the reading phase, Direct IO reading mode is used, based on the aforementioned list of physical regions to be read, such as... Figure 2 The file data of the migration file set shown is read directly in batches in sequence. The reading process skips the file system cache and page cache management modules, avoiding repeated addressing and system intervention, and maximizing the raw device read bandwidth. All read data is temporarily cached in the I / O cache pool, along with a meta tag indicating its source file number and logical location, to prepare for data alignment during reconstruction at the target end.

[0038] (3) Multi-stage migration stage. To ensure the consistency of the target end structure reconstruction order, the system adopts a phased transmission strategy, such as... Figure 3 As shown, the previously parsed file metadata and file data block content are organized and transmitted in stages according to a reasonable structure, consistent order, and controllable recovery. A preferred transmission strategy is to prioritize data content, followed by metadata, and then other structures, ensuring the target end has a clear reconstruction context and guaranteed order dependencies during reception. In the first stage, the sequentially read block content (data blocks corresponding to all files) is transmitted. After all data blocks have been transmitted and a confirmation response has been received from the target node, the file metadata is sent to the target end. After sending the metadata, specific data structures for global data block indexing and corresponding backfill mapping tables are sent to the target end. Finally, the system sends directory structure information to the target end, ensuring the target node can build a complete directory tree based on the existing file structure. The staged confirmation mechanism included in the above transmission process ensures that the next stage proceeds after each stage is successfully transmitted, guaranteeing consistency and fault tolerance during the migration process.

[0039] The system requires the target node to provide an explicit response confirmation after each stage of transmission is completed. The next stage only proceeds after the target node confirms successful reception of the previous stage, ensuring the atomicity of the transmission process. In other words, each stage of transmission requires confirmation from the target node before proceeding to the next stage, ensuring the continuity of the data and structure construction context and preventing errors caused by parts of the structure being written before the data.

[0040] (4) Target End Reception and Sequential Layout Reconstruction Stage. The target end reception and sequential layout reconstruction stage is the final execution stage of the entire migration process. It is responsible for efficiently reconstructing the received file data content and structure content into a mountable file system structure with complete file semantics at the target end. Its core is: to perform defractional data reconstruction by pre-allocating sequential physical space, and at the same time write back the data information and metadata information of the file set in batches.

[0041] Furthermore, before or during the reception of file data blocks, the target end calculates the required sequential space based on the logical size of each file in the file set and requests a physically contiguous free space region from the underlying storage device, also requiring the allocation of a contiguous logical block address. This pre-allocation mechanism prioritizes the use of contiguous space segments with high free space in the target device to avoid cross-region fragmentation. After obtaining the backfill mapping table sent by the source end, the target end integrates file-by-file sequential data blocks from the entire file data region sent by the source end, ensuring that all data blocks of each file are sequential, and writes the data block content of each file in the entire file set sequentially into the allocated contiguous storage region. After writing is completed, the system constructs the data block mapping structure required by the target end file system based on the recorded logical block offsets and actual physical addresses. Specifically, based on the sequential layout storage, the backfill mapping table, and the data structure used for global data block indexing, the system determines the new logical block addresses corresponding to the logical addresses of each valid file on the target end to construct the data block mapping structure required by the target end file system. Based on this data block mapping structure, the metadata information of all valid files is modified in batches at once to obtain a reconstruction of the metadata information of all valid files.

[0042] Because the target end has already been optimized into a continuous physical layout, the traditional complex mapping structure can be simplified into a single continuous interval structure, containing only three fields: file logical start block number, logical block start address, and continuous block length. This structure can significantly reduce the number of inode extents or mapping node depth, reducing metadata complexity and management burden. After completing all data mapping construction, the system generates new metadata information for each file, writing fields such as logical size, timestamp, permissions, and type. This metadata information is then written to the metadata area in batches. After all metadata areas are written, the system creates directory files in hierarchical order using the filenames and their inode numbers.

[0043] In a typical implementation example, the source file system is an Ext4 formatted disk volume, where file storage distribution exhibits significant fragmentation. The goal is to efficiently migrate a file set with a total capacity of approximately 10GB (containing tens of thousands of small files and some medium-sized log files) from the source block device to the target node. To improve data organization efficiency, the system employs the migration scheme proposed in this invention for a full migration.

[0044] Figure 2The diagram illustrates the migration process of a typical file set, with a file set size of 10GB, using file X as an example. On the source side, the data blocks of file X are scattered across discrete physical blocks such as Block 7, Block 9, and Block 16. Logically, it is a continuous file, but due to frequent add, delete, and write operations, its data is severely fragmented on the device. Traditional migration solutions would reproduce this on the target side. like Figure 2 As shown, on the target end, a 10GB space is first allocated. The physical blocks of all files are arranged according to the backfill mapping table transmitted from the source end. All files in this set are arranged into sequential storage blocks. Taking file X as an example, file X is completely written to a contiguous storage area. Its new mapping structure no longer uses multiple extents or indirect block pointers, but only a single range mapping (e.g., starting logical block 0 → physical block 8, length 4). Subsequently, a new inode entry is constructed for this file, and its metadata is batch-written to the metadata blocks of the target file system.

[0045] Through this process, the system successfully achieved automatic migration from fragmented layout to sequential reconstruction, and significantly optimized the spatial layout and reading efficiency on the target device without changing the file semantics and access logic.

[0046] Finally, based on the directory path mapping information, the logical directory structure of the file system is constructed. Parent directory paths are built layer by layer, and existing directory nodes are reused where paths already exist. Each file inode is mounted to its original path, thereby completing the full recovery from data content to directory semantics on the target device.

[0047] In summary, this embodiment addresses the problem of migrating massive amounts of files in existing file systems. By employing an efficient batch migration method for massive files, it avoids issues such as redundant system calls and frequent random I / O, maximizing the performance of the raw disk.

[0048] Example 2 A method for batch migration of large-scale file sets, comprising: The information received from the source end, obtained by the batch migration method as described in Example 1, includes all disk read data, metadata information of all valid files, and data structures for global data block indexing. Allocate a contiguous logical block address region on the target end that is consistent with the size of all received disk read data, and write the disk read data corresponding to each valid file sequentially to achieve sequential layout storage. Based on the layout order, the metadata information of all valid files and the data structure, reconstruct the metadata information of all valid files and write it in batches to the target end file system metadata region.

[0049] As a preferred option, when the information sent by the source also includes backfilling the mapping table, the implementation method for reconstructing the metadata information of all valid files is as follows: Based on the sequential layout storage, the backfill mapping table, and the data structure used for global data block indexing, the new logical block address corresponding to each file logical address of each valid file on the target end is determined to construct the data block mapping structure required by the target file system. Based on this data block mapping structure, the metadata information of all valid files is changed in batches at once to obtain the reconstruction of the metadata information of all valid files.

[0050] Example 3 This application also relates to an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the method described above.

[0051] The electronic device can be a desktop computer, laptop, handheld computer, or cloud server, etc. The processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The memory can be used to store computer programs and / or modules. The processor performs various functions of the electronic device by running or executing the computer programs and / or modules stored in the memory, and by accessing data stored in the memory.

[0052] The relevant technical solutions are the same as above, and will not be repeated here.

[0053] Example 4 This application also relates to a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described above.

[0054] Specifically, the memory may include high-speed random access memory, as well as non-volatile memory, such as hard disks, RAM, plug-in hard disks, smart media cards (SMC), secure digital (SD) cards, flash cards, at least one disk storage device, flash memory device, or other volatile solid-state storage devices.

[0055] The relevant technical solutions are the same as above, and will not be repeated here.

[0056] Example 5 This application provides a computer program product or computer program that includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the steps of the method described in the above embodiments of this application.

[0057] The relevant technical solutions are the same as above, and will not be repeated here.

[0058] Example 6 A batch migration system for large-scale file sets, including source and target ends, uses the same technical solutions as above and will not be repeated here.

[0059] In summary, to address the problems of high and frequent system call overhead, strong I / O randomness, complex structure reconstruction, and inability to optimize storage structure in existing storage technologies during large-scale file migration, this invention proposes an efficient batch migration method for large-scale files, such as... Figure 4 As shown, the overall concept is to bypass the traditional single-file access path and directly extract the meta-attributes and data block mapping structure of all valid files in the file set through the metadata area of ​​the underlying block device. After sorting and merging, the data is read from the underlying storage in batches from the disk data area with minimal Direct I / O to obtain all the file data required for the file set. The data, metadata and structural information are transmitted to the destination through a structured migration process. Each file data is reconstructed sequentially on the disk of the destination and the metadata information is constructed based on the data storage, thereby greatly improving migration efficiency, reducing fragmented reads and optimizing data layout.

[0060] Based on the above concept, in one embodiment of the present invention, the migration system consists of a source node and a destination node (i.e., it consists of a target end and a source end). Both nodes are block devices with a file system structure, such as EXT4, XFS, etc., and are SSDs, HDDs, or other storage devices with read and write access capabilities that support continuous writing.

[0061] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for batch migration of large-scale file sets, characterized in that, include: The metadata information of the source file set is directly loaded and read from the source device. The metadata information of each file in the target file set is extracted in batches from the metadata information. The validity of the file is determined based on the metadata information of each file. Based on the data mapping structure in the metadata information of each valid file, the mapping relationship between all file logical blocks and their corresponding logical block addresses of the valid file is obtained. The mapping relationship corresponding to all valid files is uniformly stored in a data structure for global data block indexing. All logical block addresses in the data structure are sorted in ascending order. The sorted logical block address sequence is traversed, and a region merging operation is performed on adjacent logical block addresses with consecutive addresses to obtain a new logical block address sequence. Each logical block address in the new logical block address sequence is recorded as a continuous read region, and there are n such regions. The data of the n continuous read regions is obtained by performing n disk reads in batches through n direct I / O operations. All disk read data, metadata information of all valid files, and the data structure are sent to the target end to perform the following operations: allocate a contiguous logical block address area on the target end with the same size as all received disk read data, write the disk read data corresponding to each valid file sequentially to achieve sequential layout storage, and reconstruct the metadata information of all valid files based on the layout order, metadata information of all valid files, and the data structure, and write the metadata area of ​​the target end file system in batches.

2. The batch migration method as described in claim 1, characterized in that, The implementation method for directly loading and reading metadata information from the source file set is as follows: Determine the starting address and length of the metadata area based on the type of the source device's file system; Based on the starting address and length of the metadata region, the metadata information of the source file set is loaded and read using the raw block access interface or by bypassing the system page cache.

3. The batch migration method as described in claim 1, characterized in that, When storing all the mapping relationships corresponding to all valid files into a data structure for global data block indexing, the method further includes: constructing a backfill mapping table to record the mapping relationship between logical block addresses and corresponding files; and sending the backfill mapping table to the target end when performing the sending. The method for reconstructing the metadata information of all valid files is as follows: Based on the sequential layout storage, the backfill mapping table, and the data structure used for global data block indexing, the new logical block address corresponding to each file logical address of each valid file on the target end is determined to construct the data block mapping structure required by the target file system. Based on this data block mapping structure, the metadata information of all valid files is changed in batches at once to obtain the reconstruction of the metadata information of all valid files.

4. The batch migration method as described in claim 1, characterized in that, The method for sending all disk read data, metadata information of all valid files, and the aforementioned data structure to the target end is as follows: After all disk read data is transmitted to the target end, metadata information is transmitted to the target end only after receiving an acknowledgment signal from the target end. The data structure is then transmitted only after receiving an acknowledgment signal from the target end.

5. A method for batch migration of large-scale file sets, characterized in that, include: The information received from the source end, obtained by the batch migration method as described in any one of claims 1 to 4, includes all disk read data, metadata information of all valid files, and data structures for global data block indexing; Allocate a contiguous logical block address region on the target end that is consistent with the size of all received disk read data, and write the disk read data corresponding to each valid file sequentially to achieve sequential layout storage. Based on the layout order, the metadata information of all valid files and the data structure, reconstruct the metadata information of all valid files and write it in batches to the target end file system metadata region.

6. The batch migration method as described in claim 5, characterized in that, When the information sent by the source also includes backfilling the mapping table, the method for reconstructing the metadata information of all valid files is as follows: Based on the sequential layout storage, the backfill mapping table, and the data structure used for global data block indexing, the new logical block address corresponding to each file logical address of each valid file on the target end is determined to construct the data block mapping structure required by the target file system. Based on this data block mapping structure, the metadata information of all valid files is changed in batches at once to obtain the reconstruction of the metadata information of all valid files.

7. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method as described in any one of claims 1 to 4 and / or the steps of the method as described in claim 5 or 6.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored computer program, wherein, when the computer program is executed by a processor, it controls the device on which the storage medium is located to perform the steps of the method as described in any one of claims 1 to 4 and / or the steps of the method as described in claim 5 or 6.

9. A computer program product, comprising a computer program or instructions, characterized in that, When the computer program or instructions are executed by a processor, they implement the steps of the method as described in any one of claims 1 to 4 and / or the steps of the method as described in claim 5 or 6.

10. A batch migration system for large-scale file sets, characterized in that, It includes a source end and a target end, wherein the source end is used to perform the steps of the method as described in any one of claims 1 to 4, and the target end is used to perform the steps of the method as described in claim 5 or 6.

Images