Data migration method and device, computer device and storage medium

Abstract

The application relates to a data migration method and device, computer equipment and a storage medium, and belongs to the technical field of distributed storage, the method comprising the following steps: determining data migration task parameters according to a received data migration request, wherein the data migration task parameters at least comprise a thread concurrency number; starting at least one thread according to the thread concurrency number and reading data and metadata information of a to-be-migrated file in a source remote network file sharing system through a first client library; and in response to completion of the reading, storing the data and metadata information of the to-be-migrated file into a file system of a target storage end through a second client library. The application reduces system resource occupation and improves data transmission efficiency on the basis of ensuring the consistency of file data and metadata.

Description

Technical Field

[0001] This application relates to the field of distributed storage technology, and in particular to a data migration method, apparatus, computer equipment, and storage medium. Background Technology

[0002] With the rapid development of big data and cloud computing, data storage and migration have become crucial issues. Ceph (distributed storage system) file system, as a high-performance and highly scalable distributed storage system, is widely used for the storage and management of large-scale data. However, in practical applications, there are still many problems in how to efficiently and reliably migrate file system data from remote to local Ceph storage file system. Traditional data migration methods require the deployment of third-party software such as rsync (an open-source file synchronization tool), which usually faces problems such as low transmission efficiency, excessive system resource consumption, inflexible customization of metadata synchronization, and inability to guarantee data consistency. Summary of the Invention

[0003] Therefore, it is necessary to provide a data migration method, apparatus, computer equipment, and storage medium that can improve transmission efficiency and reduce resource consumption in response to the above-mentioned technical problems.

[0004] Firstly, a data migration method is provided, the method comprising:

[0005] Receive data migration requests;

[0006] Based on the data migration request, determine the data migration task parameters, which should include at least the number of concurrent threads.

[0007] Based on the number of concurrent threads, at least one thread is started and the data and metadata information of the files to be migrated in the source remote network file sharing system are read through the first client library. The first client library is used to directly access the source remote network file sharing system through the first programming interface.

[0008] In response to the completion of the read operation, the data and metadata information of the file to be migrated are stored in the file system of the destination storage through the second client library. The second client library is used to directly access the file system of the destination storage through the second programming interface.

[0009] Optionally, based on the data migration request, the data migration task parameters include:

[0010] Parse the data migration request and determine the data migration task parameters based on the parsing results. The data migration task parameters include at least the shared address of the remote network file sharing system, the file system identifier of the destination storage end and its corresponding write directory, and the data migration method.

[0011] The methods for determining the number of concurrent threads include:

[0012] Obtain the metadata information corresponding to the target file to be migrated;

[0013] Based on the metadata information corresponding to the target file to be migrated, determine the space occupied by the target file to be migrated;

[0014] In response to the fact that the space occupied by the target file to be migrated is less than or equal to the first preset threshold, the number of concurrent threads is determined to be single thread;

[0015] In response to the target file to be migrated occupying more space than a first preset threshold, a preset read length is obtained, and the target file to be migrated is split into multiple fragments according to the preset read length;

[0016] Based on the number of shards in the multiple shards and the number of files to be migrated, determine the number of concurrent threads corresponding to the target files to be migrated;

[0017] Based on the number of threads corresponding to multiple target files to be migrated, the total number of threads is determined, and the total number of threads is less than or equal to the number of threads in the thread pool.

[0018] Optionally, after determining the data migration task parameters based on the data migration request, the method further includes:

[0019] Integrate the first client library and the second client library on the destination storage side;

[0020] Based on the shared address of the remote network file sharing system, the first client library is used to connect to and mount the remote network file sharing system.

[0021] Optionally, depending on the number of concurrent threads, at least one thread is started and reads the data and metadata information of the files to be migrated from the source remote network file-sharing system through the first client library, including:

[0022] Based on the first client library, the files and directories to be migrated in the remote network file sharing system of the source end are traversed and scanned, and the parameter information of the files to be migrated determined according to the traversal and scan results is recorded. The parameter information of the files to be migrated includes at least the hash value.

[0023] Based on the traversal scan order, preset read length, and number of concurrent threads, at least one thread is started and reads the data of the files to be migrated from the source remote network file sharing system through the first client library.

[0024] Optionally, based on the number of concurrent threads, starting at least one thread and reading the data and metadata information of the files to be migrated from the source remote network file-sharing system through the first client library also includes:

[0025] In response to the completion of data reading of the files to be migrated, the metadata information of the files to be migrated in the source remote network file sharing system is read through the first client library.

[0026] Optionally, in response to the completion of reading the data of the file to be migrated, the method further includes:

[0027] Based on the file system identifier of the destination storage, determine the file system of the destination storage, and detect whether the target directory exists in the file system according to the write directory corresponding to the file system identifier of the destination storage.

[0028] In response to the existence of the file, the data to be migrated is stored in the corresponding target directory of the file system through the second client library;

[0029] If the target directory does not exist, the target directory is created, and after creation, the data of the files to be migrated is stored in the corresponding target directory in the file system through the second client library;

[0030] In response to the fact that the data of the file to be migrated has been stored in the corresponding target directory in the file system, the first client library reads the metadata information of the file to be migrated in the remote network file sharing system of the source end, and the second client library stores the metadata information of the file to be migrated in the corresponding target directory in the file system.

[0031] Optionally, after storing the data and metadata information of the files to be migrated to the file system of the destination storage via a second client library, the method further includes:

[0032] Based on the data migration request, determine whether consistency verification is required for the target files stored in the destination storage file system.

[0033] In response to the need to perform consistency checks on the target file, calculate the hash value of the target file;

[0034] If the hash value of the target file is successfully compared with the hash value of the file to be migrated, the target file is determined to have passed the consistency check.

[0035] If the hash value of the target file fails to match the hash value of the file to be migrated, it is determined that the target file has not passed the consistency check, and the data migration operation is re-executed.

[0036] Secondly, a data migration apparatus is provided, the apparatus comprising:

[0037] The request receiving module is used to receive data migration requests;

[0038] The parameter determination module is used to determine the data migration task parameters based on the data migration request. The data migration task parameters include at least the number of concurrent threads.

[0039] The first data migration module is used to start at least one thread according to the number of concurrent threads and read the data and metadata information of the file to be migrated in the source remote network file sharing system through the first client library. The first client library is used to directly access the source remote network file sharing system through the first programming interface.

[0040] The second data migration module is used to store the data and metadata information of the file to be migrated to the file system of the destination storage terminal through the second client library in response to the completion of reading. The second client library is used to directly access the file system of the destination storage terminal through the second programming interface.

[0041] Thirdly, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to perform the following steps:

[0042] Receive data migration requests;

[0043] Based on the data migration request, determine the data migration task parameters, which should include at least the number of concurrent threads.

[0044] Based on the number of concurrent threads, at least one thread is started and the data and metadata information of the files to be migrated in the source remote network file sharing system are read through the first client library. The first client library is used to directly access the source remote network file sharing system through the first programming interface.

[0045] In response to the completion of the read operation, the data and metadata information of the file to be migrated are stored in the file system of the destination storage through the second client library. The second client library is used to directly access the file system of the destination storage through the second programming interface.

[0046] Fourthly, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, performs the following steps:

[0047] Receive data migration requests;

[0048] Based on the data migration request, determine the data migration task parameters, which should include at least the number of concurrent threads.

[0049] Based on the number of concurrent threads, at least one thread is started and the data and metadata information of the files to be migrated in the source remote network file sharing system are read through the first client library. The first client library is used to directly access the source remote network file sharing system through the first programming interface.

[0050] In response to the completion of the read operation, the data and metadata information of the file to be migrated are stored in the file system of the destination storage through the second client library. The second client library is used to directly access the file system of the destination storage through the second programming interface.

[0051] Fifthly, a computer program product is provided, the computer program product comprising a computer program, which, when executed by a processor, performs the following steps:

[0052] Receive data migration requests;

[0053] Based on the data migration request, determine the data migration task parameters, which should include at least the number of concurrent threads.

[0054] Based on the number of concurrent threads, at least one thread is started and the data and metadata information of the files to be migrated in the source remote network file sharing system are read through the first client library. The first client library is used to directly access the source remote network file sharing system through the first programming interface.

[0055] In response to the completion of the read operation, the data and metadata information of the file to be migrated are stored in the file system of the destination storage through the second client library. The second client library is used to directly access the file system of the destination storage through the second programming interface.

[0056] The aforementioned data migration method, apparatus, computer equipment, and storage medium, the method comprising: receiving a data migration request; determining data migration task parameters according to the data migration request, the data migration task parameters including at least the number of concurrent threads; starting at least one thread according to the number of concurrent threads and reading data and metadata information of the file to be migrated from the source remote network file sharing system through a first client library, the first client library being used to directly access the source remote network file sharing system through a first programming interface; and, in response to the completion of reading, storing the data and metadata information of the file to be migrated to the file system of the destination storage end through a second client library, the second client library being used to directly access the file system of the destination storage end through a second programming interface. This application, by using the first client library and the second client library, can realize the mounting and data reading of the remote network file sharing system, and the migration of file data and metadata information through multi-threading. While ensuring the integrity and consistency of file data and metadata, it reduces the overhead of data migration based on the POSIX file system, lowers system resource consumption, and improves data transmission efficiency. Attached Figure Description

[0057] Figure 1 This is a diagram illustrating the application environment of a data migration method in one embodiment.

[0058] Figure 2This is a flowchart illustrating a data migration method in one embodiment;

[0059] Figure 3 This is a schematic diagram of the file migration timing of a data migration method in one embodiment;

[0060] Figure 4 This is a schematic diagram of the file migration module of a data migration method in one embodiment;

[0061] Figure 5 This is a structural block diagram of a data migration device in one embodiment;

[0062] Figure 6 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0063] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0064] It should be understood that, in the description of this application, unless the context explicitly requires it, the words "including," "comprising," and similar terms throughout the specification should be interpreted as including rather than exclusive or exhaustive; that is, meaning "including but not limited to."

[0065] It should also be understood that the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance. Furthermore, in the description of this application, unless otherwise stated, "a plurality of" means two or more.

[0066] It should be noted that the terms "S1," "S2," etc., are used only for descriptive purposes and do not specifically refer to the order or sequence, nor are they intended to limit this application. They are merely for the convenience of describing the method of this application and should not be construed as indicating the sequential order of the steps. Furthermore, the technical solutions of the various embodiments can be combined with each other, but this must be based on the ability of those skilled in the art to implement them. When the combination of technical solutions is contradictory or impossible to implement, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed in this application.

[0067] As the background technology indicates, traditional data migration methods have many problems. For example, to migrate data from any source storage device's file system to the destination storage, related technologies require configuring an NFS share (Remote Network File Sharing System) on the source data storage file system. Then, a new system directory is created on the destination storage, and the NFS share exported from the source data storage is mounted to this directory. Third-party file synchronization and backup tools, such as rsync, are used to configure the necessary parameters, and synchronization can be triggered manually or periodically. This copies the data from the NFS share directory to the specified file system directory on the destination storage. Configuring an NFS share on the source data file system ensures that the source data can be accessed, read, and written by other storage devices over the network; this is a prerequisite for all data migration functions. A directory is created on the destination storage system, and the source NFS share is mounted to this directory using the system mount command. This directory accesses data under the source file system. The system integrates the rsync file synchronization tool, configuring the directory mounted on the source file system as the source address and the local file system on the destination storage as the destination address. The rsync tool is executed manually or periodically to perform data copying and metadata synchronization between the two directories. Its problems include the need to create a directory on the destination storage to mount the source NFS share, and the requirement that the destination storage Ceph file system be publicly visible to ensure that the rsync tool can access data on both ends simultaneously, increasing the operation steps; rsync consumes significant CPU (Central Processing Unit) and I / O (Input / Output) resources, which can impact other services on the destination storage; in multi-user or multi-tasking scenarios, if multiple processes attempt to modify the same file simultaneously, rsync lacks a native locking mechanism to avoid conflicts, potentially leading to data inconsistency after migration.

[0068] To address the aforementioned technical problems, this application provides a data migration method, apparatus, computer equipment, and storage medium. By using a first client library and a second client library, it is possible to mount and read data from a remote network file sharing system, and migrate file data and metadata information through multi-threading. While ensuring the integrity and consistency of file data and metadata, it reduces the overhead of data migration based on the POSIX file system, lowers system resource consumption, and improves data transmission efficiency.

[0069] The data migration method provided in this application can be applied to, for example... Figure 1 In the application environment shown, terminal 102 communicates with a data processing platform set on server 104 via a network. Terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices. Server 104 can be implemented as a standalone server or a server cluster consisting of multiple servers.

[0070] In one embodiment, such as Figure 2 As shown, a data migration method is provided, which is applied to... Figure 1 Taking the terminal in the example, the explanation includes the following steps:

[0071] S1: Receive data migration requests.

[0072] It should be noted that a data migration request can be a request issued by the user in real time or a request issued by the system at regular intervals according to actual needs. The data migration request includes data migration task parameters.

[0073] S2: Based on the data migration request, determine the data migration task parameters, which should include at least the number of concurrent threads.

[0074] It should be noted that by parsing the data migration request, the data migration task parameters can be obtained. The specific data migration task parameters are: (1) the NFS (Network File System) shared address provided by the source storage, which includes IP (Internet Protocol address) and shared path; (2) the file system and directory on the destination storage to determine the write directory for data migration; (3) the data migration method selected according to the corresponding migration task in different scenarios, including full migration and incremental migration; (4) the metadata to be synchronized in the custom part, such as file ownership group, file permissions, etc.; (5) whether file consistency needs to be verified. If verification is required, after the files to be migrated are migrated, the MD5 value (hash value) of the source file and the destination file needs to be verified. If the verification fails, the migration will be attempted again. Because calculating the MD5 of the file requires additional CPU resources, the migration task duration will be extended. Therefore, the data consistency verification operation can be determined based on the sensitivity to the migration duration. The methods for determining the sensitivity to the migration duration include:

[0075] Obtain the identifier of the target operating system and the migration duration over multiple different time periods. The time periods can be set according to actual needs, such as one week.

[0076] Obtain the migration duration corresponding to multiple target time periods within different time cycles, and determine the fluctuation coefficient of the migration duration. The target time period can be set according to actual needs, such as one day of the week. The fluctuation coefficient is obtained by subtracting the minimum value from the maximum value of the migration duration within that time period.

[0077] When the fluctuation coefficient is less than the first preset value, it is determined whether a consistency check operation is performed on the file during the target time period, and the number of times it is performed is determined. The first preset value can be set according to actual needs.

[0078] When a file consistency check operation is performed on the target time period and the number of executions exceeds the second preset value, the migration duration of the target time period is determined to be less sensitive. The second preset value can be set according to actual needs.

[0079] When there is no consistency check operation performed on the file during the target time period and / or the number of operations is less than or equal to the second preset value, the migration duration of the target time period is determined to be highly sensitive.

[0080] If the migration duration is less sensitive, a data consistency check will be performed during the target time period in the next time period. If the migration duration is more sensitive, a data consistency check will not be performed during the target time period in the next time period.

[0081] In the above implementation, determining whether to perform a data consistency verification operation based on the strength of the migration duration sensitivity can improve the reliability of data migration.

[0082] In some specific implementations, determining the data migration task parameters based on the data migration request includes:

[0083] Parse the data migration request and determine the data migration task parameters based on the parsing results. The data migration task parameters include at least the shared address of the remote network file sharing system, the file system identifier of the destination storage end and its corresponding write directory, and the data migration method.

[0084] The methods for determining the number of concurrent threads include:

[0085] Obtain the metadata information corresponding to the target file to be migrated. The metadata information required here includes the size information of the target file to be migrated, such as 10 MiB.

[0086] Based on the metadata information corresponding to the target file to be migrated, determine the space occupied by the target file to be migrated, that is, the storage capacity occupied by the target file to be migrated.

[0087] In response to the fact that the space occupied by the target file to be migrated is less than or equal to a first preset threshold, the number of concurrent threads is determined to be a single thread. The first preset threshold can be set according to actual needs, such as 10MiB. That is, when the space occupied by the target file to be migrated is less than or equal to 10MiB, the data of the target file to be migrated is read by a single thread.

[0088] In response to the target file to be migrated occupying more space than a first preset threshold, a preset read length is obtained, and the target file to be migrated is split into multiple fragments according to the preset read length. The preset read length can be set according to actual needs, such as 1MiB. That is, when the target file to be migrated occupies more than 10MiB of space, the target file to be migrated is split into multiple fragments of 1MiB size.

[0089] Based on the number of shards in the multiple shards and the number of files to be migrated, determine the number of concurrent threads corresponding to the target file to be migrated. Specifically, the number of initial threads can be determined based on the number of files to be migrated, i.e., one thread corresponds to one file to be migrated. At this time, it is necessary to obtain the number of remaining threads in the thread pool. If there are remaining threads, the remaining threads are evenly distributed according to the number of shards in the multiple shards. For example, if the number of shards in the first file to be migrated is 20 and the number of shards in the second file to be migrated is 30, then 1 remaining thread is allocated to the first file to be migrated, 2 remaining threads are allocated to the second file to be migrated, and so on. This will not be elaborated further.

[0090] Based on the number of concurrent threads corresponding to multiple target files to be migrated, the total number of concurrent threads is determined. The total number of concurrent threads is less than or equal to the number of threads in the thread pool. To reduce the system resource overhead caused by frequent thread creation and destruction, a thread pool containing a fixed number of threads is pre-created. When a new thread is added, resources are directly taken from the thread pool. After the thread task is completed, no resource release is required. When the main process initiates a file open or read request, all tasks belonging to the same NFS connection are added to a queue. The tasks in the queue are distributed to threads in the thread pool for execution. When the connection between the first client library and the remote network file sharing system is broken, the thread pool is destroyed. Multiple threads can execute multiple migration tasks concurrently. That is, each migration task starts its own thread and runs independently, without affecting each other. Each task thread keeps running while consuming as few system resources as possible until all files are migrated, at which point the process automatically ends and resources are released.

[0091] Specifically, the thread pool creation method includes: defining a thread pool structure `thread_pool`, which contains a thread array `threads`, a thread count `thread_count`, a task queue `task_queue`, and a thread pool running status flag `running_flag`. The thread pool retrieves tasks from the task queue `task_queue` for execution. The thread pool is initialized to determine a specified number of threads and stored in the thread array `threads`. During subsequent file migration, all received requests are stored in the task queue and distributed by the thread pool to multiple threads for execution. The task queue `task_queue` is a newly added global task queue structure in libnfs. This queue contains: head and tail pointers of the task list to be executed, used for traversing tasks; a queue mutex thread lock and condition variable to ensure the consistency of task queue access; and a task execution status `task_status` to indicate whether the task is in progress, successful, or failed.

[0092] In some specific implementations, such as Figure 3 As shown, after determining the data migration task parameters based on the data migration request, the method further includes:

[0093] The first client library and the second client library are integrated on the destination storage side. The first client library refers to libnfs, which is a client development library for accessing NFS shares over the network. It can perform operations such as mounting, reading and writing on NFS shares, allowing applications to directly access files in NFS shares through a programming interface without explicitly mounting the NFS shares to the system directory. The second client library refers to libClient (Ceph Client Library), which is a client library for interacting with Ceph clusters. It allows applications to directly access the Ceph file system through a programming interface without needing to migrate file data through the POSIX file system interface.

[0094] Based on the shared address of the remote network file sharing system, the first client library is used to connect to and mount the remote network file sharing system. That is, before the migration task begins, the libnfs library and libClient library are integrated into the main process of the destination storage. According to the input source NFS share IP address and shared path, the nfs_mount interface of the libnfs library is called to connect to and mount the remote NFS share. After successful mounting, all files on the source can be directly accessed. Specifically, libnfs, as a client library, can efficiently implement NFS protocol communication and support access to files mounted on the source NFS in the system memory of the destination without creating directories in the destination system, thereby saving the resource overhead of the POSIX (Portable Operating System Interface) layer.

[0095] In the above implementation, by setting up multiple task processing threads, the number of requests processed per unit time can be increased. By using the libnfs library and libClient library to implement remote NFS sharing and data reading, the system resource consumption is reduced while improving file transfer efficiency.

[0096] S3: Based on the number of concurrent threads, start at least one thread and read the data and metadata information of the files to be migrated in the source remote network file sharing system through the first client library. The first client library is used to directly access the source remote network file sharing system through the first programming interface.

[0097] It should be noted that, as can be seen from step S2, the number of concurrent threads is determined by the number of threads in the thread pool and the size of the file to be migrated; the file migration includes two main parts, namely the migration of file content (data) and file attributes (metadata information). The first programming interface refers to defining a specification or protocol in programming that specifies the method of communication between classes or components. It is used to specify the communication method between the first client library and the source remote network file sharing system.

[0098] In some specific implementations, based on the number of concurrent threads, starting at least one thread and reading the data and metadata information of the files to be migrated from the source remote network file sharing system through the first client library includes:

[0099] Based on the first client library, the files and directories to be migrated in the remote network file sharing system of the source end are traversed and scanned, and the parameter information of the files to be migrated is recorded according to the traversal and scan results. The parameter information of the files to be migrated includes at least the hash value. Specifically, libnfs is used to traverse and scan the directories and files in the source NFS share, and record the total number of files to be migrated, the file md5 value (i.e., hash value), the last modification timestamp, and other information, which are used to mark the progress of the migration task and to verify the files after migration.

[0100] Based on the traversal scan order, preset read length, and number of concurrent threads, at least one thread is started and reads the data of the files to be migrated from the source remote network file sharing system through the first client library.

[0101] Specifically, in the main program of the destination storage, the file data content in the source NFS share is read through libnfs according to the scanning order. If the preset read length is 1MiB, then at most 1MiB of data content in the file to be migrated is read at a time to control the program's memory usage and improve read efficiency. If the size of the file to be migrated is less than 1MiB, then it is read according to the actual file size. After the corresponding file data is read by multiple threads, if there are other files, the reading operation of other files is continued, and so on.

[0102] In some specific implementations, starting at least one thread and reading the data and metadata information of the files to be migrated from the source remote network file-sharing system through the first client library, based on the number of concurrent threads, further includes:

[0103] In response to the completion of data reading of the files to be migrated, the metadata information of the files to be migrated in the source remote network file sharing system is read through the first client library.

[0104] Specifically, after the data of the file to be migrated is read, the source file's metadata information is obtained through libnfs. The metadata information may include access time (the time when the file was read or executed), modify time (the time when the file content was modified), change time (the time when the file status changed), mode (file permissions), uid (the user ID to which the file belongs), gid (the user group ID to which the file belongs), and size (file size information), etc. After obtaining the file metadata information, it is filtered according to the custom metadata set in the data migration request. If it is not specified in the data migration request, all the above metadata information is read by default. The number of threads for reading metadata information can be the number of threads corresponding to reading data, or it can be recalculated based on the space occupied by the metadata information. The specific calculation method is the same as the method for calculating the number of threads for reading data in the above steps, and will not be repeated here. You can choose one of the two methods to determine the number of concurrent threads according to the actual needs.

[0105] In the above implementation, libnfs reads the data and metadata information of the file to be migrated, and Ceph's metadata synchronization interface automatically matches the custom metadata items corresponding to the data request, which improves the flexibility of metadata synchronization and ensures the integrity and consistency of data and metadata, thereby improving the reliability of data migration.

[0106] S4: In response to the completion of reading, the data and metadata information of the file to be migrated are stored in the file system of the destination storage through the second client library. The second client library is used to directly access the file system of the destination storage through the second programming interface.

[0107] It should be noted that the second client library refers to libClient (Ceph Client Library), and the destination storage file system refers to the Ceph file system; the second programming interface refers to defining a specification or protocol in programming that specifies the methods for communication between classes or components, and it is used to specify the communication methods between the second client library and the destination storage file system.

[0108] In some specific implementations, in response to the completion of reading the data of the file to be migrated, the method further includes:

[0109] Based on the file system identifier of the destination storage end, determine the file system of the destination storage end, and detect whether the target directory exists in the file system according to the write directory corresponding to the file system identifier of the destination storage end. The file system identifier of the destination storage end and its corresponding write directory are obtained by the traversal scan operation in the above steps, such as file A.

[0110] In response to the existence of the file, the data to be migrated is stored in the corresponding target directory of the file system through the second client library;

[0111] If the target directory does not exist, the target directory is created, and after creation, the data of the files to be migrated is stored in the corresponding target directory in the file system through the second client library;

[0112] In response to the fact that the data of the file to be migrated has been stored in the corresponding target directory in the file system, the first client library reads the metadata information of the file to be migrated in the remote network file sharing system of the source end, and the second client library stores the metadata information of the file to be migrated in the corresponding target directory in the file system.

[0113] Specifically, such as Figure 4 As shown, when the data of the file to be migrated has been read into libnfs, the main program on the destination storage side uses libClient to write the read data sequentially into the target directory of the Ceph file system. Before writing the data, it first checks whether the directory or file exists in the Ceph file system. If it does not exist, it creates a new file or directory with the same name and then appends the data. Each time, a maximum of 1 MiB of content is written until the data of the file is read and written. Then, the creation and writing of the next file is performed. After the data of the file to be migrated has been written into the Ceph file system, the metadata information of the file to be migrated is sequentially written into the target directory of the Ceph file system using libClient.

[0114] Furthermore, after reading data and metadata information through libnfs, since there may be situations where data or metadata information is read in parallel by multiple threads in fragments, where the position read by each thread is determined by the starting offset and the read length to avoid duplicate reads, after all fragments are read, they are merged in the global cache according to the offset. Based on this, the fragment merging process is carried out by adding a global cache and a thread-local cache. The global cache is used to store shared data of the same NFS connection, such as storing all task queues under this connection and the data fragments merged by each thread. The thread-local cache is responsible for the read data stored by each thread.

[0115] More specifically, the file migration process includes: following the traversal order of the entire shared directory structure, starting from the lowest level directory, using the `nfs_open` interface of the libnfs library to open the file from the source, using the `nfs_read` interface to read the file, and specifying a maximum read buffer size of 1 MiB to prevent excessive cache usage from reading the entire file at once. Simultaneously, if the target directory does not exist in the local Ceph file system, the `mkdir` interface of the libClient library is called to create a directory with the same name as the source on the local Ceph file system's destination path. The `create_and_open` interface is then called to create a file with the same name as the source, and the read content is appended to the local file in 1 MiB increments until the file to be migrated is read and written. Finally, the `nfs_open` interface of the libnfs library is called. The `close` command closes any open files to be migrated on the source end. At this point, the data migration of the files to be migrated is complete. After the data migration is finished, metadata synchronization is performed. This involves using the `nfs_fstat64` interface of the libnfs library to obtain all metadata information from the source files, matching it with custom metadata items, and only synchronizing the set metadata items. The `setattr` interface of the libClient library is then used to set the source file metadata information onto the files in the Ceph file system. To ensure data consistency during multi-threaded task execution, additional steps are added for file opening, reading, and metadata retrieval operations. Operation-level thread locks refine the locking to the file level, allowing multiple threads to simultaneously handle open and read operations on different files within the same NFS connection session. This is achieved by adding operation-level mutexes (pthread_mutex_ops) to the NFS file handle structure (nfsfh). This allows open, read, and getattr operations on different files in the task queue to execute independently without interfering with each other. In cases where large files are not accessed in chunks, each thread performs migration operations on different files independently from the task queue. All multi-threaded task execution uses libnfs asynchronous interfaces, such as nfs_open_async and nfs_read_async.

[0116] In some specific implementations, after storing the data and metadata information of the files to be migrated to the file system of the destination storage via a second client library, the method further includes:

[0117] Based on the data migration request, determine whether consistency verification is required for the target files stored in the destination storage file system.

[0118] In response to the need to perform consistency verification on the target file, the hash value of the target file is calculated. The hash value is a fixed-length string calculated by a hash function. The calculation process is a common method and will not be described in detail here.

[0119] After obtaining the hash value of the target file, compare the hash value of the target file with the hash value of the file to be migrated;

[0120] In response to a successful comparison between the hash value of the target file and the hash value of the file to be migrated, the target file passes the consistency check, indicating that the file has been migrated. The number of migrated files is incremented by 1 to indicate the migration progress.

[0121] If the hash value of the target file fails to match the hash value of the file to be migrated, it is determined that the target file has not passed the consistency check, indicating that there is a problem with the file data or metadata information during the migration process. This file path is added to the retry queue to wait for migration again, so that the data migration operation can be re-executed.

[0122] In some specific implementations, a timeout retry and exit mechanism is set for each thread. When a network failure or other abnormality occurs, the thread executing the task first retryes. After reaching the maximum number of retry attempts, it notifies the global task queue of the task failure message and distributes the task in the task queue to other threads for execution.

[0123] In some specific implementations, the progress of the task is indicated by the number of files migrated / the total number of files to be migrated, which can be viewed by the user in real time. After the task is completed, the number of files that failed to migrate is recorded and displayed.

[0124] In the above implementation, data is saved to the file system of the destination storage through libClient, and data consistency verification is performed after the migration task is completed. This not only improves the efficiency of file migration, but also ensures the data consistency after the file migration is completed to the greatest extent, thereby further improving the reliability of data migration.

[0125] The above data migration method includes: receiving a data migration request; determining data migration task parameters according to the data migration request, the data migration task parameters including at least the number of concurrent threads; starting at least one thread according to the number of concurrent threads and reading the data and metadata information of the file to be migrated in the source remote network file sharing system through a first client library, the first client library being used to directly access the source remote network file sharing system through a first programming interface; in response to the completion of reading, storing the data and metadata information of the file to be migrated to the file system of the destination storage end through a second client library, the second client library being used to directly access the file system of the destination storage end through a second programming interface. The beneficial effects of this application include: (1) improving the efficiency of data migration: by using the libnfs library and libClient library to realize the mounting and data reading of remote NFS sharing, and at the same time utilizing the libClient read and write interface of the Ceph file system. (1) Data writing is performed through the interface, and files in NFS shares and Ceph can be manipulated in the code without relying on mounting, reducing the overhead of the POSIX file system and improving migration efficiency; (2) Support for custom metadata synchronization: Through Ceph's metadata synchronization interface, custom metadata items corresponding to data migration requests are automatically matched, which improves the flexibility of metadata synchronization and ensures the integrity and consistency of data and metadata; (3) Strong compatibility: It can be used in various NFS-based remote storage environments, with strong compatibility and applicability, and can meet the data migration needs of different application scenarios; (4) Improve data migration reliability: The high availability and fault tolerance of the Ceph file system ensure the security and reliability of data during the migration process, avoid data loss or damage caused by storage system failure, and support optional migration task data consistency verification, which can maximize the data consistency after the file migration is completed.

[0126] It should be understood that, although Figures 2-4 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order in which these steps are executed, and they can be performed in other orders. Figures 2-4 At least some of the steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.

[0127] In one embodiment, such as Figure 5As shown, a data migration apparatus is provided, comprising: a request receiving module, a parameter determining module, a first data migration module, and a second data migration module, wherein:

[0128] The request receiving module is used to receive data migration requests;

[0129] The parameter determination module is used to determine the data migration task parameters based on the data migration request. The data migration task parameters include at least the number of concurrent threads.

[0130] The first data migration module is used to start at least one thread according to the number of concurrent threads and read the data and metadata information of the file to be migrated in the source remote network file sharing system through the first client library. The first client library is used to directly access the source remote network file sharing system through the first programming interface.

[0131] The second data migration module is used to store the data and metadata information of the file to be migrated to the file system of the destination storage terminal through the second client library in response to the completion of reading. The second client library is used to directly access the file system of the destination storage terminal through the second programming interface.

[0132] In a preferred embodiment of the present invention, the parameter determination module is specifically used for:

[0133] Parse the data migration request and determine the data migration task parameters based on the parsing results. The data migration task parameters include at least the shared address of the remote network file sharing system, the file system identifier of the destination storage end and its corresponding write directory, and the data migration method.

[0134] The methods for determining the number of concurrent threads include:

[0135] Obtain the metadata information corresponding to the target file to be migrated;

[0136] Based on the metadata information corresponding to the target file to be migrated, determine the space occupied by the target file to be migrated;

[0137] In response to the fact that the space occupied by the target file to be migrated is less than or equal to the first preset threshold, the number of concurrent threads is determined to be single thread;

[0138] In response to the target file to be migrated occupying more space than a first preset threshold, a preset read length is obtained, and the target file to be migrated is split into multiple fragments according to the preset read length;

[0139] Based on the number of shards in the multiple shards and the number of files to be migrated, determine the number of concurrent threads corresponding to the target files to be migrated;

[0140] Based on the number of threads corresponding to multiple target files to be migrated, the total number of threads is determined, and the total number of threads is less than or equal to the number of threads in the thread pool.

[0141] In a preferred embodiment of the present invention, the device further includes a connection mounting module, which is specifically used for:

[0142] Integrate the first client library and the second client library on the destination storage side;

[0143] Based on the shared address of the remote network file sharing system, the first client library is used to connect to and mount the remote network file sharing system.

[0144] In a preferred embodiment of the present invention, the first data migration module is specifically used for:

[0145] Based on the first client library, the files and directories to be migrated in the remote network file sharing system of the source end are traversed and scanned, and the parameter information of the files to be migrated determined according to the traversal and scan results is recorded. The parameter information of the files to be migrated includes at least the hash value.

[0146] Based on the traversal scan order, preset read length, and number of concurrent threads, at least one thread is started and reads the data of the files to be migrated from the source remote network file sharing system through the first client library.

[0147] In a preferred embodiment of the present invention, the first data migration module is further configured to:

[0148] In response to the completion of data reading of the files to be migrated, the metadata information of the files to be migrated in the source remote network file sharing system is read through the first client library.

[0149] In a preferred embodiment of the present invention, the second data migration module is specifically used for:

[0150] Based on the file system identifier of the destination storage, determine the file system of the destination storage, and detect whether the target directory exists in the file system according to the write directory corresponding to the file system identifier of the destination storage.

[0151] In response to the existence of the file, the data to be migrated is stored in the corresponding target directory of the file system through the second client library;

[0152] If the target directory does not exist, the target directory is created, and after creation, the data of the files to be migrated is stored in the corresponding target directory in the file system through the second client library;

[0153] In response to the fact that the data of the file to be migrated has been stored in the corresponding target directory in the file system, the first client library reads the metadata information of the file to be migrated in the remote network file sharing system of the source end, and the second client library stores the metadata information of the file to be migrated in the corresponding target directory in the file system.

[0154] In a preferred embodiment of the present invention, the device further includes a verification module, which is specifically used for:

[0155] Based on the data migration request, determine whether consistency verification is required for the target files stored in the destination storage file system.

[0156] In response to the need to perform consistency checks on the target file, calculate the hash value of the target file;

[0157] If the hash value of the target file is successfully compared with the hash value of the file to be migrated, the target file is determined to have passed the consistency check.

[0158] If the hash value of the target file fails to match the hash value of the file to be migrated, it is determined that the target file has not passed the consistency check, and the data migration operation is re-executed.

[0159] Specific limitations regarding the data migration device can be found in the limitations of the data migration method described above, and will not be repeated here. Each module in the aforementioned data migration device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in the computer device in hardware form, or stored in the memory of the computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0160] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 6 As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with external terminals via a network connection. When the computer program is executed by the processor, it implements a data migration method. The display screen can be an LCD screen or an e-ink screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad mounted on the computer device casing, or an external keyboard, touchpad, or mouse.

[0161] Those skilled in the art will understand that Figure 6 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0162] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to perform the following steps:

[0163] S1: Receive data migration requests.

[0164] S2: Based on the data migration request, determine the data migration task parameters, which must include at least the number of concurrent threads;

[0165] S3: Based on the number of concurrent threads, start at least one thread and read the data and metadata information of the file to be migrated in the source remote network file sharing system through the first client library. The first client library is used to directly access the source remote network file sharing system through the first programming interface.

[0166] S4: In response to the completion of reading, the data and metadata information of the file to be migrated are stored in the file system of the destination storage through the second client library. The second client library is used to directly access the file system of the destination storage through the second programming interface.

[0167] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0168] Parse the data migration request and determine the data migration task parameters based on the parsing results. The data migration task parameters include at least the shared address of the remote network file sharing system, the file system identifier of the destination storage end and its corresponding write directory, and the data migration method.

[0169] The methods for determining the number of concurrent threads include:

[0170] Obtain the metadata information corresponding to the target file to be migrated;

[0171] Based on the metadata information corresponding to the target file to be migrated, determine the space occupied by the target file to be migrated;

[0172] In response to the fact that the space occupied by the target file to be migrated is less than or equal to the first preset threshold, the number of concurrent threads is determined to be single thread;

[0173] In response to the target file to be migrated occupying more space than a first preset threshold, a preset read length is obtained, and the target file to be migrated is split into multiple fragments according to the preset read length;

[0174] Based on the number of shards in the multiple shards and the number of files to be migrated, determine the number of concurrent threads corresponding to the target files to be migrated;

[0175] Based on the number of threads corresponding to multiple target files to be migrated, the total number of threads is determined, and the total number of threads is less than or equal to the number of threads in the thread pool.

[0176] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0177] Integrate the first client library and the second client library on the destination storage side;

[0178] Based on the shared address of the remote network file sharing system, the first client library is used to connect to and mount the remote network file sharing system.

[0179] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0180] Based on the first client library, the files and directories to be migrated in the remote network file sharing system of the source end are traversed and scanned, and the parameter information of the files to be migrated determined according to the traversal and scan results is recorded. The parameter information of the files to be migrated includes at least the hash value.

[0181] Based on the traversal scan order, preset read length, and number of concurrent threads, at least one thread is started and reads the data of the files to be migrated from the source remote network file sharing system through the first client library.

[0182] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0183] In response to the completion of data reading of the files to be migrated, the metadata information of the files to be migrated in the source remote network file sharing system is read through the first client library.

[0184] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0185] Based on the file system identifier of the destination storage, determine the file system of the destination storage, and detect whether the target directory exists in the file system according to the write directory corresponding to the file system identifier of the destination storage.

[0186] In response to the existence of the file, the data to be migrated is stored in the corresponding target directory of the file system through the second client library;

[0187] If the target directory does not exist, the target directory is created, and after creation, the data of the files to be migrated is stored in the corresponding target directory in the file system through the second client library;

[0188] In response to the fact that the data of the file to be migrated has been stored in the corresponding target directory in the file system, the first client library reads the metadata information of the file to be migrated in the remote network file sharing system of the source end, and the second client library stores the metadata information of the file to be migrated in the corresponding target directory in the file system.

[0189] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0190] Based on the data migration request, determine whether consistency verification is required for the target files stored in the destination storage file system.

[0191] In response to the need to perform consistency checks on the target file, calculate the hash value of the target file;

[0192] If the hash value of the target file is successfully compared with the hash value of the file to be migrated, the target file is determined to have passed the consistency check.

[0193] If the hash value of the target file fails to match the hash value of the file to be migrated, it is determined that the target file has not passed the consistency check, and the data migration operation is re-executed.

[0194] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor:

[0195] S1: Receive data migration requests.

[0196] S2: Based on the data migration request, determine the data migration task parameters, which must include at least the number of concurrent threads;

[0197] S3: Based on the number of concurrent threads, start at least one thread and read the data and metadata information of the file to be migrated in the source remote network file sharing system through the first client library. The first client library is used to directly access the source remote network file sharing system through the first programming interface.

[0198] S4: In response to the completion of reading, the data and metadata information of the file to be migrated are stored in the file system of the destination storage through the second client library. The second client library is used to directly access the file system of the destination storage through the second programming interface.

[0199] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0200] Parse the data migration request and determine the data migration task parameters based on the parsing results. The data migration task parameters include at least the shared address of the remote network file sharing system, the file system identifier of the destination storage end and its corresponding write directory, and the data migration method.

[0201] The methods for determining the number of concurrent threads include:

[0202] Obtain the metadata information corresponding to the target file to be migrated;

[0203] Based on the metadata information corresponding to the target file to be migrated, determine the space occupied by the target file to be migrated;

[0204] In response to the fact that the space occupied by the target file to be migrated is less than or equal to the first preset threshold, the number of concurrent threads is determined to be single thread;

[0205] In response to the target file to be migrated occupying more space than a first preset threshold, a preset read length is obtained, and the target file to be migrated is split into multiple fragments according to the preset read length;

[0206] Based on the number of shards in the multiple shards and the number of files to be migrated, determine the number of concurrent threads corresponding to the target files to be migrated;

[0207] Based on the number of threads corresponding to multiple target files to be migrated, the total number of threads is determined, and the total number of threads is less than or equal to the number of threads in the thread pool.

[0208] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0209] Integrate the first client library and the second client library on the destination storage side;

[0210] Based on the shared address of the remote network file sharing system, the first client library is used to connect to and mount the remote network file sharing system.

[0211] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0212] Based on the first client library, the files and directories to be migrated in the remote network file sharing system of the source end are traversed and scanned, and the parameter information of the files to be migrated determined according to the traversal and scan results is recorded. The parameter information of the files to be migrated includes at least the hash value.

[0213] Based on the traversal scan order, preset read length, and number of concurrent threads, at least one thread is started and reads the data of the files to be migrated from the source remote network file sharing system through the first client library.

[0214] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0215] In response to the completion of data reading of the files to be migrated, the metadata information of the files to be migrated in the source remote network file sharing system is read through the first client library.

[0216] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0217] Based on the file system identifier of the destination storage, determine the file system of the destination storage, and detect whether the target directory exists in the file system according to the write directory corresponding to the file system identifier of the destination storage.

[0218] In response to the existence of the file, the data to be migrated is stored in the corresponding target directory of the file system through the second client library;

[0219] If the target directory does not exist, the target directory is created, and after creation, the data of the files to be migrated is stored in the corresponding target directory in the file system through the second client library;

[0220] In response to the fact that the data of the file to be migrated has been stored in the corresponding target directory in the file system, the first client library reads the metadata information of the file to be migrated in the remote network file sharing system of the source end, and the second client library stores the metadata information of the file to be migrated in the corresponding target directory in the file system.

[0221] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0222] Based on the data migration request, determine whether consistency verification is required for the target files stored in the destination storage file system.

[0223] In response to the need to perform consistency checks on the target file, calculate the hash value of the target file;

[0224] If the hash value of the target file is successfully compared with the hash value of the file to be migrated, the target file is determined to have passed the consistency check.

[0225] If the hash value of the target file fails to match the hash value of the file to be migrated, it is determined that the target file has not passed the consistency check, and the data migration operation is re-executed.

[0226] In one embodiment, a computer program product is provided, the computer program product comprising a computer program that, when executed by a processor, performs the following steps:

[0227] S1: Receive data migration requests.

[0228] S2: Based on the data migration request, determine the data migration task parameters, which must include at least the number of concurrent threads;

[0229] S3: Based on the number of concurrent threads, start at least one thread and read the data and metadata information of the file to be migrated in the source remote network file sharing system through the first client library. The first client library is used to directly access the source remote network file sharing system through the first programming interface.

[0230] S4: In response to the completion of reading, the data and metadata information of the file to be migrated are stored in the file system of the destination storage through the second client library. The second client library is used to directly access the file system of the destination storage through the second programming interface.

[0231] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0232] Parse the data migration request and determine the data migration task parameters based on the parsing results. The data migration task parameters include at least the shared address of the remote network file sharing system, the file system identifier of the destination storage end and its corresponding write directory, and the data migration method.

[0233] The methods for determining the number of concurrent threads include:

[0234] Obtain the metadata information corresponding to the target file to be migrated;

[0235] Based on the metadata information corresponding to the target file to be migrated, determine the space occupied by the target file to be migrated;

[0236] In response to the fact that the space occupied by the target file to be migrated is less than or equal to the first preset threshold, the number of concurrent threads is determined to be single thread;

[0237] In response to the target file to be migrated occupying more space than a first preset threshold, a preset read length is obtained, and the target file to be migrated is split into multiple fragments according to the preset read length;

[0238] Based on the number of shards in the multiple shards and the number of files to be migrated, determine the number of concurrent threads corresponding to the target files to be migrated;

[0239] Based on the number of threads corresponding to multiple target files to be migrated, the total number of threads is determined, and the total number of threads is less than or equal to the number of threads in the thread pool.

[0240] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0241] Integrate the first client library and the second client library on the destination storage side;

[0242] Based on the shared address of the remote network file sharing system, the first client library is used to connect to and mount the remote network file sharing system.

[0243] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0244] Based on the first client library, the files and directories to be migrated in the remote network file sharing system of the source end are traversed and scanned, and the parameter information of the files to be migrated determined according to the traversal and scan results is recorded. The parameter information of the files to be migrated includes at least the hash value.

[0245] Based on the traversal scan order, preset read length, and number of concurrent threads, at least one thread is started and reads the data of the files to be migrated from the source remote network file sharing system through the first client library.

[0246] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0247] In response to the completion of data reading of the files to be migrated, the metadata information of the files to be migrated in the source remote network file sharing system is read through the first client library.

[0248] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0249] Based on the file system identifier of the destination storage, determine the file system of the destination storage, and detect whether the target directory exists in the file system according to the write directory corresponding to the file system identifier of the destination storage.

[0250] In response to the existence of the file, the data to be migrated is stored in the corresponding target directory of the file system through the second client library;

[0251] If the target directory does not exist, the target directory is created, and after creation, the data of the files to be migrated is stored in the corresponding target directory in the file system through the second client library;

[0252] In response to the fact that the data of the file to be migrated has been stored in the corresponding target directory in the file system, the first client library reads the metadata information of the file to be migrated in the remote network file sharing system of the source end, and the second client library stores the metadata information of the file to be migrated in the corresponding target directory in the file system.

[0253] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0254] Based on the data migration request, determine whether consistency verification is required for the target files stored in the destination storage file system.

[0255] In response to the need to perform consistency checks on the target file, calculate the hash value of the target file;

[0256] If the hash value of the target file is successfully compared with the hash value of the file to be migrated, the target file is determined to have passed the consistency check.

[0257] If the hash value of the target file fails to match the hash value of the file to be migrated, it is determined that the target file has not passed the consistency check, and the data migration operation is re-executed.

[0258] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0259] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0260] The embodiments described above are merely examples of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application.

Claims

1. A data migration method, characterized in that, The method includes: Receive data migration requests; Based on the data migration request, determine the data migration task parameters, which include at least the number of concurrent threads, the shared address of the remote network file sharing system, the file system identifier of the destination storage end and its corresponding write directory; Based on the number of concurrent threads, at least one thread is started and the data and metadata information of the file to be migrated in the source remote network file sharing system are read through the first client library. The first client library is used to directly access the source remote network file sharing system through the first programming interface. In response to the completion of reading, the data and metadata information of the file to be migrated are stored in the file system of the destination storage terminal through the second client library. The second client library is used to directly access the file system of the destination storage terminal through the second programming interface. After determining the data migration task parameters based on the data migration request, the method further includes: The first client library and the second client library are integrated on the destination storage end. The first client library refers to the client development library that directly accesses files in the NFS share through a programming interface, and the second client library refers to the client library that directly accesses the file system through a programming interface. Based on the shared address of the remote network file sharing system, the first client library is used to connect to and mount the remote network file sharing system. In response to the completion of reading the data of the file to be migrated, the method further includes: Based on the file system identifier of the destination storage end, determine the file system of the destination storage end, and detect whether the target directory exists in the file system according to the write directory corresponding to the file system identifier of the destination storage end. In response to the non-existence of the target directory, a target directory is created, and after creation, the data of the files to be migrated is stored in the corresponding target directory in the file system through the second client library; In response to the fact that the data of the file to be migrated has been stored in the corresponding target directory in the file system, the metadata information of the file to be migrated in the remote network file sharing system of the source end is read through the first client library, and the metadata information of the file to be migrated is stored in the corresponding target directory in the file system through the second client library.

2. The data migration method according to claim 1, characterized in that, Based on the data migration request, the data migration task parameters are determined as follows: The data migration request is parsed, and the data migration task parameters are determined based on the parsing result of the data migration request. The data migration task parameters include at least the data migration method. The method for determining the number of concurrent threads includes: Obtain the metadata information corresponding to the target file to be migrated; Based on the metadata information corresponding to the target file to be migrated, determine the space occupied by the target file to be migrated; In response to the fact that the space occupied by the target file to be migrated is less than or equal to a first preset threshold, the number of concurrent threads is determined to be single thread; In response to the target file to be migrated occupying more space than a first preset threshold, a preset reading length is obtained, and the target file to be migrated is split into multiple fragments according to the preset reading length; Based on the number of shards and the number of files to be migrated, determine the number of concurrent threads corresponding to the target file to be migrated; Based on the number of concurrent threads corresponding to multiple target files to be migrated, the total number of concurrent threads is determined, wherein the total number of concurrent threads is less than or equal to the number of threads in the thread pool.

3. The data migration method according to claim 2, characterized in that, Based on the stated number of concurrent threads, starting at least one thread and reading the data and metadata information of the files to be migrated from the source remote network file sharing system through the first client library includes: Based on the first client library, the files and directories to be migrated in the source remote network file sharing system are traversed and scanned, and the parameter information of the files to be migrated determined according to the traversal and scan results is recorded. The parameter information of the files to be migrated includes at least a hash value. Based on the traversal scan order, the preset read length, and the number of concurrent threads, at least one thread is started and reads the data of the file to be migrated in the source remote network file sharing system through the first client library.

4. The data migration method according to claim 3, characterized in that, Based on the stated number of concurrent threads, starting at least one thread and reading the data and metadata information of the files to be migrated from the source remote network file sharing system through the first client library also includes: In response to the completion of data reading of the files to be migrated, the metadata information of the files to be migrated in the source remote network file sharing system is read through the first client library.

5. The data migration method according to claim 1 or 4, characterized in that, In response to the completion of reading the data of the file to be migrated, the method further includes: Based on the file system identifier of the destination storage end, determine the file system of the destination storage end, and detect whether the target directory exists in the file system according to the write directory corresponding to the file system identifier of the destination storage end. In response to the existence of the file, the data to be migrated is stored in the corresponding target directory of the file system through the second client library; In response to the fact that the data of the file to be migrated has been stored in the corresponding target directory in the file system, the metadata information of the file to be migrated in the remote network file sharing system of the source end is read through the first client library, and the metadata information of the file to be migrated is stored in the corresponding target directory in the file system through the second client library.

6. The data migration method according to claim 1, characterized in that, After storing the data and metadata information of the files to be migrated to the file system of the destination storage terminal through the second client library, the method further includes: Based on the data migration request, determine whether consistency verification is required for the target files stored in the destination storage file system. In response to the need to perform consistency checks on the target file, calculate the hash value of the target file; If the hash value of the target file is successfully compared with the hash value of the file to be migrated, it is determined that the target file has passed the consistency check. If the hash value of the target file fails to match the hash value of the file to be migrated, it is determined that the target file has failed the consistency check, and the data migration operation is re-executed.

7. A data migration apparatus for implementing the data migration method as described in claim 1, characterized in that, The device includes: The request receiving module is used to receive data migration requests; The parameter determination module is used to determine the data migration task parameters based on the data migration request, wherein the data migration task parameters include at least the number of concurrent threads; The first data migration module is used to start at least one thread according to the number of concurrent threads and read the data and metadata information of the file to be migrated in the source remote network file sharing system through the first client library. The first client library is used to directly access the source remote network file sharing system through the first programming interface. The second data migration module is used to store the data and metadata information of the file to be migrated to the file system of the destination storage terminal through the second client library in response to the completion of reading. The second client library is used to directly access the file system of the destination storage terminal through the second programming interface.

8. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method of any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 6.

Images