Data migration method and device, computer equipment and storage medium

Abstract

The invention relates to a data migration method and device, computer equipment and a storage medium, and belongs to the technical field of distributed storage. The method comprises the steps that data migration task parameters are determined according to a received data migration request, and the data migration task parameters at least comprise the thread concurrence number; according to the thread concurrent number, starting at least one thread and reading data and metadata information of a file to be migrated in the source end remote network file sharing system through the first client library; and in response to reading completion, storing data and metadata information of the to-be-migrated file to a file system of a target storage end through the second client library. On the basis of ensuring the integrity and consistency of the file data and the metadata, the system resource occupation amount is reduced, and the data transmission efficiency is improved.

Description

Technical Field

[0001] The present application relates to the field of distributed storage technology, and in particular to a data migration method, device, computer equipment and storage medium. Background Art

[0002] With the rapid development of big data and cloud computing, data storage and migration has become a crucial issue. As a high-performance, highly scalable distributed storage system, Ceph (distributed storage system) file system is widely used in the storage and management of large-scale data. However, in practical applications, there are still many problems in how to efficiently and reliably migrate file system data from remote to local Ceph storage file system. Traditional data migration methods require the deployment of third-party software such as rsync (open source file synchronization tool), which usually faces problems such as low transmission efficiency, excessive system resource usage, inability to flexibly customize metadata synchronization, and inability to ensure data consistency. Summary of the invention

[0003] Based on this, it is necessary to provide a data migration method, device, computer equipment and storage medium that can improve transmission efficiency and reduce resource usage in response to the above technical problems.

[0004] In a first aspect, a data migration method is provided, the method comprising:

[0005] Receive data migration requests;

[0006] Determine data migration task parameters according to the data migration request, where the data migration task parameters at least include the number of concurrent threads;

[0007] According to the number of concurrent threads, at least one thread is started and data and metadata information of the file to be migrated in the remote network file sharing system at the source end is read through the first client library, where the first client library is used to directly access the remote network file sharing system at the source end through the first programming interface;

[0008] In response to the reading completion, the data and metadata information of the file to be migrated are stored in the file system of the destination storage end through the second client library, and the second client library is used to directly access the file system of the destination storage end through the second programming interface.

[0009] Optionally, according to the data migration request, determining the data migration task parameters includes:

[0010] Parsing the data migration request, and determining the data migration task parameters according to the parsing result of the data migration request, wherein the data migration task parameters at least include the sharing address of the remote network file sharing system, the file system identifier of the destination storage end and its corresponding writing directory and the data migration method;

[0011] The method for determining the number of concurrent threads includes:

[0012] Obtain metadata information corresponding to the target file to be migrated;

[0013] Determine the space occupied by the target file to be migrated according to the metadata information corresponding to the target file to be migrated;

[0014] In response to the space occupied by the target to-be-migrated file being less than or equal to a first preset threshold, determining that the number of concurrent threads is a single thread;

[0015] In response to the space occupied by the target file to be migrated being greater than a first preset threshold, obtaining a preset read length, and splitting the target file to be migrated according to the preset read length to obtain a plurality of fragments;

[0016] Determine the number of concurrent threads corresponding to the target files to be migrated according to the number of shards of the multiple shards and the number of files to be migrated;

[0017] Based on the concurrent numbers of threads corresponding to the multiple target files to be migrated, the total concurrent number of threads is determined, and the total concurrent number of threads is less than or equal to the number of threads in the thread pool.

[0018] Optionally, after determining the data migration task parameters according to the data migration request, the method further includes:

[0019] Integrate the first client library and the second client library at the destination storage end;

[0020] Based on the shared address of the remote network file sharing system, the remote network file sharing system is connected and mounted using the first client library.

[0021] Optionally, starting at least one thread and reading data and metadata information of the file to be migrated in the source-end remote network file sharing system through the first client library according to the number of concurrent threads includes:

[0022] Based on the first client library, traverse and scan the files and directories to be migrated in the remote network file sharing system at the source end, and record parameter information of the files to be migrated determined according to the traversal and scanning results, where the parameter information of the files to be migrated includes at least a hash value;

[0023] According to the traversal scanning order, the preset reading length and the number of concurrent threads, at least one thread is started and the data of the file to be migrated in the source-end remote network file sharing system is read through the first client library.

[0024] Optionally, starting at least one thread and reading data and metadata information of the to-be-migrated file in the source-end remote network file sharing system through the first client library according to the concurrent number of threads also includes:

[0025] In response to the data of the file to be migrated being read completely, metadata information of the file to be migrated in the source-end remote network file sharing system is read through the first client library.

[0026] Optionally, in response to the data of the file to be migrated being read, the method further includes:

[0027] Based on the file system identifier of the destination storage end, determine the file system of the destination storage end, and detect whether the target directory exists in the file system according to the write directory corresponding to the file system identifier of the destination storage end;

[0028] In response to existence, storing the data of the file to be migrated in a corresponding target directory in the file system through the second client library;

[0029] In response to the non-existence of the target directory, create a target directory, and after the creation is completed, store the data of the file to be migrated to the corresponding target directory in the file system through the second client library;

[0030] In response to the data of the file to be migrated having been stored in the corresponding target directory in the file system, the metadata information of the file to be migrated in the source remote network file sharing system is read through the first client library, and the metadata information of the file to be migrated is stored in the corresponding target directory in the file system through the second client library.

[0031] Optionally, after storing the data and metadata information of the file to be migrated in the file system of the destination storage end through the second client library, the method further includes:

[0032] Based on the data migration request, determine whether it is necessary to perform consistency check on the target file stored in the target storage end file system;

[0033] In response to the need to perform consistency check on the target file, calculating a hash value of the target file;

[0034] In response to a successful comparison between the hash value of the target file and the hash value of the file to be migrated, determining that the target file passes the consistency check;

[0035] In response to a failure in comparing the hash value of the target file with the hash value of the file to be migrated, it is determined that the target file fails the consistency check, and the data migration operation is re-executed.

[0036] In a second aspect, a data migration device is provided, the device comprising:

[0037] A request receiving module, used for receiving a data migration request;

[0038] A parameter determination module, used to determine the data migration task parameters according to the data migration request, where the data migration task parameters at least include the number of concurrent threads;

[0039] A first data migration module is used to start at least one thread according to the number of concurrent threads, and read the data and metadata information of the file to be migrated in the source-end remote network file sharing system through a first client library, and the first client library is used to directly access the source-end remote network file sharing system through a first programming interface;

[0040] The second data migration module is used to store the data and metadata information of the file to be migrated to the file system of the destination storage end through the second client library in response to the completion of the reading. The second client library is used to directly access the file system of the destination storage end through the second programming interface.

[0041] In a third aspect, a computer device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the following steps are implemented:

[0042] Receive data migration requests;

[0043] Determine data migration task parameters according to the data migration request, where the data migration task parameters at least include the number of concurrent threads;

[0044] According to the number of concurrent threads, at least one thread is started and data and metadata information of the file to be migrated in the remote network file sharing system at the source end is read through the first client library, where the first client library is used to directly access the remote network file sharing system at the source end through the first programming interface;

[0045] In response to the reading completion, the data and metadata information of the file to be migrated are stored in the file system of the destination storage end through the second client library, and the second client library is used to directly access the file system of the destination storage end through the second programming interface.

[0046] In a fourth aspect, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0047] Receive data migration requests;

[0048] Determine data migration task parameters according to the data migration request, where the data migration task parameters at least include the number of concurrent threads;

[0049] According to the number of concurrent threads, at least one thread is started and data and metadata information of the file to be migrated in the remote network file sharing system at the source end is read through the first client library, where the first client library is used to directly access the remote network file sharing system at the source end through the first programming interface;

[0050] In response to the reading completion, the data and metadata information of the file to be migrated are stored in the file system of the destination storage end through the second client library, and the second client library is used to directly access the file system of the destination storage end through the second programming interface.

[0051] In a fifth aspect, a computer program product is provided, the computer program product comprising a computer program, and when the computer program is executed by a processor, the following steps are implemented:

[0052] Receive data migration requests;

[0053] Determine data migration task parameters according to the data migration request, where the data migration task parameters at least include the number of concurrent threads;

[0054] According to the number of concurrent threads, at least one thread is started and data and metadata information of the file to be migrated in the remote network file sharing system at the source end is read through the first client library, where the first client library is used to directly access the remote network file sharing system at the source end through the first programming interface;

[0055] In response to the reading completion, the data and metadata information of the file to be migrated are stored in the file system of the destination storage end through the second client library, and the second client library is used to directly access the file system of the destination storage end through the second programming interface.

[0056] The above-mentioned data migration method, device, computer equipment and storage medium, the method includes: receiving a data migration request; determining data migration task parameters according to the data migration request, the data migration task parameters at least including the number of concurrent threads; starting at least one thread according to the number of concurrent threads and reading the data and metadata information of the file to be migrated in the remote network file sharing system of the source end through the first client library, the first client library is used to directly access the remote network file sharing system of the source end through the first programming interface; in response to the completion of the reading, the data and metadata information of the file to be migrated are stored in the file system of the destination storage end through the second client library, the second client library is used to directly access the file system of the destination storage end through the second programming interface. The present application can realize the mounting and data reading of the remote network file sharing system by using the first client library and the second client library, and migrate the file data and metadata information through multiple threads. On the basis of ensuring the complete consistency of the file data and metadata, the overhead of data migration based on the POSIX file system is reduced, the system resource usage is reduced, and the data transmission efficiency is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 An application environment diagram of a data migration method in an embodiment;

[0058] Figure 2A schematic diagram of a data migration method in one embodiment;

[0059] Figure 3 A schematic diagram of a file migration sequence of a data migration method in an embodiment;

[0060] Figure 4 A schematic diagram of a file migration module of a data migration method in an embodiment;

[0061] Figure 5 is a structural block diagram of a data migration device in one embodiment;

[0062] Figure 6 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0063] In order to make the purpose, technical solutions and advantages of this application clearer, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0064] It should be understood that in the description of the present application, unless the context clearly requires otherwise, words such as "include", "comprises", and the like throughout the specification should be interpreted as including rather than being exclusive or exhaustive; that is, as including but not limited to.

[0065] It should also be understood that the terms "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or implying relative importance. In addition, in the description of this application, unless otherwise specified, "plurality" means two or more.

[0066] It should be noted that the terms "S1", "S2", etc. are only used for the purpose of describing the steps, and do not specifically refer to the order or sequence, nor are they used to limit the present application. They are only for the convenience of describing the method of the present application, and cannot be understood as indicating the order of the steps. In addition, the technical solutions between the various embodiments can be combined with each other, but they must be based on the ability of ordinary technicians in this field to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection required by the present application.

[0067] According to the background technology, it can be known that traditional data migration methods have many problems. For example, in order to migrate data from any source storage device file system to the destination storage, the relevant technology needs to configure NFS sharing (remote network file sharing system) in the source data storage file system, and then create a new system directory on the destination storage end, mount the NFS sharing exported from the source data storage to this directory, use third-party file synchronization and backup tools such as rsync to configure the required parameters, and manually or time-triggered synchronization can be used to copy data from the NFS sharing directory to the specified file system directory of the destination storage. Among them, configuring NFS sharing on the source data file system end is to ensure that the source data can be accessed, read and written by other storage through the network. This is the premise of all data migration functions. Create a directory in the destination storage system, use the system mount command to mount the source NFS sharing to this directory, which can be done through This directory accesses the data under the source file system. The system integrates the rsync file synchronization tool, configures the directory where the source file system is mounted as the source address, and configures the local file system in the destination storage as the destination address. The rsync tool is executed manually or periodically, and it performs data replication and metadata synchronization under the two directories. The existing problems include the need to create a directory on the destination storage to mount the source NFS share, and the need for the destination storage Ceph file system to be visible to the outside world to ensure that the synchronization tool rsync can access data on both ends at the same time, which increases the operation steps. rsync consumes more CPU (central processing unit) and IO (input / output) resources, which will affect other services on the destination storage. In multi-user or multi-task scenarios, if multiple processes try to modify the same file at the same time, rsync has no native locking mechanism to avoid conflicts, which may cause inconsistent data after migration.

[0068] To solve the above technical problems, the present application provides a data migration method, apparatus, computer equipment and storage medium. By using a first client library and a second client library, the mounting and data reading of a remote network file sharing system can be realized, and file data and metadata information can be migrated through multi-threading. On the basis of ensuring the complete consistency of file data and metadata, the overhead of data migration based on the POSIX file system is reduced, the system resource usage is reduced, and the data transmission efficiency is improved.

[0069] The data migration method provided in this application can be applied to Figure 1 In the application environment shown, the terminal 102 communicates with the data processing platform set on the server 104 through the network, wherein the terminal 102 can be but not limited to various personal computers, laptops, smart phones, tablet computers and portable wearable devices, and the server 104 can be implemented by an independent server or a server cluster composed of multiple servers.

[0070] In one embodiment, Figure 2 As shown, a data migration method is provided, which is applied to Figure 1 The terminal in is used as an example to illustrate, including the following steps:

[0071] S1: Receive a data migration request.

[0072] It should be noted that the data migration request is a request issued by a user in real time or a request issued by the system at a scheduled time according to actual needs, and the data migration request includes data migration task parameters.

[0073] S2: Determine data migration task parameters according to the data migration request, where the data migration task parameters at least include the number of concurrent threads.

[0074] It should be noted that the data migration task parameters can be obtained by parsing the data migration request. The data migration task parameters are specifically: (1) the NFS (remote network file sharing system) sharing address provided by the source storage, which includes the IP (Internet Protocol address) and the sharing path; (2) the file system and directory selected on the destination storage to determine the write directory for data migration; (3) the data migration method selected according to the corresponding migration task in different scenarios, including full migration and incremental migration; (4) the metadata to be synchronized, such as file group, file permissions, etc.; (5) whether it is necessary to verify the consistency of the file. If verification is required, after the migration of the file to be migrated is completed, the md5 value (hash value) of the source file and the destination file needs to be verified. If the verification fails, the migration will be attempted again. Because the md5 calculation of the file requires additional CPU resources, the duration of the migration task is extended. Therefore, it can be determined whether the data consistency verification operation needs to be performed based on the sensitivity to the migration duration. Among them, the method for determining the sensitivity of the migration duration includes:

[0075] Obtain an identifier of the target operating system and the duration of the migration in multiple different time periods, where the time period can be set according to actual needs, such as one week;

[0076] Obtain the migration durations corresponding to multiple target time periods in different time periods, and determine the fluctuation coefficient of the migration duration. The target time period can be set according to actual needs, such as a day of a week, and the fluctuation coefficient is obtained by subtracting the minimum value from the maximum value of the migration duration in the time period.

[0077] In response to the fluctuation coefficient being less than a first preset value, determining whether a consistency check operation is performed on the file in the target time period, and determining the number of times of execution, wherein the first preset value can be set according to actual needs;

[0078] In response to the fact that a consistency check operation is performed on the file in the target time period and the number of times the consistency check operation is performed is greater than a second preset value, determining that the migration duration sensitivity of the target time period is relatively weak, wherein the second preset value can be set according to actual needs;

[0079] In response to the fact that no consistency check operation is performed on the file in the target time period and / or the number of executions is less than or equal to a second preset value, determining that the migration duration sensitivity of the target time period is relatively strong;

[0080] In response to the migration duration sensitivity being relatively weak, a data consistency check operation is performed in a target time period within a next time period; in response to the migration duration sensitivity being relatively strong, a data consistency check operation is not performed in the target time period within the next time period.

[0081] In the above implementation, whether to perform a data consistency check operation is determined by the sensitivity of the migration duration, so that the reliability of data migration can be improved.

[0082] In some specific implementations, determining the data migration task parameters according to the data migration request includes:

[0083] Parsing the data migration request, and determining the data migration task parameters according to the parsing result of the data migration request, wherein the data migration task parameters at least include the sharing address of the remote network file sharing system, the file system identifier of the destination storage end and its corresponding writing directory and the data migration method;

[0084] The method for determining the number of concurrent threads includes:

[0085] Obtain metadata information corresponding to the target file to be migrated, where the metadata information to be used here includes the size information of the target file to be migrated, such as 10 MiB;

[0086] Determine the space occupied by the target file to be migrated, that is, the storage capacity occupied by the target file to be migrated, according to the metadata information corresponding to the target file to be migrated;

[0087] In response to the space occupied by the target file to be migrated being less than or equal to a first preset threshold, determining that the number of concurrent threads is a single thread, wherein the first preset threshold can be set according to actual needs, such as 10 MiB, that is, when the space occupied by the target file to be migrated is less than or equal to 10 MiB, data of the target file to be migrated is read through a single thread;

[0088] In response to the space occupied by the target file to be migrated being greater than a first preset threshold, a preset read length is obtained, and the target file to be migrated is split according to the preset read length to obtain a plurality of fragments, wherein the preset read length can be set according to actual needs, such as 1 MiB, that is, when the space occupied by the target file to be migrated is greater than 10 MiB, the target file to be migrated is split into a plurality of fragments of 1 MiB in size;

[0089] According to the number of shards of the multiple shards and the number of files to be migrated, determine the number of concurrent threads corresponding to the target file to be migrated. Specifically, the number of initial threads can be determined according to the number of files to be migrated, that is, one thread corresponds to one file to be migrated. At this time, it is necessary to obtain the number of remaining threads in the thread pool. If there are remaining threads, the remaining threads are evenly distributed according to the number of shards of the multiple shards. For example, if the number of shards of the first file to be migrated is 20 and the number of shards of the second file to be migrated is 30, then 1 remaining thread is allocated to the first file to be migrated, and 2 remaining threads are allocated to the second file to be migrated, and so on. No further details are given.

[0090] Based on the concurrent number of threads corresponding to multiple target files to be migrated, the total concurrent number of threads is determined, and the total concurrent number of threads is less than or equal to the number of threads in the thread pool. In order to reduce the system resource overhead generated by frequent creation and destruction of threads, a thread pool containing a fixed number of threads is created in advance. When a new thread is added, resources are directly taken from the thread pool. After the thread task is completed, there is no need to release resources. When the main process initiates a file open and read request, all tasks belonging to the same NFS connection are added to a queue, and the tasks in the queue are distributed to the threads in the thread pool for execution. When the connection between the first client library and the remote network file sharing system is disconnected, the thread pool is destroyed. In particular, multiple threads can execute multiple migration tasks concurrently, that is, each migration task starts a thread Thread to run, and each does not affect each other. Each task thread is kept running while occupying as few system resources as possible until all file migrations are completed, the task ends automatically, and the resources are released.

[0091] Specifically, the thread pool creation method includes: defining a thread pool thread_pool structure, which includes a thread array threads, a thread number thread_count, a task queue task_queue and a thread pool running status flag running_flag, the thread pool takes tasks from the task queue task_queue for execution, initializes the thread pool to determine a specified number of threads, and stores them in the thread array threads, and in the subsequent file migration process, stores all received requests in the task queue and distributes them to multiple threads for execution by the thread pool, wherein the task queue task_queue is a global task queue task_queue structure newly added in libnfs, and the queue includes: head and tail pointers of the task list to be executed, which are used to traverse tasks; queue mutex thread locks and conditional variables to ensure the consistency of task queue access; task execution status task_status, which is used to identify the task in execution, success or failure.

[0092] In some specific embodiments, Figure 3 As shown, after determining the data migration task parameters according to the data migration request, the method further includes:

[0093] Integrate a first client library and a second client library at the destination storage end, wherein the first client library refers to libnfs, which is a client development library for accessing NFS shares through the network, and can realize operations such as mounting, reading and writing NFS shares, allowing applications to directly access files in NFS shares through programming interfaces without explicitly mounting NFS shares to system directories, and the second client library refers to libClient (Ceph Client Library), which is a client library for interacting with a Ceph cluster, and allows applications to directly access the Ceph file system through programming interfaces without the need to implement file data migration through a POSIX file system interface;

[0094] Based on the shared address of the remote network file sharing system, the first client library is used to connect and mount the remote network file sharing system. That is, before the migration task starts, the libnfs library and the libClient library are integrated in the main process of the destination storage. According to the input IP address and shared path of the source NFS share, the nfs_mount interface of the libnfs library is called to connect and mount the remote NFS share. After the mount is successful, all files on the source side can be directly accessed. Specifically, libnfs, as a client library, can efficiently implement NFS protocol communication, and support access to files mounted on the source NFS in the system memory of the destination side without creating a directory in the destination system, thereby saving resource overhead at the POSIX (Portable Operating System Interface) layer.

[0095] In the above implementation, by setting multiple task processing threads, the number of request processing per unit time can be increased. By using the libnfs library and libClient library to realize the mounting and data reading of remote NFS sharing, the system resource usage is reduced on the basis of improving file transfer efficiency.

[0096] S3: According to the number of concurrent threads, at least one thread is started and data and metadata information of the files to be migrated in the source remote network file sharing system are read through a first client library, where the first client library is used to directly access the source remote network file sharing system through a first programming interface.

[0097] It should be noted that, from step S2, it can be seen that the number of concurrent threads is determined by the number of threads in the thread pool and the size of the file to be migrated; the migration of files includes two major parts, namely, the migration of file content (data) and file attributes (metadata information); the first programming interface refers to a specification or protocol defined in programming that specifies the method for classes or components to communicate with each other, which is used to specify the communication method between the first client library and the source remote network file sharing system.

[0098] In some specific implementations, starting at least one thread and reading data and metadata information of a file to be migrated in a remote network file sharing system at a source end through a first client library according to the number of concurrent threads includes:

[0099] Based on the first client library, the files and directories to be migrated in the remote network file sharing system on the source side are traversed and scanned, and parameter information of the files to be migrated determined according to the traversal and scanning results is recorded, and the parameter information of the files to be migrated at least includes a hash value. Specifically, the directories and files in the NFS share on the source side are traversed and scanned through libnfs, and the total number of files to be migrated, the file md5 value (i.e., the hash value), the last modified timestamp and other information are recorded to mark the progress of the migration task and the verification after the file migration;

[0100] According to the traversal scanning order, the preset reading length and the number of concurrent threads, at least one thread is started and the data of the file to be migrated in the source-end remote network file sharing system is read through the first client library.

[0101] Specifically, in the destination storage main program, the file data content in the source NFS share is read through libnfs in the order of scanning. If the preset reading length is 1MiB, only 1MiB of data content in the file to be migrated is read at most each time, so as to control the program memory usage and improve the reading efficiency. If the size of the file to be migrated is less than 1MiB, it is read according to the actual file size. After the corresponding file data is read based on multiple threads, if there are other files, the reading operation of other files will continue, and so on.

[0102] In some specific implementations, starting at least one thread and reading data and metadata information of the file to be migrated in the source remote network file sharing system through the first client library according to the number of concurrent threads also includes:

[0103] In response to the data of the file to be migrated being read completely, metadata information of the file to be migrated in the source-end remote network file sharing system is read through the first client library.

[0104] Specifically, after the data of the file to be migrated is read, the metadata information of the source file is obtained through libnfs. The metadata information may include access time (the time when the file is read or executed), modify time (the time when the file content is modified), change time (the time when the file status changes), mode (file permission), uid (the user ID to which the file belongs), gid (the user group ID to which the file belongs) and size (file size information). After obtaining the file metadata information, it is filtered according to the custom metadata set in the data migration request. If it is not specified in the data migration request, all the above metadata information is read by default. The number of threads for reading the metadata information can be the number of threads corresponding to the reading data, and can also be recalculated according to the occupied space of the metadata information. The specific calculation method is the same as the method for calculating the number of threads for reading data in the above step, which will not be repeated here. You can select one of the two methods to determine the number of concurrent threads according to actual needs.

[0105] In the above implementation, the data and metadata information of the files to be migrated are read separately through libnfs, and the custom metadata items corresponding to the data requests are automatically matched through the metadata synchronization interface of Ceph, which improves the flexibility of metadata synchronization, ensures the complete consistency of data and metadata, and improves the reliability of data migration.

[0106] S4: In response to the reading completion, the data and metadata information of the file to be migrated are stored in the file system of the destination storage end through the second client library, and the second client library is used to directly access the file system of the destination storage end through the second programming interface.

[0107] It should be noted that the second client library refers to libClient (Ceph Client Library), and the file system of the destination storage end refers to the Ceph file system; the second programming interface refers to a specification or protocol defined in programming to specify the method for classes or components to communicate with each other, which is used to specify the communication method between the second client library and the file system of the destination storage end.

[0108] In some specific implementations, in response to the data of the file to be migrated having been read, the method further includes:

[0109] Based on the file system identifier of the destination storage end, determine the file system of the destination storage end, and detect whether the target directory exists in the file system according to the write directory corresponding to the file system identifier of the destination storage end, wherein the file system identifier of the destination storage end and the write directory corresponding to the destination storage end are obtained by the traversal scanning operation in the above step, such as file A;

[0110] In response to existence, storing the data of the file to be migrated in a corresponding target directory in the file system through the second client library;

[0111] In response to the non-existence of the target directory, create a target directory, and after the creation is completed, store the data of the file to be migrated to the corresponding target directory in the file system through the second client library;

[0112] In response to the data of the file to be migrated having been stored in the corresponding target directory in the file system, the metadata information of the file to be migrated in the source remote network file sharing system is read through the first client library, and the metadata information of the file to be migrated is stored in the corresponding target directory in the file system through the second client library.

[0113] Specifically, Figure 4 As shown, when the data of the file to be migrated has been read into libnfs, in the main program of the destination storage end, the read data is written into the target directory of the Ceph file system in sequence through libClient. Before writing the data, it is first detected whether there is a directory or file in the Ceph file system. If not, a new file or directory with the same name is created and appended to it. A maximum of 1MiB of content is written each time until the reading and writing of the file data is completed. Then, the next file is created and written. After the data of the file to be migrated has been written into the Ceph file system, the metadata information of the file to be migrated that has been read is written into the target directory of the Ceph file system in sequence through libClient.

[0114] Furthermore, after reading the data and metadata information through libnfs, because there may be a situation where the data or metadata information is read in parallel by multiple threads in fragments, where the reading position of each thread is determined by the starting offset and the reading length, to avoid repeated reading, all fragments are merged in the global cache according to the offset after reading. Based on this, the fragments are merged by adding a global cache and a thread-local cache, where the global cache is used to save shared data of the same NFS connection, such as storing all task queues under this connection and the data after the fragments are merged by each thread, and the thread-local cache is responsible for the read data saved by each thread.

[0115] More specifically, the file migration process includes: following the traversal order of the entire shared directory structure, starting from the bottom-level directory, using the nfs_open interface of the libnfs library to open the file from the source, using the nfs_read interface to read the file, and specifying the maximum cache size for each read as 1 MiB to prevent the cache from being occupied too much by reading the entire file at one time. At the same time, if the target directory does not exist in the local Ceph file system, in the local Ceph file system destination path, calling the mkdir interface of the libClient library to create a directory with the same name on the source side, calling the create_and_open interface to create a file with the same name on the source side, and appending the read content to the local file with a size of 1 MiB until the reading and writing of the file to be migrated is completed, and finally calling the nfs of the libnfs library close Close the file to be migrated opened on the source side. At this time, the data part of the file to be migrated has been migrated. After the data migration of the file to be migrated is completed, the metadata information is synchronized, that is, the nfs_fstat64 interface of the libnfs library is used to obtain all metadata information of the source file. After matching with the custom metadata items, only the set metadata items are synchronized, and the setattr interface of the libClient library is used to set the source file metadata information to the files on the Ceph file system in turn; among them, in order to ensure data consistency when multi-threaded tasks are running, for operations such as opening (open), reading (read), and metadata acquisition of files, add Operational thread locks refine the locks to the file level, that is, multiple threads are allowed to simultaneously process the opening and reading operations of different files in the same NFS connection session, that is, the operation-class mutex lock pthread_mutex_ops is added to the nfs file handle structure nfsfh, allowing the open, read, and getattr operations of different files in the task queue to be executed separately without affecting each other. In the case of non-large file fragment access, each thread executes migration operations on different files from the task queue. Among them, multi-threaded task execution uses the libnfs asynchronous interface, such as nfs_open_async and nfs_read_async.

[0116] In some specific implementations, after storing the data and metadata information of the file to be migrated in the file system of the destination storage end through the second client library, the method further includes:

[0117] Based on the data migration request, determine whether it is necessary to perform consistency check on the target file stored in the target storage end file system;

[0118] In response to the need to perform consistency check on the target file, a hash value of the target file is calculated, wherein the hash value is a string of fixed length obtained by calculating a hash function, and the calculation process is a common method and will not be described in detail here;

[0119] After obtaining the hash value of the target file, the hash value of the target file is compared with the hash value of the file to be migrated;

[0120] In response to the hash value of the target file being successfully compared with the hash value of the file to be migrated, it is determined that the target file has passed the consistency check, indicating that the migration of this file has been completed, and the migrated file count is increased by 1 to indicate the migration progress;

[0121] In response to the failure of comparison between the hash value of the target file and the hash value of the file to be migrated, it is determined that the target file has failed the consistency check, indicating that there is a problem with the file data or metadata information during the migration process. This file path is added to a retry queue and waits for migration again to re-execute the data migration operation.

[0122] In some specific implementations, a timeout retry and exit mechanism is set for each thread. When a network failure or other abnormality occurs, the thread executing the task will first retry. After reaching the upper limit of the number of retries, the task failure message is notified to the global task queue, and the tasks in the task queue are distributed to other threads for execution.

[0123] In some specific implementations, the progress of the task is indicated by the number of files migrated / the total number of files to be migrated, which can be viewed by the user in real time. After the task is completed, the number of files that failed to migrate is recorded and displayed.

[0124] In the above implementation, data is saved to the file system of the destination storage end through libClient, and data consistency verification is performed after the migration task is completed. While improving the efficiency of file migration, it can maximize the consistency of data after the file migration is completed, further improving the reliability of data migration.

[0125] In the above data migration method, the method includes: receiving a data migration request; determining data migration task parameters according to the data migration request, the data migration task parameters at least including the number of concurrent threads; starting at least one thread according to the number of concurrent threads and reading the data and metadata information of the file to be migrated in the remote network file sharing system of the source end through the first client library, the first client library is used to directly access the remote network file sharing system of the source end through the first programming interface; in response to the completion of the reading, storing the data and metadata information of the file to be migrated in the file system of the destination storage end through the second client library, the second client library is used to directly access the file system of the destination storage end through the second programming interface. The beneficial effects of the present application include: (1) improving the efficiency of data migration: by using the libnfs library and the libClient library to realize the mounting and data reading of the remote NFS share, and at the same time utilizing the libClient read-write interface of the Ceph file system (1) Supports custom metadata synchronization: Through Ceph's metadata synchronization interface, it automatically matches custom metadata items corresponding to data migration requests, which improves the flexibility of metadata synchronization and ensures the complete consistency of data and metadata; (2) Supports custom metadata synchronization: Through Ceph's metadata synchronization interface, it automatically matches custom metadata items corresponding to data migration requests, which improves the flexibility of metadata synchronization and ensures the complete consistency of data and metadata; (3) Strong compatibility: It can be used in various NFS-based remote storage environments, has strong compatibility and applicability, and can meet the data migration requirements of different application scenarios; (4) Improves data migration reliability: The high availability and fault tolerance of the Ceph file system ensure the security and reliability of data during the migration process, avoids data loss or damage caused by storage system failures, and supports optional migration task data consistency verification, which can maximize the guarantee of data consistency after the file migration is completed.

[0126] It should be understood that although Figure 2-Figure 4 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 2-Figure 4 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0127] In one embodiment, Figure 5As shown, a data migration device is provided, comprising: a request receiving module, a parameter determining module, a first data migration module and a second data migration module, wherein:

[0128] A request receiving module, used for receiving a data migration request;

[0129] A parameter determination module, used to determine the data migration task parameters according to the data migration request, where the data migration task parameters at least include the number of concurrent threads;

[0130] A first data migration module is used to start at least one thread according to the number of concurrent threads, and read the data and metadata information of the file to be migrated in the source-end remote network file sharing system through a first client library, and the first client library is used to directly access the source-end remote network file sharing system through a first programming interface;

[0131] The second data migration module is used to store the data and metadata information of the file to be migrated to the file system of the destination storage end through the second client library in response to the completion of the reading. The second client library is used to directly access the file system of the destination storage end through the second programming interface.

[0132] As a preferred implementation, in the embodiment of the present invention, the parameter determination module is specifically used for:

[0133] Parsing the data migration request, and determining the data migration task parameters according to the parsing result of the data migration request, wherein the data migration task parameters at least include the sharing address of the remote network file sharing system, the file system identifier of the destination storage end and its corresponding writing directory and the data migration method;

[0134] The method for determining the number of concurrent threads includes:

[0135] Obtain metadata information corresponding to the target file to be migrated;

[0136] Determine the space occupied by the target file to be migrated according to the metadata information corresponding to the target file to be migrated;

[0137] In response to the space occupied by the target to-be-migrated file being less than or equal to a first preset threshold, determining that the number of concurrent threads is a single thread;

[0138] In response to the space occupied by the target file to be migrated being greater than a first preset threshold, obtaining a preset read length, and splitting the target file to be migrated according to the preset read length to obtain a plurality of fragments;

[0139] Determine the number of concurrent threads corresponding to the target files to be migrated according to the number of shards of the multiple shards and the number of files to be migrated;

[0140] Based on the concurrent numbers of threads corresponding to the multiple target files to be migrated, the total concurrent number of threads is determined, and the total concurrent number of threads is less than or equal to the number of threads in the thread pool.

[0141] As a preferred implementation, in the embodiment of the present invention, the device further includes a connection mounting module, and the connection mounting module is specifically used for:

[0142] Integrate the first client library and the second client library at the destination storage end;

[0143] Based on the shared address of the remote network file sharing system, the remote network file sharing system is connected and mounted using the first client library.

[0144] As a preferred implementation, in the embodiment of the present invention, the first data migration module is specifically used for:

[0145] Based on the first client library, traverse and scan the files and directories to be migrated in the remote network file sharing system at the source end, and record parameter information of the files to be migrated determined according to the traversal and scanning results, where the parameter information of the files to be migrated includes at least a hash value;

[0146] According to the traversal scanning order, the preset reading length and the number of concurrent threads, at least one thread is started and the data of the file to be migrated in the source-end remote network file sharing system is read through the first client library.

[0147] As a preferred implementation manner, in the embodiment of the present invention, the first data migration module is further configured to:

[0148] In response to the data of the file to be migrated being read completely, metadata information of the file to be migrated in the source-end remote network file sharing system is read through the first client library.

[0149] As a preferred implementation, in the embodiment of the present invention, the second data migration module is specifically used for:

[0150] Based on the file system identifier of the destination storage end, determine the file system of the destination storage end, and detect whether the target directory exists in the file system according to the write directory corresponding to the file system identifier of the destination storage end;

[0151] In response to existence, storing the data of the file to be migrated in a corresponding target directory in the file system through the second client library;

[0152] In response to the non-existence of the target directory, create a target directory, and after the creation is completed, store the data of the file to be migrated to the corresponding target directory in the file system through the second client library;

[0153] In response to the data of the file to be migrated having been stored in the corresponding target directory in the file system, the metadata information of the file to be migrated in the source remote network file sharing system is read through the first client library, and the metadata information of the file to be migrated is stored in the corresponding target directory in the file system through the second client library.

[0154] As a preferred implementation, in the embodiment of the present invention, the device further includes a verification module, and the verification module is specifically used to:

[0155] Based on the data migration request, determine whether it is necessary to perform consistency check on the target file stored in the target storage end file system;

[0156] In response to the need to perform consistency check on the target file, calculating a hash value of the target file;

[0157] In response to a successful comparison between the hash value of the target file and the hash value of the file to be migrated, determining that the target file passes the consistency check;

[0158] In response to a failure in comparing the hash value of the target file with the hash value of the file to be migrated, it is determined that the target file fails the consistency check, and the data migration operation is re-executed.

[0159] For the specific definition of the data migration device, please refer to the definition of the data migration method above, which will not be repeated here. Each module in the above-mentioned data migration device can be implemented in whole or in part by software, hardware and a combination thereof. Each of the above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to each of the above modules.

[0160] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Figure 6 As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a data migration method is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a key, trackball or touchpad set on the computer device housing, or an external keyboard, touchpad or mouse, etc.

[0161] Those skilled in the art will understand that Figure 6 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0162] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the following steps when executing the computer program:

[0163] S1: Receive a data migration request.

[0164] S2: Determine data migration task parameters according to the data migration request, where the data migration task parameters at least include the number of concurrent threads;

[0165] S3: starting at least one thread according to the number of concurrent threads and reading data and metadata information of the file to be migrated in the source-end remote network file sharing system through a first client library, where the first client library is used to directly access the source-end remote network file sharing system through a first programming interface;

[0166] S4: In response to the reading completion, the data and metadata information of the file to be migrated are stored in the file system of the destination storage end through the second client library, and the second client library is used to directly access the file system of the destination storage end through the second programming interface.

[0167] In one embodiment, when the processor executes the computer program, the following steps are also implemented:

[0168] Parsing the data migration request, and determining the data migration task parameters according to the parsing result of the data migration request, wherein the data migration task parameters at least include the sharing address of the remote network file sharing system, the file system identifier of the destination storage end and its corresponding writing directory and the data migration method;

[0169] The method for determining the number of concurrent threads includes:

[0170] Obtain metadata information corresponding to the target file to be migrated;

[0171] Determine the space occupied by the target file to be migrated according to the metadata information corresponding to the target file to be migrated;

[0172] In response to the space occupied by the target to-be-migrated file being less than or equal to a first preset threshold, determining that the number of concurrent threads is a single thread;

[0173] In response to the space occupied by the target file to be migrated being greater than a first preset threshold, obtaining a preset read length, and splitting the target file to be migrated according to the preset read length to obtain a plurality of fragments;

[0174] Determine the number of concurrent threads corresponding to the target files to be migrated according to the number of shards of the multiple shards and the number of files to be migrated;

[0175] Based on the concurrent numbers of threads corresponding to the multiple target files to be migrated, the total concurrent number of threads is determined, and the total concurrent number of threads is less than or equal to the number of threads in the thread pool.

[0176] In one embodiment, when the processor executes the computer program, the following steps are also implemented:

[0177] Integrate the first client library and the second client library at the destination storage end;

[0178] Based on the shared address of the remote network file sharing system, the remote network file sharing system is connected and mounted using the first client library.

[0179] In one embodiment, when the processor executes the computer program, the following steps are also implemented:

[0180] Based on the first client library, traverse and scan the files and directories to be migrated in the remote network file sharing system at the source end, and record parameter information of the files to be migrated determined according to the traversal and scanning results, where the parameter information of the files to be migrated includes at least a hash value;

[0181] According to the traversal scanning order, the preset reading length and the number of concurrent threads, at least one thread is started and the data of the file to be migrated in the source-end remote network file sharing system is read through the first client library.

[0182] In one embodiment, when the processor executes the computer program, the following steps are also implemented:

[0183] In response to the data of the file to be migrated being read completely, metadata information of the file to be migrated in the source-end remote network file sharing system is read through the first client library.

[0184] In one embodiment, when the processor executes the computer program, the following steps are also implemented:

[0185] Based on the file system identifier of the destination storage end, determine the file system of the destination storage end, and detect whether the target directory exists in the file system according to the write directory corresponding to the file system identifier of the destination storage end;

[0186] In response to existence, storing the data of the file to be migrated in a corresponding target directory in the file system through the second client library;

[0187] In response to the non-existence of the target directory, create a target directory, and after the creation is completed, store the data of the file to be migrated to the corresponding target directory in the file system through the second client library;

[0188] In response to the data of the file to be migrated having been stored in the corresponding target directory in the file system, the metadata information of the file to be migrated in the source remote network file sharing system is read through the first client library, and the metadata information of the file to be migrated is stored in the corresponding target directory in the file system through the second client library.

[0189] In one embodiment, when the processor executes the computer program, the following steps are also implemented:

[0190] Based on the data migration request, determine whether it is necessary to perform consistency check on the target file stored in the target storage end file system;

[0191] In response to the need to perform consistency check on the target file, calculating a hash value of the target file;

[0192] In response to a successful comparison between the hash value of the target file and the hash value of the file to be migrated, determining that the target file passes the consistency check;

[0193] In response to a failure in comparing the hash value of the target file with the hash value of the file to be migrated, it is determined that the target file fails the consistency check, and the data migration operation is re-executed.

[0194] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0195] S1: Receive a data migration request.

[0196] S2: Determine data migration task parameters according to the data migration request, where the data migration task parameters at least include the number of concurrent threads;

[0197] S3: starting at least one thread according to the number of concurrent threads and reading data and metadata information of the file to be migrated in the source-end remote network file sharing system through a first client library, where the first client library is used to directly access the source-end remote network file sharing system through a first programming interface;

[0198] S4: In response to the reading completion, the data and metadata information of the file to be migrated are stored in the file system of the destination storage end through the second client library, and the second client library is used to directly access the file system of the destination storage end through the second programming interface.

[0199] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0200] Parsing the data migration request, and determining the data migration task parameters according to the parsing result of the data migration request, wherein the data migration task parameters at least include the sharing address of the remote network file sharing system, the file system identifier of the destination storage end and its corresponding writing directory and the data migration method;

[0201] The method for determining the number of concurrent threads includes:

[0202] Obtain metadata information corresponding to the target file to be migrated;

[0203] Determine the space occupied by the target file to be migrated according to the metadata information corresponding to the target file to be migrated;

[0204] In response to the space occupied by the target to-be-migrated file being less than or equal to a first preset threshold, determining that the number of concurrent threads is a single thread;

[0205] In response to the space occupied by the target file to be migrated being greater than a first preset threshold, obtaining a preset read length, and splitting the target file to be migrated according to the preset read length to obtain a plurality of fragments;

[0206] Determine the number of concurrent threads corresponding to the target files to be migrated according to the number of shards of the multiple shards and the number of files to be migrated;

[0207] Based on the concurrent numbers of threads corresponding to the multiple target files to be migrated, the total concurrent number of threads is determined, and the total concurrent number of threads is less than or equal to the number of threads in the thread pool.

[0208] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0209] Integrate the first client library and the second client library at the destination storage end;

[0210] Based on the shared address of the remote network file sharing system, the remote network file sharing system is connected and mounted using the first client library.

[0211] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0212] Based on the first client library, traverse and scan the files and directories to be migrated in the remote network file sharing system at the source end, and record parameter information of the files to be migrated determined according to the traversal and scanning results, where the parameter information of the files to be migrated includes at least a hash value;

[0213] According to the traversal scanning order, the preset reading length and the number of concurrent threads, at least one thread is started and the data of the file to be migrated in the source-end remote network file sharing system is read through the first client library.

[0214] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0215] In response to the data of the file to be migrated being read completely, metadata information of the file to be migrated in the source-end remote network file sharing system is read through the first client library.

[0216] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0217] Based on the file system identifier of the destination storage end, determine the file system of the destination storage end, and detect whether the target directory exists in the file system according to the write directory corresponding to the file system identifier of the destination storage end;

[0218] In response to existence, storing the data of the file to be migrated in a corresponding target directory in the file system through the second client library;

[0219] In response to the non-existence of the target directory, create a target directory, and after the creation is completed, store the data of the file to be migrated to the corresponding target directory in the file system through the second client library;

[0220] In response to the data of the file to be migrated having been stored in the corresponding target directory in the file system, the metadata information of the file to be migrated in the source remote network file sharing system is read through the first client library, and the metadata information of the file to be migrated is stored in the corresponding target directory in the file system through the second client library.

[0221] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0222] Based on the data migration request, determine whether it is necessary to perform consistency check on the target file stored in the target storage end file system;

[0223] In response to the need to perform consistency check on the target file, calculating a hash value of the target file;

[0224] In response to a successful comparison between the hash value of the target file and the hash value of the file to be migrated, determining that the target file passes the consistency check;

[0225] In response to a failure in comparing the hash value of the target file with the hash value of the file to be migrated, it is determined that the target file fails the consistency check, and the data migration operation is re-executed.

[0226] In one embodiment, a computer program product is provided, the computer program product comprising a computer program, the computer program when executed by a processor implements the following steps:

[0227] S1: Receive a data migration request.

[0228] S2: Determine data migration task parameters according to the data migration request, where the data migration task parameters at least include the number of concurrent threads;

[0229] S3: starting at least one thread according to the number of concurrent threads and reading data and metadata information of the file to be migrated in the source-end remote network file sharing system through a first client library, where the first client library is used to directly access the source-end remote network file sharing system through a first programming interface;

[0230] S4: In response to the reading completion, the data and metadata information of the file to be migrated are stored in the file system of the destination storage end through the second client library, and the second client library is used to directly access the file system of the destination storage end through the second programming interface.

[0231] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0232] Parsing the data migration request, and determining the data migration task parameters according to the parsing result of the data migration request, wherein the data migration task parameters at least include the sharing address of the remote network file sharing system, the file system identifier of the destination storage end and its corresponding writing directory and the data migration method;

[0233] The method for determining the number of concurrent threads includes:

[0234] Obtain metadata information corresponding to the target file to be migrated;

[0235] Determine the space occupied by the target file to be migrated according to the metadata information corresponding to the target file to be migrated;

[0236] In response to the space occupied by the target to-be-migrated file being less than or equal to a first preset threshold, determining that the number of concurrent threads is a single thread;

[0237] In response to the space occupied by the target file to be migrated being greater than a first preset threshold, obtaining a preset read length, and splitting the target file to be migrated according to the preset read length to obtain a plurality of fragments;

[0238] Determine the number of concurrent threads corresponding to the target files to be migrated according to the number of shards of the multiple shards and the number of files to be migrated;

[0239] Based on the concurrent numbers of threads corresponding to the multiple target files to be migrated, the total concurrent number of threads is determined, and the total concurrent number of threads is less than or equal to the number of threads in the thread pool.

[0240] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0241] Integrate the first client library and the second client library at the destination storage end;

[0242] Based on the shared address of the remote network file sharing system, the remote network file sharing system is connected and mounted using the first client library.

[0243] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0244] Based on the first client library, traverse and scan the files and directories to be migrated in the remote network file sharing system at the source end, and record parameter information of the files to be migrated determined according to the traversal and scanning results, where the parameter information of the files to be migrated includes at least a hash value;

[0245] According to the traversal scanning order, the preset reading length and the number of concurrent threads, at least one thread is started and the data of the file to be migrated in the source-end remote network file sharing system is read through the first client library.

[0246] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0247] In response to the data of the file to be migrated being read completely, metadata information of the file to be migrated in the source-end remote network file sharing system is read through the first client library.

[0248] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0249] Based on the file system identifier of the destination storage end, determine the file system of the destination storage end, and detect whether the target directory exists in the file system according to the write directory corresponding to the file system identifier of the destination storage end;

[0250] In response to existence, storing the data of the file to be migrated in a corresponding target directory in the file system through the second client library;

[0251] In response to the non-existence of the target directory, create a target directory, and after the creation is completed, store the data of the file to be migrated to the corresponding target directory in the file system through the second client library;

[0252] In response to the data of the file to be migrated having been stored in the corresponding target directory in the file system, the metadata information of the file to be migrated in the source remote network file sharing system is read through the first client library, and the metadata information of the file to be migrated is stored in the corresponding target directory in the file system through the second client library.

[0253] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0254] Based on the data migration request, determine whether it is necessary to perform consistency check on the target file stored in the target storage end file system;

[0255] In response to the need to perform consistency check on the target file, calculating a hash value of the target file;

[0256] In response to a successful comparison between the hash value of the target file and the hash value of the file to be migrated, determining that the target file passes the consistency check;

[0257] In response to a failure in comparing the hash value of the target file with the hash value of the file to be migrated, it is determined that the target file fails the consistency check, and the data migration operation is re-executed.

[0258] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0259] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0260] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application.

Claims

1. A data migration method, characterized in that: The method comprises: Receive data migration requests; Determining data migration task parameters according to the data migration request, wherein the data migration task parameters at least include the number of concurrent threads; According to the concurrent number of threads, at least one thread is started and data and metadata information of the file to be migrated in the source-end remote network file sharing system is read through a first client library, wherein the first client library is used to directly access the source-end remote network file sharing system through a first programming interface; In response to the reading completion, the data and metadata information of the file to be migrated are stored in the file system of the destination storage end through the second client library, and the second client library is used to directly access the file system of the destination storage end through the second programming interface.

2. The data migration method according to claim 1, characterized in that: Determining data migration task parameters according to the data migration request includes: Parsing the data migration request, and determining the data migration task parameters according to the parsing result of the data migration request, wherein the data migration task parameters at least include a sharing address of a remote network file sharing system, a file system identifier of a destination storage end and a corresponding write directory and a data migration method; The method for determining the number of concurrent threads includes: Obtain metadata information corresponding to the target file to be migrated; Determining the space occupied by the target file to be migrated according to the metadata information corresponding to the target file to be migrated; In response to the space occupied by the target file to be migrated being less than or equal to a first preset threshold, determining that the number of concurrent threads is a single thread; In response to the space occupied by the target file to be migrated being greater than a first preset threshold, obtaining a preset read length, and splitting the target file to be migrated according to the preset read length to obtain a plurality of fragments; Determining the number of concurrent threads corresponding to the target files to be migrated according to the number of shards of the multiple shards and the number of files to be migrated; Based on the concurrent numbers of threads corresponding to the multiple target files to be migrated, the total concurrent number of threads is determined, where the total concurrent number of threads is less than or equal to the number of threads in the thread pool.

3. The data migration method according to claim 2, characterized in that: After determining the data migration task parameters according to the data migration request, the method further includes: Integrate the first client library and the second client library on the destination storage end; Based on the shared address of the remote network file sharing system, the remote network file sharing system is connected and mounted using the first client library.

4. The data migration method according to claim 2, characterized in that: Starting at least one thread according to the concurrent number of threads and reading data and metadata information of the file to be migrated in the source-end remote network file sharing system through the first client library includes: Performing a traversal scan on the files and directories to be migrated in the remote network file sharing system at the source end based on the first client library, and recording parameter information of the files to be migrated determined according to the traversal scan result, wherein the parameter information of the files to be migrated includes at least a hash value; According to the traversal scanning order, the preset reading length and the concurrent number of threads, at least one thread is started and the data of the file to be migrated in the source-end remote network file sharing system is read through the first client library.

5. The data migration method according to claim 4, characterized in that: Starting at least one thread according to the concurrent number of threads and reading data and metadata information of the file to be migrated in the source-end remote network file sharing system through the first client library also includes: In response to the data of the file to be migrated being read completely, metadata information of the file to be migrated in the source-end remote network file sharing system is read through the first client library.

6. The data migration method according to claim 1 or 5, characterized in that: In response to the data of the file to be migrated having been read, the method further includes: Based on the file system identifier of the destination storage end, determine the file system of the destination storage end, and detect whether there is a target directory in the file system according to the write directory corresponding to the file system identifier of the destination storage end; In response to existence, storing the data of the file to be migrated in a corresponding target directory in the file system through a second client library; In response to the non-existence of the target directory, create a target directory, and after the creation is completed, store the data of the file to be migrated to the corresponding target directory in the file system through the second client library; In response to the data of the file to be migrated having been stored in the corresponding target directory in the file system, the metadata information of the file to be migrated in the source remote network file sharing system is read through the first client library, and the metadata information of the file to be migrated is stored in the corresponding target directory in the file system through the second client library.

7. The data migration method according to claim 1, characterized in that: After storing the data and metadata information of the file to be migrated in the file system of the destination storage end through the second client library, the method further includes: Based on the data migration request, determine whether it is necessary to perform consistency check on the target file stored in the target storage end file system; In response to the need to perform consistency check on the target file, calculating a hash value of the target file; In response to a successful comparison between the hash value of the target file and the hash value of the file to be migrated, determining that the target file passes the consistency check; In response to a failure in comparing the hash value of the target with the hash value of the file to be migrated, it is determined that the target file fails the consistency check, and the data migration operation is re-executed.

8. A data migration device, characterized in that: The device comprises: A request receiving module, used for receiving a data migration request; A parameter determination module, configured to determine a data migration task parameter according to the data migration request, wherein the data migration task parameter includes at least a concurrent number of threads; A first data migration module, configured to start at least one thread according to the concurrent number of threads, and read data and metadata information of files to be migrated in a source-end remote network file sharing system through a first client library, wherein the first client library is configured to directly access the source-end remote network file sharing system through a first programming interface; The second data migration module is used to store the data and metadata information of the file to be migrated to the file system of the destination storage end through the second client library in response to the completion of the reading. The second client library is used to directly access the file system of the destination storage end through the second programming interface.

9. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images