A data migration method and related components of a distributed file system

Abstract

The present invention discloses a data migration method for a distributed file system and related components, and relates to the field of distributed file systems. The method comprises obtaining attribute information of each file under each directory in a fast pool of the distributed file system, storing the attribute information in a database in the form of a data table, so as to determine a target migration file that meets preset migration conditions through the data table, determining a final file to be migrated by comparing the target migration file with a historical target migration file obtained last time, and finally migrating the file to be migrated in the fast pool to a slow pool of the distributed file system. The speed of determining that the target migration file is relatively fast by comparing the data table can thus accelerate the migration of the file to be migrated from the fast pool to the slow pool.

Description

Technical Field

[0001] The present invention relates to the field of distributed file systems, and in particular to a data migration method and related components of a distributed file system. Background Art

[0002] In the context of digital transformation, the amount of data stored in servers is constantly increasing. To increase the available disk space, distributed file systems are currently often used to build a high-performance, large-capacity, and low-cost storage system. Distributed file systems typically utilize the fast access speed of NVME SSDs (Non-Volatile Memory Express Solid State Disks) to serve as the fast pool in the distributed file system to store frequently accessed hot data. They also utilize the slow access speed but large capacity of ordinary mechanical hard disks to serve as the slow pool to store infrequently accessed cold data. When using a distributed file system, as the amount of data stored in the fast pool gradually increases, some of the data in the fast pool needs to be migrated to the slow pool. Therefore, it is very important to speed up the migration of data from the fast pool to the slow pool to meet the needs of users for continuously writing data to the fast pool at high speed. Summary of the Invention

[0003] The object of the present invention is to provide a data migration method and related components of a distributed file system, which can accelerate the migration of files to be migrated from a fast pool to a slow pool.

[0004] To solve the above technical problems, the present invention provides a data migration method for a distributed file system, comprising:

[0005] Obtaining attribute information of each file in each directory in the distributed file system's fast pool, and storing the attribute information in a database in the form of a data table;

[0006] Determining target migration files that meet preset migration conditions through the data table;

[0007] Determine the file to be migrated by comparing the target migration file with the historical target migration file obtained last time;

[0008] Migrate the files to be migrated to the slow pool of the distributed file system.

[0009] Preferably, after determining the target migration file that meets the preset migration conditions through the data table, the method further includes:

[0010] Determine whether the table structure information of the data table obtained this time is consistent with the table structure information of the historical data table obtained last time;

[0011] If yes, proceed to the step of determining the file to be migrated by comparing the target migration file with the historical target migration file obtained last time;

[0012] If not, the process proceeds to the step of obtaining the attribute information of each file in each directory in the distributed file system's buffer pool.

[0013] Preferably, determining the file to be migrated by comparing the target migration file with the historical target migration file obtained last time includes:

[0014] Comparing the column offset of the target migration file with the column offset of the historical target migration file, and the row information of the current target migration file with the row information of the historical target migration file to determine the files in the target migration file that have been modified and added compared to the historical target migration file;

[0015] The files in the target migration file that are modified and added compared to the historical target migration file are used as the files to be migrated.

[0016] Preferably, migrating the to-be-migrated file to the slow pool of the distributed file system includes:

[0017] Creating a temporary file for storing the file to be migrated;

[0018] Migrating the file to be migrated to the temporary file, and after the migration is completed, detecting whether the content of the temporary file is consistent with the content of the file to be migrated;

[0019] If they are inconsistent, the migration is determined to have failed, and the process proceeds to the step of obtaining the attribute information of each file in each directory in the distributed file system's fast pool;

[0020] If they are consistent, the temporary file is changed to a formal file.

[0021] Preferably, when migrating the files to be migrated to the slow pool of the distributed file system, the method further includes:

[0022] Determining a write speed at which a user writes data to the fast pool;

[0023] If the writing speed is greater than the speed of migrating the to-be-migrated file from the fast pool to the slow pool, the data written by the user is directly stored in the slow pool.

[0024] Preferably, migrating the to-be-migrated file to the slow pool of the distributed file system includes:

[0025] The files to be migrated are distributed from the master node to the plurality of slave nodes according to a preset distribution rule so that each of the slave nodes can simultaneously migrate the files to be migrated distributed to itself from the fast pool to the slow pool.

[0026] This application also provides a distributed file system data migration system, including:

[0027] A data table storage unit, configured to obtain attribute information of each file in each directory in the distributed file system's buffer pool, and store the attribute information in a database in the form of a data table;

[0028] a target migration file determining unit, configured to determine, through the data table, a target migration file that meets a preset migration condition;

[0029] a file-to-be-migrated determining unit, configured to determine the file to be migrated by comparing the target migration file with a previously obtained historical target migration file;

[0030] A migration unit is configured to migrate the file to be migrated to a slow pool of the distributed file system.

[0031] The present application also provides a data migration device for a distributed file system, comprising:

[0032] memory for storing computer programs;

[0033] The processor is configured to implement the steps of the above-mentioned distributed file system data migration method when executing the computer program.

[0034] The present application also provides a distributed file system, including the data migration device of the above-mentioned distributed file system, and also including a fast pool and a slow pool for storing data.

[0035] The present application also provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the above-mentioned distributed file system data migration method are implemented.

[0036] In summary, the present invention provides a data migration method and related components for a distributed file system, including obtaining attribute information of each file under each directory in a fast pool of the distributed file system, and storing the attribute information in a database in the form of a data table so as to determine a target migration file that meets preset migration conditions through the data table, determine the final file to be migrated by comparing the target migration file with the historical target migration file obtained last time, and finally migrate the file to be migrated in the fast pool to the slow pool of the distributed file system, and determine that the target migration file is faster by comparing the data table, thereby accelerating the migration of the file to be migrated from the fast pool to the slow pool. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the prior art and the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0038] Figure 1 A schematic diagram of a data migration method for a distributed file system provided by the present invention;

[0039] Figure 2 A schematic diagram of the structure of a data migration system for a distributed file system provided by the present invention;

[0040] Figure 3 A schematic structural diagram of a data migration device for a distributed file system provided by the present invention;

[0041] Figure 4 A schematic structural diagram of a computer-readable storage medium provided by the present invention. DETAILED DESCRIPTION

[0042] The core of the present invention is to provide a data migration method and related components of a distributed file system, which can accelerate the migration of files to be migrated from a fast pool to a slow pool.

[0043] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0044] In the prior art, when using a distributed file system, tools for migrating data from a fast pool to a slow pool are generally not provided. Even when migrating data from a fast pool to a slow pool, the migration is performed on a single file basis, and an additional server is required for the migration operation. Therefore, when faced with the problem of gradually increasing data stored in the fast pool, how to quickly migrate data from the fast pool to the slow pool to meet the needs of users for high-speed and continuous data writing to the fast pool is an urgent problem that needs to be solved.

[0045] Please refer to the figure, Figure 1 A schematic diagram of a data migration method for a distributed file system provided by the present invention includes:

[0046] S1: Obtain attribute information of each file in each directory in the distributed file system's fast pool, and store the attribute information in the form of a data table in the database;

[0047] In this application, the attribute information of each file in the fast pool of the distributed file system is first obtained. Based on the file attribute information, whether the file needs to be migrated from the fast pool to the slow pool is determined. The file attribute information includes information such as the file name, file size, file permissions, file creation time, and creation user, which is not specifically limited in this application. Furthermore, considering that the workload required to determine the files to be migrated by directly comparing file content or file attribute changes is too cumbersome, in this application, the file attribute information is converted into structured relational data, that is, stored in the form of a data table in a database, and the files to be migrated are determined by reading the data table.

[0048] Furthermore, the present application can utilize multiple threads to simultaneously obtain the attribute information of each file in the distributed file system, thereby accelerating the efficiency of data migration. The present application does not specifically limit the specific form of the data table. For example, the attribute information of each file in each directory of the distributed file system is stored under the Entries table, and the tree table records the relationship between the directories, subdirectories, and files of the distributed file system. The present application does not specifically limit the type of database that stores the data table. For example, it can be a sqlite3 database.

[0049] S2: Determine the target migration file that meets the preset migration conditions through the data table;

[0050] After converting the file's attribute information into a data table, the database file is parsed by reading the table's structure information. The target migration file is determined using the preset migration criteria. This means that all files in the distributed file system that meet the preset migration criteria are identified. The target migration file may include files that have already been migrated to the slow pool during a previous data migration. The table structure information includes the table's column names, column types, and whether it contains a primary key.

[0051] This application does not specifically limit the preset migration conditions, such as whether the creation time of the file is earlier than the preset creation time, whether the time when the file was last modified is earlier than the preset time node, and whether the size of the file is greater than the preset threshold.

[0052] S3: Determine the files to be migrated by comparing the target migration file with the previously obtained historical target migration file;

[0053] Considering that files in a distributed file system that meet preset migration criteria include both target migration files that have already been migrated from the fast pool to the slow pool during historical migrations, as well as target migration files that have been newly added or modified by the user and have not yet been migrated from the fast pool to the slow pool, in order to improve the speed of file migration, in this application, before migrating files from the fast pool to the slow pool, the target migration file obtained this time is compared with the historical target migration file obtained previously. The file after removing the historical target migration file from the target migration file is used as the final file to be migrated, thereby reducing the workload required for file migration and further improving the speed of file migration.

[0054] S4: Migrate the files to be migrated to the slow pool of the distributed file system.

[0055] After determining the files to be migrated, the files to be migrated are migrated from the fast pool to the slow pool, and multiple threads can be used to simultaneously migrate the files to be migrated from the fast pool to the slow pool during the migration, thereby further improving the speed of file migration. In summary, the present invention provides a data migration method for a distributed file system, including obtaining attribute information of each file under each directory in the fast pool of the distributed file system, and storing the attribute information in the form of a data table in a database so as to determine the target migration file that meets the preset migration conditions through the data table, determine the final file to be migrated by comparing the target migration file with the historical target migration file obtained last time, and finally migrate the files to be migrated in the fast pool to the slow pool of the distributed file system, and determine that the speed of the target migration file is relatively fast by comparing the data table, thereby accelerating the speed of migrating the files to be migrated from the fast pool to the slow pool.

[0056] Based on the above embodiment:

[0057] As a preferred embodiment, after determining the target migration file that meets the preset migration conditions through the data table, the method further includes:

[0058] Determine whether the table structure information of the data table obtained this time is consistent with the table structure information of the historical data table obtained last time;

[0059] If yes, the process proceeds to the step of determining the file to be migrated by comparing the target migration file with the historical target migration file obtained last time;

[0060] If not, the process proceeds to the step of obtaining the attribute information of each file in each directory in the distributed file system's buffer pool.

[0061] In this embodiment, to ensure the accuracy of file migration, after determining the target migration file that meets the preset migration conditions through the data table, the correctness of the data table itself is further determined. Specifically, the table structure information of the newly obtained data table is determined to be consistent with the table structure information of the previously obtained historical data table. If the table structure information of the newly obtained data table is consistent with the table structure information of the previously obtained historical data table, the newly obtained data table is determined to be correct. Specifically, this can be achieved by comparing the column names, column types, and whether it is a primary key in the newly obtained data table, which is not particularly limited in this application. When it is determined that the table structure information of the data table obtained this time is consistent with the table structure information of the historical data table obtained last time, the data table obtained this time is determined to be correct, so the next step of file migration is to determine the files to be migrated by comparing the target migration file with the historical target migration file obtained last time; when it is determined that the table structure information of the data table obtained this time is inconsistent with the table structure information of the historical data table obtained last time, it is considered that the data table obtained this time is incorrect, so it is necessary to regenerate the data table again, that is, to obtain the attribute information of each file under each directory in the fast pool of the distributed file system. It should be noted that the step of regenerating the data table can be entered immediately, or the current file migration can be terminated first, and then the step of obtaining the attribute information of each file under each directory in the fast pool of the distributed file system can be entered after receiving the instruction for the next file migration.

[0062] In summary, this embodiment ensures the accuracy of file migration by further verifying the correctness of the data table.

[0063] As a preferred embodiment, determining the file to be migrated by comparing the target migration file with the historical target migration file obtained last time includes:

[0064] Compare the column offsets of the target migration file with those of the historical target migration file, as well as the row information of the current target migration file with those of the historical target migration file, to determine the files in the target migration file that have been modified or added compared to the historical target migration file.

[0065] The files in the target migration file that are modified or added compared to the historical target migration file are regarded as files to be migrated.

[0066] In this embodiment, the files to be migrated are determined based on the target migration file obtained this time and the historical target migration file obtained last time by comparing the column offsets of the target migration file and the historical target migration file, as well as comparing the row information of the target migration file and the historical target migration file. The information in the data table is stored in binary form. By reading the offsets, the specific content of each column in the data table can be obtained. The comparison results of the row information in the data table are further combined to determine whether each target migration file in the distributed file system has been added, modified, or deleted relative to the historical target migration file. In this embodiment, the files in the target migration file that have been modified or newly added compared to the historical target migration file are regarded as the final files to be migrated, thereby more accurately migrating files from the fast pool to the slow pool.

[0067] As a preferred embodiment, migrating the files to be migrated to the slow pool of the distributed file system includes:

[0068] Create a temporary file to store the files to be migrated;

[0069] Migrate the files to be migrated to a temporary file, and after the migration is complete, check whether the contents of the temporary file are consistent with those of the files to be migrated;

[0070] If they are inconsistent, the migration is determined to have failed, and the process proceeds to the step of obtaining the attribute information of each file in each directory in the distributed file system's fast pool;

[0071] If they are consistent, the temporary file will be changed to the official file.

[0072] In this embodiment, the correctness of the content of the files to be migrated in the slow pool is further determined, thereby further ensuring the correctness of the file migration. Specifically, the specific process of migrating the files to be migrated from the fast pool to the slow pool is as follows: first, a temporary file is created in the slow pool; then, each file to be migrated is migrated from the fast pool to the temporary file in the slow pool in a certain order; and after the migration is completed, the content of the temporary file is checked to see if it is consistent with the content of the files to be migrated. The files to be migrated are the original files stored in the fast pool. If the user continues to write new content to the files to be migrated or performs other modifications during the migration process, the content of the temporary file in the slow pool will be inconsistent with the content of the files to be migrated in the fast pool. Therefore, the migration is determined to have failed, and the step of obtaining attribute information for each file in each directory in the fast pool of the distributed file system is restarted to start the next file migration. If the user does not write new content to the files to be migrated in the fast pool or perform other modifications during the migration process, the content of the temporary file in the slow pool remains the same as the content of the files to be migrated in the fast pool. At this time, the migration is determined to be successful, and the temporary file is modified to the official file, completing the file migration process.

[0073] As a preferred embodiment, when migrating the files to be migrated to the slow pool of the distributed file system, the method further includes:

[0074] Determine the write speed at which users write data to the fast pool;

[0075] If the write speed is greater than the speed of migrating the files to be migrated from the fast pool to the slow pool, the data written by the user is directly stored in the slow pool.

[0076] Considering that migrating the files to be migrated from the fast pool to the slow pool requires a certain migration time, if the speed at which the user writes data to the fast pool is greater than the speed at which the files to be migrated are migrated from the fast pool to the slow pool during the migration period, files will accumulate in the fast pool, making the storage space of the fast pool unable to meet the user's needs. To solve the above technical problems, in this embodiment, the write speed at which the user writes data to the fast pool is also obtained during the process of migrating the files to be migrated from the fast pool to the slow pool. Specifically, the write speed can be obtained in real time or according to a certain period, and this application does not impose any special restrictions on this. After determining the write speed, it is determined whether the write speed is greater than the speed at which the files to be migrated are migrated from the fast pool to the slow pool. If so, in order to avoid the problem that the remaining storage space in the fast pool cannot meet the user's write needs and thus cause write failures, the data written by the user is directly stored in the slow pool.

[0077] In addition, when users write files to the fast pool at high speed, the preset migration condition can be modified to determine whether the file size is greater than a preset threshold. Large files will be migrated from the fast pool to the slow pool first. After the large files are migrated, small files will be migrated, thereby clearing more available space for the fast pool to meet the user's writing needs.

[0078] As a preferred embodiment, migrating the files to be migrated to the slow pool of the distributed file system includes:

[0079] The files to be migrated are distributed from the master node to the multiple slave nodes according to the preset distribution rules so that each slave node can simultaneously migrate the files to be migrated distributed to it from the fast pool to the slow pool.

[0080] In this embodiment, in order to further improve the file migration speed, a multi-threaded simultaneous migration method is adopted when migrating the files to be migrated from the fast pool to the slow pool of the distributed file system. Specifically, the master node distributes the files to be migrated to multiple slave nodes according to the preset distribution rules. Each slave node migrates its corresponding files to be migrated from the fast pool to the slow pool. Each slave node executes the file migration operation concurrently. This application does not make specific restrictions on the preset distribution rules, such as average distribution.

[0081] Please refer to Figure 2 , Figure 2This is a schematic diagram of the structure of a data migration system for a distributed file system provided by the present invention, the system comprising:

[0082] The data table storage unit 11 is used to obtain the attribute information of each file in each directory in the distributed file system's buffer pool and store the attribute information in the form of a data table in the database;

[0083] A target migration file determining unit 12 is configured to determine a target migration file that meets a preset migration condition through a data table;

[0084] A file to be migrated determining unit 13 is configured to determine a file to be migrated by comparing a target migration file with a previously obtained historical target migration file;

[0085] The migration unit 14 is configured to migrate the files to be migrated to the slow pool of the distributed file system.

[0086] The present invention provides a data migration system for a distributed file system, comprising obtaining attribute information of each file in each directory in a fast pool of the distributed file system, storing the attribute information in a database in the form of a data table so as to determine a target migration file that meets preset migration conditions through the data table, determining a final file to be migrated by comparing the target migration file with a previously obtained historical target migration file, and finally migrating the file to be migrated in the fast pool to a slow pool of the distributed file system. The speed of determining that the target migration file is faster by comparing the data table can thus accelerate the migration of the file to be migrated from the fast pool to the slow pool.

[0087] For a detailed introduction to a distributed file system data migration system provided in this application, please refer to the embodiment of the distributed file system data migration method described above, which will not be elaborated in this application.

[0088] Based on the above embodiment:

[0089] As a preferred embodiment, the present invention further comprises:

[0090] The judgment unit is used to determine whether the table structure information of the data table obtained this time is consistent with the table structure information of the historical data table obtained last time after determining the target migration file that meets the preset migration conditions through the data table; if so, trigger the to-be-migrated file determination unit 13; if not, trigger the data table storage unit 11.

[0091] As a preferred embodiment, the to-be-migrated file determination unit 13 includes:

[0092] a comparison and determination unit, configured to compare the column offsets of the target migration file with the column offsets of the historical target migration file, and the row information of the current target migration file with the row information of the historical target migration file, to determine the files in the target migration file that have been modified or added compared to the historical target migration file;

[0093] The to-be-migrated file determination subunit is configured to select files in the target migration file that are modified or added compared to the historical target migration file as to-be-migrated files.

[0094] As a preferred embodiment, the migration unit 14 includes:

[0095] A temporary file creation unit, used to create a temporary file for storing files to be migrated;

[0096] A detection unit is used to migrate the file to be migrated to a temporary file, and after the migration is completed, detect whether the content of the temporary file is consistent with the content of the file to be migrated; if not, trigger the data table storage unit 11; if consistent, trigger the formal file modification unit;

[0097] The formal file change unit is used to change a temporary file into a formal file.

[0098] As a preferred embodiment, the present invention further comprises:

[0099] a write speed determining unit, configured to determine a write speed at which a user writes data to a fast pool when migrating a file to be migrated to a slow pool of a distributed file system;

[0100] The slow pool storage unit is used to directly store the data written by the user in the slow pool when the writing speed is higher than the speed of migrating the files to be migrated from the fast pool to the slow pool.

[0101] As a preferred embodiment, the migration unit 14 is specifically configured to distribute the files to be migrated from the master node to multiple slave nodes according to a preset distribution rule so that each slave node can simultaneously migrate the files to be migrated distributed to it from the fast pool to the slow pool.

[0102] Please refer to Figure 3 , Figure 3 This is a schematic diagram of the structure of a data migration device for a distributed file system provided by the present invention. The data migration device for a distributed file system includes:

[0103] Memory 21, for storing computer programs;

[0104] The processor 22 is configured to implement the steps of the above-mentioned distributed file system data migration method when executing a computer program.

[0105] The present invention provides a data migration device for a distributed file system. When a computer program in a processor is executed, the computer program can obtain attribute information of each file in each directory in a fast pool of the distributed file system, and store the attribute information in a database in the form of a data table so as to determine a target migration file that meets preset migration conditions through the data table. The target migration file is compared with a historical target migration file obtained last time to determine the final file to be migrated. Finally, the file to be migrated in the fast pool is migrated to the slow pool of the distributed file system. The speed of determining that the target migration file is faster by comparing the data table can thus accelerate the migration of the file to be migrated from the fast pool to the slow pool.

[0106] For a detailed introduction to a data migration device for a distributed file system provided in this application, please refer to the embodiment of the data migration method for the above-mentioned distributed file system, and this application will not elaborate on it here.

[0107] The present application also provides a distributed file system, including the data migration device of the above-mentioned distributed file system, and also including a fast pool and a slow pool for storing data.

[0108] For a detailed introduction to a distributed file system provided in this application, please refer to the embodiment of the data migration method of the above-mentioned distributed file system, and this application will not elaborate on it here.

[0109] Please refer to Figure 4 , Figure 4 This is a structural diagram of a computer-readable storage medium provided by the present invention. The computer-readable storage medium 31 stores a computer program, which implements the steps of the above-mentioned distributed file system data migration method when executed by a processor.

[0110] For a detailed introduction to a computer-readable storage medium provided in this application, please refer to the embodiment of the data migration method of the above-mentioned distributed file system, and this application will not elaborate on it here.

[0111] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Reference can be made to the common and similar parts between the various embodiments. For the devices disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the method description.

[0112] It should also be noted that, in this specification, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or apparatus comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or apparatus comprising the element.

[0113] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the components and steps of each example according to their functions. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present invention.

[0114] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), internal memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0115] The above description of the disclosed embodiments is intended to enable one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention is not limited to the embodiments shown herein but is intended to conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A data migration method for a distributed file system, characterized in that: include: Obtaining attribute information of each file in each directory of the distributed file system's fast pool, and storing the attribute information in a database in the form of a data table; the attribute information includes the file name, file size, file permissions, file creation time, and creation user; Determining target migration files that meet preset migration conditions through the data table; Determine the file to be migrated by comparing the target migration file with the historical target migration file obtained last time; Migrating the files to be migrated to the slow pool of the distributed file system; Determining the files to be migrated by comparing the target migration file with the previously obtained historical target migration file includes: Comparing the column offset of the target migration file with the column offset of the historical target migration file, and the row information of the target migration file with the row information of the historical target migration file to determine the files in the target migration file that are modified and added compared to the historical target migration file; The files in the target migration file that are modified and added compared to the historical target migration file are used as the files to be migrated; Migrating the to-be-migrated file to the slow pool of the distributed file system includes: A temporary file is created for storing the file to be migrated; the file to be migrated is migrated to the temporary file, and after the migration is completed, the content of the temporary file is detected to be consistent with the content of the file to be migrated; if they are inconsistent, the migration is determined to have failed, and the step of obtaining attribute information of each file under each directory in the fast pool of the distributed file system is entered; if they are consistent, the temporary file is changed to a formal file.

2. The data migration method of a distributed file system according to claim 1, wherein: After determining the target migration file that meets the preset migration conditions through the data table, the method further includes: Determine whether the table structure information of the data table obtained this time is consistent with the table structure information of the historical data table obtained last time; If yes, proceed to the step of determining the file to be migrated by comparing the target migration file with the historical target migration file obtained last time; If not, the process proceeds to the step of obtaining the attribute information of each file in each directory in the distributed file system's buffer pool.

3. The data migration method of a distributed file system according to claim 1, wherein: When migrating the to-be-migrated file to the slow pool of the distributed file system, the method further includes: Determining a write speed at which a user writes data to the fast pool; If the writing speed is greater than the speed of migrating the to-be-migrated file from the fast pool to the slow pool, the data written by the user is directly stored in the slow pool.

4. The data migration method of a distributed file system according to any one of claims 1 to 3, characterized in that: Migrating the to-be-migrated file to the slow pool of the distributed file system includes: The files to be migrated are distributed from the master node to the plurality of slave nodes according to a preset distribution rule so that each of the slave nodes can simultaneously migrate the files to be migrated distributed to itself from the fast pool to the slow pool.

5. A data migration system for a distributed file system, characterized in that: include: A data table storage unit is used to obtain attribute information of each file in each directory in the distributed file system's fast pool and store the attribute information in the form of a data table in a database; the attribute information includes the file name, file size, file permissions, file creation time, and creating user; a target migration file determining unit, configured to determine, through the data table, a target migration file that meets a preset migration condition; a file-to-be-migrated determining unit, configured to determine the file to be migrated by comparing the target migration file with a previously obtained historical target migration file; A migration unit, configured to migrate the file to be migrated to a slow pool of the distributed file system; The unit for determining files to be migrated includes: a comparison and determination unit, configured to compare the column offset of the target migration file with the column offset of the historical target migration file, and the row information of the target migration file with the row information of the historical target migration file to determine the files in the target migration file that are modified and added compared to the historical target migration file; a to-be-migrated file determination subunit, configured to select files in the target migration file that are modified and added compared to the historical target migration file as the to-be-migrated files; A temporary file creation unit, used to create a temporary file for storing files to be migrated; The detection unit is used to migrate the file to be migrated to a temporary file and detect whether the content of the temporary file is consistent with the content of the file to be migrated after the migration is completed; if they are inconsistent, the data table storage unit is triggered; if they are consistent, the formal file change unit is triggered; The formal file change unit is used to change a temporary file into a formal file.

6. A data migration device for a distributed file system, characterized in that: include: memory for storing computer programs; A processor, configured to implement the steps of the distributed file system data migration method according to any one of claims 1 to 4 when executing the computer program.

7. A distributed file system, characterized in that: The data migration device of the distributed file system as claimed in claim 6 further comprises a fast pool and a slow pool for storing data.

8. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the distributed file system data migration method according to any one of claims 1 to 4.

Images