A glusterfs-based data balancing method and system

Abstract

The application relates to the technical field of storage, in particular to a data balancing method and system based on GlusterFS, which comprises a GlusterFS distributed storage unit, an AWS S3 interface processing layer, a calculation module, a data balancing module, a link creation module and an index module. The AWS S3 interface processing layer is located at the uppermost layer of the data balancing system and is used for connecting the AWS S3 interface. The calculation module is used for calculating the hash value of a storage file. The data balancing module is used for publishing a command. The link creation module is used for creating a file link corresponding to the storage file. The index module can point the file link corresponding to the storage file to the storage file. Through the improved data balancing algorithm, unnecessary data balancing movement can be eliminated, the storage load can be reduced, the performance overhead can be reduced, and unnecessary hardware loss can be reduced.

Description

Technical Field

[0001] This invention relates to the field of storage technology, and specifically to a data balancing method and system based on GlusterFS. Background Technology

[0002] In today's big data era, the growth of data volume is exponential, and the need for permanent data storage is also increasing. Traditional storage systems are struggling to cope with this explosive growth, leading to the emergence of distributed storage systems. Distributed storage systems use networks and software-defined storage to combine multiple independent servers to address storage capacity and scalability issues. Adding and removing nodes are common challenges in the daily operation and maintenance of distributed storage systems, as seen in systems like Ceph, GlusterFS, and HDFS.

[0003] Taking adding nodes as an example, as data grows, enterprises need to add storage nodes to expand the capacity of the distributed storage cluster. However, after adding new storage nodes, the data storage volume distribution of each node in the cluster becomes uneven. Old data only exists on the original storage nodes, while the newly added storage nodes only store the newly added data. Over time, the original nodes will experience overload, while the newly added storage nodes will not be utilized. Thus, data imbalance occurs between storage nodes.

[0004] like Figure 1 As shown, the file access process in GlusterFS is as follows:

[0005] 1. The data balancing module will calculate a hash value based on the file name;

[0006] 2. Based on the hash value, locate the directory hash subspace where it is located, which means finding the corresponding data volume;

[0007] 3. Access the corresponding data volume and move the file into that data volume.

[0008] However, in GlusterFS, renaming a file will cause the file's hash value to change, such as... Figure 2 As shown, assuming the number of data volumes remains unchanged (i.e., there's no expansion or contraction), the directory hash range of each data volume also remains unchanged. According to the data balancing algorithm, the new storage location of the file is determined. If the new storage location is different from the current data volume, the file should be migrated to the correct data volume. The file's actual content hasn't changed, and the storage environment (physical expansion or contraction) hasn't changed either. However, the file with the changed name has been moved, unnecessarily increasing the storage load and hard drive wear. Summary of the Invention

[0009] The purpose of this invention is to provide a data balancing method and system based on GlusterFS to solve the problems mentioned in the background art.

[0010] The technical solution of the present invention includes:

[0011] GlusterFS distributed storage units are deployed within the cluster management unit, and data volumes can be created within the distributed storage units.

[0012] The AWS S3 interface processing layer is located at the top layer of the data balancing system and is used to connect to the AWS S3 interface.

[0013] The calculation module is used to calculate the original hash value of the original stored file and can automatically recalculate the hash value of the renamed stored file;

[0014] The data balancing module allows for customization and modification of the data balancing system, thereby modifying the commands issued by the data balancing module and using these commands to perform data balancing and rebalancing on data volumes.

[0015] The link creation module can create file links corresponding to storage files in the data volume corresponding to the calculated hash value.

[0016] The index module can point the file links corresponding to the stored files to the stored files;

[0017] The GlusterFS distributed storage unit serves as the storage backend for the data balancing system.

[0018] Furthermore, the file path and file name of the original storage file are input into the Davies-Meyer hash algorithm, the calculation module calculates the hash value, and the data volume where the original storage file is located is determined according to the hash range division recorded in the data volume extension attribute.

[0019] Furthermore, the calculation module automatically receives the file path and file name of the renamed storage file, recalculates the new hash value according to the Davies-Meyer hash algorithm, and determines the data volume where the renamed storage file is located based on the hash range division recorded in the data volume extension attribute.

[0020] Furthermore, the data balancing system also includes a deletion module for deleting old file links.

[0021] Furthermore, the specific modifications to the data balancing command are as follows:

[0022] gluster v rebalance<VOL_NAME> start & gluster v rebalance<VOL_NAME> Start force.

[0023] A method for a data balancing system based on GlusterFS, comprising the following steps:

[0024] S1. The calculation module calculates a hash value based on the name of the original storage file. Assume that the hash value of file1 is 0x22222222.

[0025] S2. Based on the hash value calculated in S1, find the data volume where it is located, and create a link module to create a file link a corresponding to the original storage file in that data volume;

[0026] S3. The index module points file link 'a' to the original storage file;

[0027] S4. When the name of the original storage file in the data volume created inside the GlusterFS distributed storage unit is changed, but the actual content of the original storage file itself has not changed, the computing module automatically recalculates the hash value of the file. Assume that the hash value at this time is 0xDDDDDDDD.

[0028] S5. The delete module removes file link 'a';

[0029] S7. Based on the hash value calculated in S4, find the data volume where it is located, and create the link module to create the file link b corresponding to the renamed storage file in the data volume;

[0030] S8. The index module points the file link b to the original storage file.

[0031] This invention provides an improved data balancing method and system based on GlusterFS. Compared with the prior art, the improved data balancing algorithm of this invention can eliminate unnecessary data balancing movements, reduce storage load, reduce performance overhead, and reduce unnecessary hardware consumption. Attached Figure Description

[0032] The present invention will be further explained below with reference to the accompanying drawings and embodiments:

[0033] Figure 1 It is the data balancing algorithm in the prior art of GlusterFS;

[0034] Figure 2 This invention relates to the GlusterFS data balancing algorithm's handling of filename changes;

[0035] Figure 3 This is a data balancing processing method for filename changes under the new algorithm of this invention. Detailed Implementation

[0036] The present invention will now be described in detail, and the technical solutions in the embodiments of the present invention will be clearly and completely described. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0037] This invention provides an improved data balancing method and system based on GlusterFS, such as... Figure 3 As shown, it includes:

[0038] GlusterFS distributed storage units are deployed within the cluster management unit, and data volumes can be created within the distributed storage units.

[0039] The AWS S3 interface processing layer is located at the top layer of the data balancing system and is used to connect to the AWS S3 interface.

[0040] The calculation module is used to calculate the original hash value of the original stored file and can automatically recalculate the hash value of the renamed stored file;

[0041] The data balancing module allows for customization and modification of the data balancing system, thereby modifying the commands issued by the data balancing module and using these commands to perform data balancing and rebalancing on data volumes.

[0042] The link creation module can create file links corresponding to storage files in the data volume corresponding to the calculated hash value.

[0043] The index module can point the file links corresponding to the stored files to the stored files;

[0044] The GlusterFS distributed storage unit serves as the storage backend for the data balancing system.

[0045] Specifically, the file path and file name of the original storage file are input into the Davies-Meyer hash algorithm, the calculation module calculates the hash value, and the data volume where the original storage file is located is determined according to the hash range division recorded in the data volume extension attribute.

[0046] Specifically, the calculation module automatically receives the file path and file name of the renamed storage file, recalculates the new hash value according to the Davies-Meyer hash algorithm, and determines the data volume where the renamed storage file is located based on the hash range division recorded in the data volume extension attribute.

[0047] Specifically, the data balancing system also includes a deletion module for deleting old file links.

[0048] Specifically, the relevant processing for modifying the data balancing command is as follows:

[0049] gluster v rebalance<VOL_NAME> start & gluster v rebalance<VOL_NAME> Start force.

[0050] A method for a data balancing system based on GlusterFS, comprising the following steps:

[0051] S1. The calculation module calculates a hash value based on the name of the original storage file. Assume that the hash value of file1 is 0x22222222.

[0052] S2. Based on the hash value calculated in S1, find the data volume where it is located, and create a link module to create a file link a corresponding to the original storage file in that data volume;

[0053] S3. The index module points file link 'a' to the original storage file;

[0054] S4. When the name of the original storage file in the data volume created inside the GlusterFS distributed storage unit is changed, but the actual content of the original storage file itself has not changed, the computing module automatically recalculates the hash value of the file. Assume that the hash value at this time is 0xDDDDDDDD.

[0055] S5. The delete module removes file link 'a';

[0056] S7. Based on the hash value calculated in S4, find the data volume where it is located, and create the link module to create the file link b corresponding to the renamed storage file in the data volume;

[0057] S8. The index module points the file link b to the original storage file.

[0058] In the existing GlusterFS, suppose there is a 200MB file A, and assume the number of data volumes remains unchanged (i.e., no expansion or contraction). If only file A is renamed, ensuring the actual content of file A remains unchanged, the renaming will change its hash value. If the new hash value of the renamed file A differs from the current data volume, then file A should be migrated to the correct data volume. However, the actual content of file A has not been altered, and the storage environment (physical expansion or contraction) has not changed. Yet, the renamed file A has been moved, unnecessarily increasing the storage load and disk wear. The larger the file, the greater the storage load and disk wear.

[0059] In the system of this invention, when the name of file A changes but the actual content of file A remains unchanged, the system simply creates a file link in the corresponding data volume through the link creation module. This link may be only 1KB in size. The file A with the changed name is not moved; only the 1KB file link is moved, and this file link always points to file A. Through the improved data balancing algorithm of this invention, unnecessary data balancing movements can be eliminated, reducing storage load, performance overhead, and unnecessary hardware consumption.

[0060] If file A is renamed again, the calculation module recalculates the hash value of the stored file, the deletion module deletes the old file link, and the link creation module creates a file link corresponding to the renamed stored file based on the new hash value in the data volume where it is located. This new file link always points to file A.

Claims

1. A data balancing system based on GlusterFS, characterized in that, include: GlusterFS distributed storage units are deployed within the cluster management unit, and data volumes can be created within the distributed storage units. The AWS S3 interface processing layer is located at the top layer of the data balancing system and is used to connect to the AWS S3 interface. The calculation module is used to calculate the original hash value of the original stored file and can automatically recalculate the hash value of the renamed stored file; The data balancing module allows for customization and modification of the data balancing system, thereby modifying the commands issued by the data balancing module and using these commands to perform data balancing and rebalancing on data volumes. The link creation module can create file links corresponding to storage files in the data volume corresponding to the calculated hash value. The index module can point the file links corresponding to the stored files to the stored files; The GlusterFS distributed storage unit serves as the storage backend for the data balancing system. The file path and file name of the original storage file are input into the Davies-Meyer hash algorithm. The calculation module calculates the hash value and determines the data volume where the original storage file is located based on the hash range division recorded in the data volume extension attribute. The calculation module automatically receives the file path and file name of the renamed storage file, and recalculates the new hash value according to the Davies-Meyer hash algorithm. It then determines the data volume where the renamed storage file is located based on the hash range division recorded in the data volume extension attribute. The data balancing system also includes a deletion module for deleting old file links; When the name of a storage file changes but the actual content of the storage file itself remains unchanged, the data balancing system simply creates a file link in the corresponding data volume through the link creation module. The storage file whose name has been changed will not be moved; only the file link will be moved, and the file link always points to the storage file.

2. The data balancing system based on GlusterFS according to claim 1, characterized in that, The specific steps for modifying the data balancing command are as follows: gluster v rebalance<VOL_NAME> start & gluster v rebalance<VOL_NAME> Start force.

3. A method for a data balancing system based on GlusterFS, comprising a data balancing system based on GlusterFS as described in claim 1 or 2, characterized in that, The specific steps are as follows: S1. The calculation module calculates a hash value based on the name of the original storage file; S2. Based on the hash value calculated in S1, find the data volume where it is located, and create a link module to create a file link a corresponding to the original storage file in that data volume; S3. The index module points file link 'a' to the original storage file; S4. When the name of the original storage file in the data volume created inside the GlusterFS distributed storage unit is changed, but the actual content of the original storage file itself has not changed, the computing module automatically recalculates the hash value of the file. S5. The delete module removes file link 'a'; S6. Based on the hash value calculated in S4, find the data volume where it is located, and create the link module to create the file link b corresponding to the renamed storage file in the data volume; S7. The index module points the file link b to the original storage file.

Images