Distributed file storage system, method, medium and computer program product

Abstract

The invention provides a distributed file storage system and method, a medium and a computer program product, and the system comprises a server node which is used for storing files; the address configurator is used for forming an address ring consisting of all IP addresses according to the configured IP address length; the node manager is used for managing the server nodes and comprises the following steps of: generating a unique hash value for the server nodes by using a predetermined consistent hash function; saving a hash value range of a file stored by the server node; the router is used for maintaining a routing table and comprises a server node, an IP address, a hash value and a hash value range of a stored file; and the file manager is used for maintaining file information on all server nodes and migrating files on the server nodes according to the routing table. By means of the technical scheme, distributed file storage can be simplified, the storage space can be conveniently and dynamically expanded, and the whole system is slightly influenced when the nodes are added and leave.

Description

Technical Field

[0001] The present invention relates to the field of computer storage technology, and in particular to a distributed file storage system, method, medium and computer program product. Background Art

[0002] With the rapid development of the information age, the amount of data has exploded, and traditional file storage systems can no longer meet the needs of large-scale, high-concurrency data storage. The current distributed file storage methods have the characteristics of being unable to be decentralized, complex routing information, and complex storage location calculations, which are not conducive to the simple and efficient implementation of distributed file storage. Therefore, designing an efficient, scalable, and secure file storage system has become an urgent problem to be solved. Summary of the invention

[0003] In view of the above problems in the prior art, embodiments of the present invention provide a distributed file storage system, method, medium and computer program product.

[0004] In order to achieve the above object, on the one hand, a distributed file storage system is provided, comprising:

[0005] at least one server node, for storing files;

[0006] An address configurator, used to form an address ring consisting of all IP addresses according to the configured IP address length, and configure an IP address for each server node in the distributed file storage system;

[0007] A node manager is used to manage server nodes in a distributed file storage system, wherein the management includes: using a predetermined consistent hash function to generate a unique hash value for each server node, wherein the hash value of the server node corresponds to the unique identifier of the server node; managing the IP address of each server node; and saving the hash value range of the files stored by each server node, wherein the hash values ​​of all server nodes divide the address ring into N segments, and the hash value range corresponding to each segment is the hash value range of the files stored by the corresponding server node, wherein N is the total number of server nodes currently included in the distributed file storage system;

[0008] A router, used to maintain a routing table, wherein the routing table includes the following information of all server nodes in the distributed storage system: an IP address of the server node, a hash value of the server node, and a hash value range of a file stored in the server node;

[0009] The file manager is used to maintain the file information on all the server nodes and migrate the files on the server nodes according to the routing table, wherein the file information includes: a hash value of the file, a directory to which the file belongs, and the file content.

[0010] Preferably, in the distributed file storage system, when a file needs to be stored in the distributed file storage system, the file manager is used to:

[0011] Calculate a first hash value of the file using the consistent hash function according to the attribute information of the file;

[0012] Determine whether the first hash value is within a hash value range of files stored by a first server node associated with the file; if so, store the file in the first server node; otherwise, search the routing table for a second server node whose hash value range of stored files includes the first hash value, and store the file in the second server node.

[0013] Preferably, in the distributed file storage system, when retrieving a file in the distributed file storage system, the file manager is used to:

[0014] Calculate a first hash value of the file using the consistent hash function according to the attribute information of the file;

[0015] Determine whether the first hash value is within a hash value range of files stored by a first server node associated with the file; if so, search for the file from the first server node; otherwise, search from the routing table for a second server node whose hash value range for stored files includes the first hash value, and search for the file from the second server node.

[0016] Preferably, the distributed file storage system, wherein:

[0017] The node manager is used for, when a newly added third server node joins the distributed file storage system, using the middle value of the stored file hash range corresponding to the fourth server node associated with the third server node at the time of joining as the hash value of the third server node, and equally dividing the original stored file hash range of the fourth server node into a first part and a second part, wherein the first part between the hash value of the fourth server node and the hash value of the third server node is used as the hash value range of the files currently stored by the fourth server node, and the remaining second part is used as the hash value range of the files stored by the third server node;

[0018] The router is used to update the routing table according to the hash value of the third server node and the hash value range of the stored file and the hash value range of the stored file of the fourth server node, and broadcast the updated routing table;

[0019] The file manager is used to migrate files stored in the server node according to the updated routing table.

[0020] Preferably, the distributed file storage system, wherein:

[0021] The node manager is used to allocate the hash value range of the files stored by the server node that has left the distributed storage system to the two server nodes that are adjacent to the server node before and after the server node that has left the distributed storage system according to a predetermined allocation principle, and update the hash value range of the files stored by the two server nodes that are adjacent to the server node before and after the server node that has left the distributed storage system;

[0022] The router is used to update the routing table according to the hash value range of the stored files of the two adjacent server nodes before and after the leaving server node after the update, and broadcast the updated routing table;

[0023] The file manager is used to migrate the files in the leaving server node to two server nodes adjacent to and before the leaving server node according to the updated routing table.

[0024] Preferably, the distributed file storage system, wherein:

[0025] The node manager is used to determine the fifth server node that currently stores the largest number of files according to the routing table when a new third server node is added to the distributed file storage system, and determine the hash value of the third server node and the hash range of the stored files according to the hash range of the stored files currently corresponding to the fifth server node, and adjust the hash range of the stored files corresponding to the fifth server node;

[0026] The router is used to update the routing table according to the hash value of the third server node and the hash value range of the stored file and the adjusted hash value range of the stored file of the fifth server node, and broadcast the updated routing table;

[0027] The file manager is used to migrate files stored in the server node according to the updated routing table.

[0028] Preferably, in the distributed file storage system, determining the hash value of the third server node and the hash range of the stored files according to the hash range of the stored files currently corresponding to the fifth server node and adjusting the hash range of the stored files corresponding to the fifth server node comprises:

[0029] The middle value of the hash range of the stored files corresponding to the fifth server node is used as the hash value of the third server node, and the original hash range of the stored files of the fifth server node is equally divided into a first part and a second part, wherein the first part from the hash value of the fifth server node to the hash value of the third server node is used as the hash value range of the files currently stored by the fifth server node, and the remaining second part is used as the hash value range of the files stored by the third server node.

[0030] On the other hand, a method for implementing distributed file storage using any of the distributed file storage systems described above is provided.

[0031] On the other hand, a computer-readable storage medium is provided, wherein at least one program is stored in the storage medium, and the at least one program is executed by a processor to implement the steps of the method described above.

[0032] On the other hand, a computer program product is provided, comprising a computer program, characterized in that when the computer program is executed by a processor, the steps of the method described above are implemented.

[0033] The above technical solution has the following technical effects:

[0034] In the technical solution of the embodiment of the present invention, all IP addresses in the distributed file storage system form an address ring, and a consistent hash function or algorithm is used to generate a unique hash value for the server node, the hash value corresponds to the unique identifier of the server node, and each server node has a corresponding hash range of stored files, wherein the hash values ​​of all server nodes divide the address ring into N segments, and the hash value range corresponding to each segment is the hash value range of the file stored by the corresponding server node; the routing table maintained on the router stores the hash value of the server node and the hash value range of the stored file; the file information maintained in the file manager includes the file hash value; the technical solution of the embodiment of the present invention simplifies the distributed file storage by allocating the hash value range of the stored file to the server node, and can conveniently and dynamically expand the storage space;

[0035] In a further embodiment, when performing file storage and retrieval, the server node for file storage is determined by checking the hash value range of the file stored by the server node, thereby improving the efficiency of file storage and retrieval.

[0036] In a further embodiment, when a server node joins or leaves, the hash value of the server node is determined according to a predetermined principle, and file migration is performed according to the corresponding adjusted storage file hash value range, simplifying the logic of node joining or leaving. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 A schematic diagram of the allocation of hash value ranges of files stored by server nodes in a distributed storage system according to an embodiment of the present invention;

[0038] Figure 2 A schematic diagram of a process for storing files using a distributed storage system according to an embodiment of the present invention;

[0039] Figure 3 The figure is a flowchart of file retrieval using a distributed storage system according to an embodiment of the present invention. DETAILED DESCRIPTION

[0040] To further illustrate the various embodiments, the present invention provides drawings. These drawings are part of the disclosure of the present invention, which are mainly used to illustrate the embodiments and can be used in conjunction with the relevant descriptions in the specification to explain the operating principles of the embodiments. With reference to these contents, a person of ordinary skill in the art should be able to understand other possible implementations and advantages of the present invention. The components in the figures are not drawn to scale, and similar component symbols are generally used to represent similar components.

[0041] The present invention will now be further described with reference to the accompanying drawings and specific implementation methods.

[0042] Embodiment 1:

[0043] An embodiment of the present invention provides a distributed file storage system, including:

[0044] at least one server node, for storing files;

[0045] An address configurator is used to form an address ring consisting of all IP addresses according to the configured IP address length, and configure an IP address for each server node in the distributed file storage system; the address configurator manages all address lengths of the distributed file storage system; in a specific implementation, the default IP address length is 128 bits, and the IP address length used can be modified in the initial setting according to the needs;

[0046] A node manager is used to manage server nodes in a distributed file storage system, wherein the management includes: using a predetermined consistent hash function or algorithm to generate a unique hash value for each server node, wherein the hash value of the server node corresponds to the unique identifier of the server node; managing the IP address of each server node; and saving the hash value range of the files stored by each server node, wherein the hash values ​​of all server nodes divide the address ring into N segments, and the hash value range corresponding to each segment is the hash value range of the files stored by the corresponding server node, wherein N is the total number of server nodes currently included in the distributed file storage system;

[0047] A router is used to maintain a routing table, which includes the following information of all server nodes in the distributed storage system: the IP address of the server node, the hash value of the server node, and the hash value range of the files stored in the server node;

[0048] The file manager is used to maintain the file information on all server nodes and migrate the files on the server nodes according to the routing table, wherein the file information includes: the hash value of the file, the directory to which the file belongs, and the file content.

[0049] In a specific implementation, an embodiment of the present invention adopts a consistent hash function or algorithm and also uses a modulo method, such as taking the modulo of the number of servers; the consistent hash function or algorithm constructs the entire address ring into a hash space, also called a hash ring; when hashing the server, specific keywords such as the server's IP address or host name can be selected to determine the position of the server node on the hash ring, i.e., the address ring.

[0050] Figure 1 FIG. 1 is a schematic diagram of the allocation of hash value ranges of files stored in a server node in a system according to an embodiment of the present invention. Figure 1 , assuming that the current system contains N server nodes: server node 1 to server node N; the hash values ​​of these N server nodes correspond to a point in the address ring respectively; the hash value range of the file stored in server node 1 is the range between the hash value of server node 1 on the ring and the hash value of its next server node, namely server node 2.

[0051] In a specific implementation, a newly added server node generates a unique node ID through a node manager using a consistent hash function. When there is no server node initially, the ID of the first server node connected is randomly generated; subsequent nodes will be associated with a server node when connected, based on the hash value of the associated server node and the hash range of the files stored in it; in a further implementation, the hash value of the newly added server node and the hash value range of the stored files are jointly determined based on the hash value or identifier of the associated server node and the next server node of the associated node in the routing table.

[0052] Figure 2 FIG. 1 is a flow chart of using a distributed storage system according to an embodiment of the present invention to store files. Figure 2 ,When a file needs to be stored in a distributed file storage system, the steps of file storage include:

[0053] According to the attribute information of the file, such as the key of the file data, a first hash value of the file is calculated using a consistent hash function;

[0054] Determine whether the first hash value is within the hash value range of the file stored by the first server node associated with the file; if so, store the file in the first server node; otherwise, search the routing table for a second server node whose hash value range for storing the file includes the first hash value, and store the file in the second server node; that is, calculate or determine the server node to which the file belongs, i.e., the second server node, according to the routing table, and store the file in the determined server node.

[0055] In a specific implementation, the above file storage steps are implemented through a node manager.

[0056] Figure 3 FIG. 1 is a flow chart of using a distributed storage system according to an embodiment of the present invention to store files. Figure 3 , the steps of retrieving files in a distributed file storage system include:

[0057] Calculate the first hash value of the file using a consistent hash function according to the attribute information of the file;

[0058] Determine whether the first hash value is within the hash value range of the file stored by the first server node associated with the file; if so, search for the file from the first server node; otherwise, search for the second server node whose hash value range of the stored file includes the first hash value from the routing table, and search for the file from the second server node; that is, calculate or determine the server node to which the file belongs, that is, the second server node, according to the routing table, obtain the file from the second server node, and end the process; further, if the file acquisition fails, feedback the acquisition exception information.

[0059] In a specific implementation, the above-mentioned file retrieval step is implemented through a file manager.

[0060] The distributed file storage system of the embodiment of the present invention includes the following processing flow when a server node joins:

[0061] When a newly added third server node joins the distributed file storage system, the middle value of the hash range of the stored files corresponding to the fourth server node associated with the third server node when joining is used as the hash value of the third server node, and the original hash range of the stored files of the fourth server node is evenly divided into a first part and a second part, wherein the first part from the hash value of the fourth server node to the hash value of the third server node is used as the hash value range of the files currently stored by the fourth server node, and the remaining second part is used as the hash value range of the files stored by the third server node; in a specific implementation, the newly added third server node has an associated server node, namely the fourth server node, when joining; if there is no associated node, an associated server node is configured for it, such as configuring a neighboring server node as the fourth server node; the hash value range of the files stored by the fourth server node is determined by its hash value with the next node after it in the routing table, such as Figure 1 , the hash value range of the file stored in server node 4 is determined on the ring by the hash value of the file stored in server node 4 and server node 5; in a specific implementation, this step can be implemented by a node manager;

[0062] updating the routing table according to the hash value of the third server node and the hash value range of the stored file and the hash value range of the stored file of the fourth server node, and broadcasting the updated routing table; in a specific implementation, this step is implemented by a router;

[0063] Migrate the files stored in the server nodes according to the updated routing table, such as migrating some files on the fourth server to the newly added third server; in a specific implementation, this step is implemented by a file manager.

[0064] In another implementation, when a newly added third server node joins the distributed file storage system, the fifth server node that currently stores the largest number of files is determined according to the routing table, and the hash value of the third server node and the hash range of the stored files are determined and the hash range of the stored files corresponding to the fifth server node are adjusted according to the hash range of the stored files currently corresponding to the fifth server node; then, the router updates the routing table according to the hash value of the third server node and the hash value range of the stored files and the hash value range of the stored files after adjustment of the fifth server node, and broadcasts the updated routing table; then, the file manager migrates the files stored in the server nodes according to the updated routing table.

[0065] In yet another implementation, determining the hash value of the third server node and the hash range of the stored file according to the hash range of the stored file currently corresponding to the fifth server node and adjusting the hash range of the stored file corresponding to the fifth server node includes:

[0066] The middle value of the hash range of the stored files corresponding to the fifth server node is used as the hash value of the third server node, and the original hash range of the stored files of the fifth server node is equally divided into a first part and a second part, wherein the first part from the hash value of the fifth server node to the hash value of the third server node is used as the hash value range of the files currently stored by the fifth server node, and the remaining second part is used as the hash value range of the files stored by the third server node.

[0067] The distributed file storage system of the embodiment of the present invention includes the following processing flow when a server node leaves:

[0068] When a server node leaves the distributed storage system, the hash value range of the file stored in the server node is distributed to the two server nodes adjacent to the server node before and after the server node according to a predetermined distribution principle, and the hash value range of the file stored in the two server nodes adjacent to the server node before and after the server node is updated; in a specific implementation, this step is implemented by a node manager; in a specific implementation, the predetermined distribution principle is equal distribution, that is, the hash value range of the file stored in the server node is evenly distributed to the two server nodes adjacent to the server node before and after in the routing table; in other implementations, in addition to equal distribution, the distribution ratio can be determined according to the file load of the adjacent nodes, and the file with a large load can be distributed less; the file with a small load can be distributed more; ordinary technicians in this field can adopt the required distribution principle without paying creativity;

[0069] Update the routing table according to the hash value range of the stored files of the two adjacent server nodes before and after the updated leaving server node, and broadcast the updated routing table; in a specific implementation, this step is implemented by a router;

[0070] The files in the leaving server node are migrated to two adjacent server nodes before and after the leaving server node according to the updated routing table; in a specific implementation, this step is implemented by a file manager.

[0071] The system of the embodiment of the present invention can quickly locate the node server and file storage address by using a consistent hash function for all addresses, server nodes and files, thereby facilitating file storage and retrieval; servers can be quickly added and left through routers and file managers without affecting a large range of server nodes; efficient and convenient distributed file storage is achieved.

[0072] Embodiment 2:

[0073] The present invention provides a method for implementing distributed file storage using the above-mentioned distributed file storage system, and the processes of file storage, file retrieval, and joining and leaving of server nodes are as described above.

[0074] Embodiment three:

[0075] The present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above method in the embodiment of the present invention are implemented.

[0076] If the module / unit integrated in the computer unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory) and software distribution medium, etc. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction.

[0077] Embodiment 4:

[0078] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the steps of the method described above are implemented.

[0079] Although the present invention has been specifically shown and described in conjunction with the preferred embodiments, it should be understood by those skilled in the art that various changes may be made to the present invention in form and details without departing from the spirit and scope of the present invention as defined by the appended claims, all of which are within the scope of protection of the present invention.

Claims

1. A distributed file storage system, characterized in that: include: at least one server node, for storing files; An address configurator, used to form an address ring consisting of all IP addresses according to the configured IP address length, and configure an IP address for each server node in the distributed file storage system; A node manager is used to manage server nodes in a distributed file storage system, wherein the management includes: using a predetermined consistent hash function to generate a unique hash value for each server node, wherein the hash value of the server node corresponds to the unique identifier of the server node; managing the IP address of each server node; and saving the hash value range of the files stored by each server node, wherein the hash values ​​of all server nodes divide the address ring into N segments, and the hash value range corresponding to each segment is the hash value range of the files stored by the corresponding server node, wherein N is the total number of server nodes currently included in the distributed file storage system; A router, used to maintain a routing table, wherein the routing table includes the following information of all server nodes in the distributed storage system: an IP address of the server node, a hash value of the server node, and a hash value range of a file stored in the server node; The file manager is used to maintain the file information on all the server nodes and migrate the files on the server nodes according to the routing table, wherein the file information includes: a hash value of the file, a directory to which the file belongs, and the file content.

2. The distributed file storage system according to claim 1, characterized in that: When a file needs to be stored in the distributed file storage system, the file manager is used to: Calculate a first hash value of the file using the consistent hash function according to the attribute information of the file; Determining whether the first Hash value is within a range of Hash values ​​of files stored in a first server node associated with the file; If yes, storing the file in the first server node; Otherwise, search the routing table for a second server node whose hash value range of the stored file includes the first hash value, and store the file in the second server node.

3. The distributed file storage system according to claim 1, characterized in that: When retrieving files in a distributed file storage system, the file manager is used to: Calculate a first hash value of the file using the consistent hash function according to the attribute information of the file; Determining whether the first Hash value is within a range of Hash values ​​of files stored in a first server node associated with the file; If yes, search the first server node for the file; otherwise, search the routing table for a second server node whose hash value range of the stored file includes the first hash value, and search the second server node for the file.

4. The distributed file storage system according to claim 1, characterized in that: The node manager is used for, when a newly added third server node joins the distributed file storage system, using the middle value of the stored file hash range corresponding to the fourth server node associated with the third server node at the time of joining as the hash value of the third server node, and equally dividing the original stored file hash range of the fourth server node into a first part and a second part, wherein the first part between the hash value of the fourth server node and the hash value of the third server node is used as the hash value range of the files currently stored by the fourth server node, and the remaining second part is used as the hash value range of the files stored by the third server node; The router is used to update the routing table according to the hash value of the third server node and the hash value range of the stored file and the hash value range of the stored file of the fourth server node, and broadcast the updated routing table; The file manager is used to migrate files stored in the server node according to the updated routing table.

5. The distributed file storage system according to claim 1, characterized in that: The node manager is used to allocate the hash value range of the files stored by the server node that has left the distributed storage system to the two server nodes that are adjacent to the server node before and after the server node that has left the distributed storage system according to a predetermined allocation principle, and update the hash value range of the files stored by the two server nodes that are adjacent to the server node before and after the server node that has left the distributed storage system; The router is used to update the routing table according to the hash value range of the stored files of the two adjacent server nodes before and after the leaving server node after the update, and broadcast the updated routing table; The file manager is used to migrate the files in the leaving server node to two server nodes adjacent to and before the leaving server node according to the updated routing table.

6. The distributed file storage system according to claim 1, characterized in that: The node manager is used to determine the fifth server node that currently stores the largest number of files according to the routing table when a new third server node is added to the distributed file storage system, and determine the hash value of the third server node and the hash range of the stored files according to the hash range of the stored files currently corresponding to the fifth server node, and adjust the hash range of the stored files corresponding to the fifth server node; The router is used to update the routing table according to the hash value of the third server node and the hash value range of the stored file and the adjusted hash value range of the stored file of the fifth server node, and broadcast the updated routing table; The file manager is used to migrate files stored in the server node according to the updated routing table.

7. The distributed file storage system according to claim 6, characterized in that: Determining the hash value of the third server node and the hash range of the stored file according to the hash range of the stored file currently corresponding to the fifth server node and adjusting the hash range of the stored file corresponding to the fifth server node includes: The middle value of the hash range of the stored files corresponding to the fifth server node is used as the hash value of the third server node, and the original hash range of the stored files of the fifth server node is equally divided into a first part and a second part, wherein the first part from the hash value of the fifth server node to the hash value of the third server node is used as the hash value range of the files currently stored by the fifth server node, and the remaining second part is used as the hash value range of the files stored by the third server node.

8. A method for implementing distributed file storage using the distributed file storage system according to any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that: The storage medium stores at least one program, and the at least one program is executed by a processor to implement the steps of the method according to claim 8.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to claim 8 are implemented.

Images