A method and system for enhancing the availability of a distributed file system

Abstract

The present application discloses a method and system for enhancing the availability of a distributed file system. The method comprises: an input step: writing any data into a fastDFS distributed file system through a client; a cache layer creation step: creating a cache layer between a tracking server and a storage server of the fastDFS distributed file system; a processing step: processing the data through the cache layer; an output step: after the data is processed by the cache layer and uploaded to each storage server of the fastDFS distributed file system, a successful upload mark is returned to the client. The present invention solves the problem of data loss when writing data in the fastDFS distributed file system.

Description

Technical Field

[0001] The invention belongs to the field of methods for enhancing the availability of distributed file systems, and in particular relates to a method for enhancing the availability of distributed file systems. Background Art

[0002] The FastDFS distributed file system is committed to achieving high availability and high concurrency of file operations. It is particularly suitable for the storage operations of small and medium-sized files. It is widely used by major domestic companies for its three unique features: lightweight, grouped storage, and peer-to-peer structure.

[0003] FastDFS consists of three parts: tracker server, storage server and client. The tracker server is mainly responsible for scheduling and load balancing. It records the status information of all storage groups and storage servers in the cluster in memory and is the hub for interaction between the client and the data server. It is responsible for managing all storage servers and groups. After startup, each storage will connect to the Tracker to inform the group it belongs to and maintain periodic heartbeats. The tracker establishes a mapping table of group==>[storage server list] based on the heartbeat information of the storage. The storage server is responsible for storing files and file attributes (metadata). As the initiator of the business request, the client uses the TCP / IP protocol to interact with the tracker server or storage node through a proprietary interface.

[0004] The high availability of distributed file systems has always been a highly concerned issue. How to enhance the availability of distributed file systems and improve system performance is of great significance. Although the fastDFS distributed file system has the advantages of low machine configuration requirements and fast upload speed, in fastDFS, within the time window from writing files to synchronizing to other storages in the group, once the source storage fails, user data may be lost, and data loss is usually unacceptable for the storage system. At the same time, fastDFS selects storage nodes in a polling manner when synchronizing data of newly added nodes, and the node may store a large amount of data or be in use, so the synchronization efficiency will be very low. It can be seen that the existing fastDFS distributed file system does not reasonably solve the above problems, and its synchronization method and synchronization mechanism still need to be further improved to avoid the problems of low efficiency and poor stability that may be caused by traditional methods.

[0005] FastDFS distributed file system upload mechanism schematic diagram, such as Figure 1As shown:

[0006] The StorageServer will upload status information to the tracker at regular intervals. When the client initiates an upload connection request, the Trackerserver will return an available storage IP and port to the client. At this time, the client can directly connect to the StorageServer through the IP and port to upload the file. In the prior art, when the client writes a file to the fastDFS distributed file system, the client considers that the file is written successfully when it writes the file to a storage server in the group. After the storage server writes the file, the background thread will synchronize the file to other storageservers in the same group.

[0007] The original synchronization scheme of the system is to select an existing storage node in a round-robin manner and use the existing storage node to synchronize files with the new storage node one by one. Summary of the invention

[0008] The embodiment of the present application provides a method for enhancing the availability of a distributed file system, so as to at least solve the problem of subjective factors in the related art.

[0009] The present invention provides a method for enhancing the availability of a distributed file system, which comprises:

[0010] Input step: Write any data to the fastDFS distributed file system through the client;

[0011] Creating a cache layer step: creating a cache layer between the tracking server and the storage server of the fastDFS distributed file system;

[0012] Processing step: processing the data through the cache layer;

[0013] Output step: After the data is processed by the cache layer and uploaded to each storage server of the fastDFS distributed file system, a successful upload flag is returned to the client.

[0014] In the above availability enhancement method, the step of creating a cache layer includes:

[0015] The cache layer is a storage pool created on an SSD cluster.

[0016] In the above-mentioned method for enhancing availability, the processing step comprises:

[0017] Write-back mode cache layer processing steps: write data to the cache layer through the client, and then immediately receive the write confirmation, the cache layer migrates the data from the cache layer to the storage server and synchronizes it on the remaining storage servers in the same group, and finally the cache agent deletes the data from the cache layer; and / or;

[0018] Read-only mode cache layer processing steps: When processing a read operation from a client, the cache layer agent copies the requested data from the storage layer to the cache layer and deletes inactive objects from the cache layer.

[0019] In the above-mentioned method for enhancing availability, the processing step comprises:

[0020] The storage server informs the cache layer of the status information of all current storage nodes and assigns different weights to each storage node.

[0021] In the above-mentioned method for enhancing availability, the processing step comprises:

[0022] A storage node is selected according to the weight to complete the synchronization task.

[0023] The present invention also includes a system for enhancing the availability of a distributed file system, which includes:

[0024] Input module: writes any data to the fastDFS distributed file system through the client;

[0025] Creating a cache layer module: creating a cache layer between the tracking server and the storage server of the fastDFS distributed file system;

[0026] Processing module: processing the data through the cache layer;

[0027] Output module: After the data is processed by the cache layer and uploaded to each storage server of the fastDFS distributed file system, a successful upload mark is returned to the client.

[0028] The above availability enhancement system, wherein the cache layer creation module comprises:

[0029] The cache layer is a storage pool created on an SSD cluster.

[0030] The above-mentioned availability enhancement system, wherein the processing module comprises:

[0031] Write-back mode cache layer processing unit: writes data to the cache layer through the client, and then immediately receives the write confirmation, the cache layer migrates the data from the cache layer to the storage server and synchronizes it on the remaining storage servers in the same group, and finally the cache agent deletes the data from the cache layer; and / or;

[0032] Read-only mode cache layer processing unit: When processing a read operation from a client, the cache layer proxy copies the requested data from the storage layer to the cache layer and deletes inactive objects from the cache layer.

[0033] In the above-mentioned availability enhancement system, the processing module is used for:

[0034] The storage server informs the cache layer of the status information of all current storage nodes and assigns different weights to each storage node.

[0035] In the above-mentioned availability enhancement system, the processing module is used for:

[0036] A storage node is selected according to the weight to complete the synchronization task.

[0037] The present invention also provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method for enhancing the availability of a distributed file system as described in any one of the above when executing the computer program.

[0038] The present invention also provides a storage medium having a computer program stored thereon, characterized in that when the program is executed by a processor, the method for enhancing the availability of a distributed file system as described in any one of the above is implemented.

[0039] The beneficial effects of the present invention are:

[0040] 1. The present invention solves the problem of data loss when writing data in the fastDFS distributed file system.

[0041] 2. The present invention also solves the synchronization problem of newly added nodes, avoiding the problems of low efficiency and poor stability that may be caused by traditional methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] The drawings described herein are used to provide further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute improper limitations on the present application.

[0043] In the attached picture:

[0044] Figure 1 is a flow chart of a method for enhancing the availability of a distributed file system;

[0045] Figure 2 yes Figure 1 The step-by-step flow chart of step S3;

[0046] Figure 3 This is the schematic diagram of the upload mechanism of the fastDFS distributed file system;

[0047] Figure 4 This is the schematic diagram of the fastDFS distributed file system;

[0048] Figure 5 It is a schematic diagram of the structure of the availability enhancement system of the distributed file system;

[0049] Figure 6 is a framework diagram of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0050] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is described and illustrated below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. Based on the embodiments provided in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.

[0051] Obviously, the drawings described below are only some examples or embodiments of the present application. For ordinary technicians in this field, the present application can also be applied to other similar scenarios based on these drawings without creative work. In addition, it can also be understood that although the efforts made in this development process may be complicated and lengthy, for ordinary technicians in this field related to the content disclosed in this application, some changes in design, manufacturing or production based on the technical content disclosed in this application are just conventional technical means, and should not be understood as insufficient content disclosed in this application.

[0052] Reference to "embodiments" in this application means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those of ordinary skill in the art that the embodiments described in this application may be combined with other embodiments without conflict.

[0053] Unless otherwise defined, the technical terms or scientific terms involved in this application should be understood by people with ordinary skills in the technical field to which this application belongs. The words "one", "a", "a", "the" and the like involved in this application do not indicate a quantitative limitation, and may represent the singular or plural. The terms "include", "comprise", "have" and any of their variations involved in this application are intended to cover non-exclusive inclusions; for example, a process, method, system, product or device that includes a series of steps or modules (units) is not limited to the listed steps or units, but may also include steps or units that are not listed, or may also include other steps or units inherent to these processes, methods, products or devices. The words "connect", "connected", "coupled" and the like involved in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. The "multiple" involved in this application refers to two or more. "And / or" describes the association relationship of associated objects, indicating that there may be three relationships, for example, "A and / or B" can represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the objects before and after are in an "or" relationship. The terms "first", "second", "third", etc. involved in this application are only used to distinguish similar objects and do not represent a specific ordering of the objects.

[0054] The present invention is described in detail below in conjunction with the various embodiments shown in the accompanying drawings, but it should be noted that these embodiments are not limitations of the present invention, and any equivalent transformations or substitutions in functions, methods, or structures made by ordinary technicians in the field based on these embodiments are all within the scope of protection of the present invention.

[0055] Before elaborating on each embodiment of the present invention in detail, the core inventive concept of the present invention is summarized and elaborated in detail through the following several embodiments.

[0056] Please refer to Figure 1 , Figure 1 This is a flow chart of a method for enhancing the availability of a distributed file system. Figure 1 As shown, the method for enhancing the availability of a distributed file system of the present invention includes:

[0057] Input step S1: write any data to the fastDFS distributed file system through the client;

[0058] Creating a cache layer step S2: Creating a cache layer between the tracking server and the storage server of the fastDFS distributed file system;

[0059] Processing step S3: processing the data through the cache layer;

[0060] Output step S4: After the data is processed by the cache layer and uploaded to each storage server of the fastDFS distributed file system, a successful upload flag is returned to the client.

[0061] In the above method, the step of creating a cache layer comprises:

[0062] The cache layer is a storage pool created on an SSD cluster.

[0063] Please refer to Figure 2 , Figure 2 yes Figure 1 The step-by-step flow chart of step S3 in FIG. Figure 2 As shown, the processing step S3 includes:

[0064] Write-back mode cache layer processing step S31: write data to the cache layer through the client, and then immediately receive the write confirmation, the cache layer migrates the data from the cache layer to the storage server and synchronizes it on the remaining storage servers in the same group, and finally the cache proxy deletes the data from the cache layer;

[0065] Read-only mode cache layer processing step S32: When processing a read operation from a client, the cache layer agent copies the requested data from the storage layer to the cache layer, and deletes inactive objects from the cache layer.

[0066] Wherein, the processing step S3 includes:

[0067] The storage server informs the cache layer of the status information of all current storage nodes and assigns different weights to each storage node.

[0068] Wherein, the processing step S3 further includes:

[0069] A storage node is selected according to the weight to complete the synchronization task.

[0070] The following examples are given to specifically illustrate the method for enhancing the availability of a distributed file system of the present invention.

[0071] Embodiment 1:

[0072] The present invention improves the system availability performance by solving the problem of data loss during synchronization in the fastDFS distributed file system. When uploading data, after a client uploads a file to a storage server node, it will be informed that the file is stored successfully. Later, the storage will have a separate thread to synchronize to other storage services in the same group. If the storage data is damaged before synchronization, the data uploaded by the user will be permanently lost.

[0073] In order to solve the problem of data loss during data upload synchronization in the prior art, as well as the problem of low efficiency when selecting synchronization nodes in a round-robin manner when synchronizing data of newly added nodes, this paper proposes a method for enhancing the availability of a distributed file system, which enhances the availability performance of the distributed file system and improves the efficiency of data synchronization.

[0074] like Figure 3 The figure shows the existing architecture and working principle of the fastDFS distributed file system. Based on the original architecture, the present invention adds a cache layer between the tracker server and the storage server. The so-called cache layer is a storage pool created on a small SSD cluster. Then all client I / O operations are first processed by the cache pool, and then the data is written back to the existing data storage pool, so that the client can enjoy better performance on the cache pool, and the final data is still written to the storage server; in addition, the present invention also proposes a method for synchronizing files on the new storage node by using an improved optimization algorithm instead of the original round-robin algorithm for the new storage node.

[0075] The principle of the present invention is as follows Figure 4 As shown:

[0076] exist Figure 4 Under the working scheme shown, each storage will connect to the cache layer after startup, inform the group it belongs to and other information, and maintain periodic heartbeats. Then the cache layer connects to the tracking server and informs the tracking server of each update. When the client uploads a file to the fastDFS distributed file system, the tracking server stores the idle IP and port number in the storage layer, and then the client directly uploads the file to the cache layer. The cache layer then accesses the storage server storageserver based on the IP and port number, and only when the cache layer uploads the file to each storageserver does it return a successful upload flag to the client.

[0077] The cache layer can be configured in the following two modes:

[0078] 1) Write-back mode: When the cache pool is configured in write-back mode, the client writes data to the cache pool and immediately receives a write confirmation. The cache pool then migrates the data from the cache pool to the storage server and synchronizes it on other storages in the same group, and the cache proxy eventually deletes it from the cache tier. When processing read operations from the client, the cache tier proxy first migrates the data from the storage tier to the cache tier and then returns it to the client. The data is deleted from the cache pool only when it becomes inactive or cold data.

[0079] 2) Read-only mode: When the cache pool is configured in read-only mode, it is only applicable to processing client read operations. Client write operations do not involve the cache layer, and all client writes are completed on the storage layer. When processing read operations from clients, the cache tiering agent copies the requested data from the storage layer to the cache layer. The cache pool removes inactive objects from the cache pool based on the policy we configure for it.

[0080] In order to solve the data synchronization problem of newly added storage nodes, the storage server not only informs the cache layer of its existing basic information, but also informs the cache layer of the status information of all current storage nodes and assigns different weights to each storage node. When a new node is added to the storage system, according to the peer-to-peer idea in the fastDFS distributed file system, the new storage node must complete all synchronization work of the system's original files. Therefore, based on the different weights previously assigned to each storage node, a storage node with high performance can be selected to complete the synchronization task.

[0081] Embodiment 2:

[0082] Please refer to Figure 5 , Figure 5 Schematic diagram of the structure of the system for enhancing the availability of the distributed file system of the present invention. Figure 5 A distributed file system availability enhancement system of the present invention includes:

[0083] Input module: writes any data to the fastDFS distributed file system through the client;

[0084] Creating a cache layer module: creating a cache layer between the tracking server and the storage server of the fastDFS distributed file system;

[0085] Processing module: processing the data through the cache layer;

[0086] Output module: After the data is processed by the cache layer and uploaded to each storage server of the fastDFS distributed file system, a successful upload mark is returned to the client.

[0087] The above system, wherein the cache layer creation module comprises:

[0088] The cache layer is a storage pool created on an SSD cluster.

[0089] The above system, wherein the processing module comprises:

[0090] Write-back mode cache layer processing unit: writes data to the cache layer through the client, and then immediately receives the write confirmation, the cache layer migrates the data from the cache layer to the storage server and synchronizes it on the remaining storage servers in the same group, and finally the cache agent deletes the data from the cache layer; and / or;

[0091] Read-only mode cache layer processing unit: When processing a read operation from a client, the cache layer proxy copies the requested data from the storage layer to the cache layer and deletes inactive objects from the cache layer.

[0092] Wherein, the processing module is used for:

[0093] The storage server informs the cache layer of the status information of all current storage nodes and assigns different weights to each storage node.

[0094] Wherein, the processing module is used for:

[0095] A storage node is selected according to the weight to complete the synchronization task.

[0096] Embodiment three:

[0097] Combination Figure 6 As shown, this embodiment discloses a specific implementation of a computer device. The computer device may include a processor 81 and a memory 82 storing computer program instructions.

[0098] Specifically, the processor 81 may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present application.

[0099] Among them, the memory 82 may include a large-capacity memory for data or instructions. By way of example and not limitation, the memory 82 may include a hard disk drive (HDD), a floppy disk drive, a solid-state drive (SSD), a flash memory, an optical disk, a magneto-optical disk, a magnetic tape, or a universal serial bus (USB) drive, or a combination of two or more of these. Where appropriate, the memory 82 may include a removable or non-removable (or fixed) medium. Where appropriate, the memory 82 may be inside or outside the data processing device. In a specific embodiment, the memory 82 is a non-volatile memory. In a specific embodiment, the memory 82 includes a read-only memory (ROM) and a random access memory (RAM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable ROM (Programmable Read-Only Memory, PROM for short), an erasable PROM (Erasable ProgrammableRead-Only Memory, EPROM for short), an electrically erasable PROM (Electrically Erasable ProgrammableRead-Only Memory, EEPROM for short), an electrically alterable ROM (Electrically Alterable Read-Only Memory, EAROM for short) or a flash memory (FLASH) or a combination of two or more of these. Under appropriate circumstances, the RAM can be a static random access memory (SRAM) or a dynamic random access memory (DRAM), wherein the DRAM can be a fast page mode dynamic random access memory (FPMDRAM), an extended data output dynamic random access memory (EDODRAM), a synchronous dynamic random access memory (SDRAM), etc.

[0100] The memory 82 may be used to store or cache various data files required for processing and / or communication, as well as possible computer program instructions executed by the processor 81 .

[0101] The processor 81 reads and executes the computer program instructions stored in the memory 82 to implement any one of the methods for enhancing the availability of the distributed file system in the above embodiments.

[0102] In some of these embodiments, the computer device may further include a communication interface 83 and a bus 80. Figure 5 As shown, the processor 81, the memory 82, and the communication interface 83 are connected via a bus 80 and communicate with each other.

[0103] The communication interface 83 is used to implement communication between the modules, devices, units and / or equipment in the embodiment of the present application. The communication interface 83 can also implement data communication with other components such as: external devices, image / data acquisition equipment, databases, external storage, and image / data processing workstations.

[0104] The bus 80 includes hardware, software or both, and couples the components of the computer device to each other. The bus 80 includes but is not limited to at least one of the following: a data bus, an address bus, a control bus, an expansion bus, and a local bus. By way of example and not limitation, bus 80 may include an Accelerated Graphics Port (AGP) or other graphics bus, an Extended Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a Hyper Transport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an InfiniBand interconnect, a Low Pin Count (LPC) bus, a memory bus, a Micro Channel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local Bus (VLB) bus, or other suitable buses or a combination of two or more of these. Where appropriate, bus 80 may include one or more buses. Although embodiments of the present application describe and illustrate a particular bus, the present application contemplates any suitable bus or interconnect.

[0105] The computer device can be based on the enhancement of the availability of the distributed file system, thereby realizing the combination Figure 1-Figure 2 Describe the method.

[0106] In addition, in combination with the method for enhancing the availability of a distributed file system in the above embodiments, the embodiments of the present application may provide a computer-readable storage medium for implementation. The computer-readable storage medium stores computer program instructions; when the computer program instructions are executed by a processor, any one of the methods for enhancing the availability of a distributed file system in the above embodiments is implemented.

[0107] The technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0108] In summary, the beneficial effect of the present invention is that the present invention solves the problem of data loss when writing data in the fastDFS distributed file system; the present invention also solves the synchronization problem of newly added nodes, avoiding the problems of low efficiency and poor stability that may be caused by traditional methods.

[0109] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A method for enhancing the availability of a distributed file system, characterized in that: include: Input step: Write any data to the fastDFS distributed file system through the client; Creating a cache layer step: creating a cache layer between the tracking server and the storage server of the fastDFS distributed file system; Processing step: processing the data through the cache layer; Output step: after the data is processed by the cache layer and uploaded to each storage server of the fastDFS distributed file system, a successful upload mark is returned to the client; The cache layer is a storage pool created on an SSD cluster. The cache layer accesses the storage server according to the idle IP and port number provided by the tracking server. The cache layer returns a successful upload mark to the client only after uploading the file to each storage server. The processing steps include: Write-back mode cache layer processing steps: the client writes data to the cache layer, and then immediately receives a write confirmation, the cache layer migrates the data from the cache layer to the storage server and synchronizes it on the remaining storage servers in the same group, and finally the cache proxy deletes the data from the cache layer; Read-only mode cache layer processing steps: When processing a read operation from a client, the cache layer agent copies the requested data from the storage layer to the cache layer and deletes inactive objects from the cache layer.

2. The method for enhancing the availability of a distributed file system according to claim 1, wherein: The processing steps include: The storage server informs the cache layer of the status information of all current storage nodes and assigns different weights to each storage node.

3. The method for enhancing the availability of a distributed file system according to claim 2, characterized in that: The processing steps include: A storage node is selected according to the weight to complete the synchronization task.

4. A distributed file system availability enhancement system, characterized in that: include: Input module: writes any data to the fastDFS distributed file system through the client; Creating a cache layer module: creating a cache layer between the tracking server and the storage server of the fastDFS distributed file system; Processing module: processing the data through the cache layer; Output module: after the data is processed by the cache layer and uploaded to each storage server of the fastDFS distributed file system, a successful upload mark is returned to the client; The cache layer creation module includes: the cache layer is a storage pool created on an SSD cluster, the cache layer accesses the storage server according to the idle IP and port number provided by the tracking server, and returns a successful upload mark to the client only after the cache layer uploads the file to each storage server; Wherein, the processing module includes: Write-back mode cache layer processing unit: writes data to the cache layer through the client, and then immediately receives the write confirmation, the cache layer migrates the data from the cache layer to the storage server and synchronizes it on the remaining storage servers in the same group, and finally the cache agent deletes the data from the cache layer; and / or; Read-only mode cache layer processing unit: When processing a read operation from a client, the cache layer proxy copies the requested data from the storage layer to the cache layer and deletes inactive objects from the cache layer.

5. The system for enhancing the availability of a distributed file system as claimed in claim 4, characterized in that: The processing module is used for: The storage server informs the cache layer of the status information of all current storage nodes and assigns different weights to each storage node.

6. The system for enhancing the availability of a distributed file system as claimed in claim 5, characterized in that: The processing module is used for: A storage node is selected according to the weight to complete the synchronization task.

Images