Large file storage method and system for specifying Tracker Server based on FastDFS, electronic equipment and computer readable storage medium

Abstract

The invention relates to a large file storage method and system for specifying a Tracker Server based on FastDFS, electronic equipment and a computer readable storage medium. The method comprises the steps that a client obtains a corresponding storage address from a database of a service system or other address configuration files; a specified Tracker Server address is obtained according to a storage address calling method; according to the address of the Tracker Server, determining a corresponding Tracker Server, and according to the address of the Tracker Server; obtaining a corresponding Storage Server by using the determined Tracker Server, and returning the Storage Server to the client side; and the client side uses the returned Storage Server to carry out corresponding file uploading or downloading operation. According to the invention, the function of the FastDFS Client is expanded, the FastDFS supports the uploading of the file through the specified Tracker Server, and the file is stored in the specified file server, so that the files uploaded in different regions can be managed; meanwhile, the service end system can carry out corresponding authority control on the storage files through a specified storage address and control corresponding users or roles to access the data files so as to achieve a corresponding authority control function.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and particularly relates to a large file storage method, system, electronic device, and computer-readable storage medium based on specifying a Tracker Server in FastDFS. Background Art

[0002] With the rapid development of the Internet, especially the rise of fields such as social media, e-commerce, and online video, the requirements for file storage systems have become increasingly complex and diverse. FastDFS is an open-source lightweight distributed file system. The advantages of FastDFS are very obvious, namely: high performance, high availability, scalability, and simplicity of use; however, the disadvantage is that its functions are relatively simple, and even secondary expansion is required in some scenarios.

[0003] FastDFS emerged to address the limitations of traditional file systems in handling large-scale file storage and high-concurrency access. Through its high-performance, high-availability, scalable, and simple-to-use design, FastDFS has become a widely used distributed file system, especially suitable for large file storage, high-concurrency access, and small and medium-sized projects. It has an active open-source community that provides rich documentation, tutorials, and tools, which helps to solve problems and optimize the system. For example: A certain system is used by both the head office and branch offices at the same time, but both the head office and branch offices provide their own file storage servers. For a system with a geographical hierarchy, the file data of some lower-level regions is expected to be stored on the servers of their own regions, rather than on the servers of the upper-level regions, in order to achieve corresponding data file isolation. However, the files uploaded to FastDFS are stored by FastDFS obtaining a random Tracker Server for corresponding file storage, so it is impossible to determine which server the corresponding file will be stored on. Since the functions of FastDFS are relatively simple, it cannot meet the above requirements. Therefore, it is necessary to be able to specify the corresponding Tracker Server for corresponding file storage and enable the business system to implement access control over the users or roles of the files. Summary of the Invention

[0004] In order to solve the above technical problems, the purpose of the present invention is to provide a large file storage method, system, electronic device, and computer-readable storage medium based on specifying a Tracker Server in FastDFS. The specific technical solutions adopted are as follows:

[0005] In a first aspect, an embodiment of the present invention provides a large file storage method based on specifying a Tracker Server in FastDFS. The FastDFS includes a client and a server, where the server includes a Tracker Server Cluster and a Storage Server Cluster. The method includes the following steps:

[0006] The client obtains the corresponding storage address from the database of the business system or other address configuration files.

[0007] Call a method according to the storage address to obtain the specified Tracker Server address.

[0008] Determine the corresponding Tracker Server according to the above Tracker Server address.

[0009] Use the determined Tracker Server to obtain the corresponding Storage Server and return it to the client.

[0010] The client uses the returned Storage Server to perform corresponding file upload or download operations.

[0011] Further, the file upload operation includes:

[0012] Call uploadAppenderFile(String groupName, InputStream inputStream, long fileSize, String fileExtName, String…args) provided by the FastDFS client, where String…args is the newly added variable parameter.

[0013] Judge whether the storage address is passed in through this variable parameter.

[0014] If there is a storage address, obtain the storage address corresponding to the specified Tracker Server from the Tracker Server list, and judge whether the corresponding Tracker Server is normal when calling canTryToConnect. If it is normal, return the connected Tracker Server; use the determined Tracker Server to obtain the corresponding StorageServer for file upload operations.

[0015] If there is no storage address, randomly select a Tracker Server address, and determine the corresponding Tracker Server according to this Tracker Server address; use the determined Tracker Server to obtain the corresponding StorageServer for file upload operations.

[0016] Further, the file download operation includes:

[0017] Call downloadfile(String groupName, String path, String…args) provided by the FastDFS client, where String…args are newly added variable parameters;

[0018] Judge whether the storage address of the Tracker Server is passed in through this variable parameter;

[0019] If there is a storage address, obtain the storage address corresponding to the specified Tracker Server from the Tracker Server list for return; obtain the corresponding Storage Server through the Tracker Server and return it to the client;

[0020] If there is no storage address, randomly select a Tracker Server address, and determine the corresponding Tracker Server according to this Tracker Server address; use the determined Tracker Server to obtain the corresponding StorageServer and return it to the client.

[0021] After the client obtains the corresponding Storage Server, it downloads and outputs the file.

[0022] Further, if the file is a large file, an open-source upload component webupload will be called for chunked upload. The specific steps are as follows:

[0023] Perform chunking operations on the large file data;

[0024] Call the above uploadAppenderFile(String groupName, InputStream inputStream, long fileSize, String fileExtName, String…args) to perform the upload operation of the first chunk;

[0025] Use the appendFile method to upload the remaining chunks until all chunk data is uploaded successfully;

[0026] Merge all the successfully uploaded chunk data to obtain a complete file.

[0027] Further, the step of obtaining the Tracker Server address includes:

[0028] Obtain the specified IP from getTrackerAddress(String…args);

[0029] Traverse all the addresses, grab the currently valid address, and use it as the Tracker Server address.

[0030] Further, the Tracker Server Cluster and the Storage Server Cluster are each composed of one or more servers.

[0031] Further, the Storage Server takes group as the unit. After each Storage Server starts, it will actively connect to the Tracker Server, inform the storage-related information of the group it belongs to, and maintain a periodic heartbeat.

[0032] In a second aspect, an embodiment of the present invention further provides a large file storage system based on FastDFS to specify a Tracker Server, including a memory, a processor, and a computer program stored in the memory and running on the processor. When the processor executes the computer program, the steps of the method described in any one of the above are implemented.

[0033] In a third aspect, an embodiment of the present invention further provides an electronic device, including a processor, a communication interface, a memory, and a communication bus. Among them, the processor, the communication interface, and the memory complete communication with each other through the communication bus; the memory is used to store a computer program; the processor is used to implement the steps of the method described in any one of the above when executing the computer program stored on the memory.

[0034] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the method described in any one of the above are implemented.

[0035] The embodiments of the present invention have at least the following beneficial effects: The present invention provides a large file storage method, system, electronic device, and computer-readable storage medium based on specifying a Tracker Server in FastDFS, which expands the functions of the FastDFS Client, enabling FastDFS to support uploading files to a specified file server through a specified Tracker Server and manage files uploaded from different regions. Since FastDFS does not provide a permission control function, in this way, the business-side system can perform corresponding permission control on the stored files by specifying the storage address, and control which users or roles can access these data files to achieve the corresponding permission control function. At the same time, the corresponding functions of the FastDFS Client are expanded by adding variable parameters while retaining the original functions; and using variable parameters can improve the flexibility and readability of the code. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the technical solutions and advantages in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the following drawings are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0037] Figure 1 It is a flowchart of the steps of a large file storage method based on specifying a Tracker Server in FastDFS provided by an embodiment of the present invention;

[0038] Figure 2 It is a system architecture diagram of the FastDFS distributed file system provided by an embodiment of the present invention;

[0039] Figure 3 It is a flowchart of the steps of file upload provided by an embodiment of the present invention;

[0040] Figure 4 It is a flowchart of the steps of file download provided by an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0041] To further elaborate on the technical means and effects adopted by the present invention to achieve the intended invention purpose, the following specifically describes, with reference to the accompanying drawings and preferred embodiments, the method and system for storing large files based on specifying Tracker Server in FastDFS according to the present invention, including its specific implementation manner, structure, features, and effects. In the following description, different "one embodiment" or "another embodiment" do not necessarily refer to the same embodiment. In addition, specific features, structures, or characteristics in one or more embodiments can be combined in any suitable form.

[0042] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the technical field to which the present invention belongs.

[0043] The following specifically describes the specific solution of a method and system for storing large files based on specifying Tracker Server in FastDFS provided by the present invention with reference to the accompanying drawings.

[0044] Please refer to Figure 1 , which shows the flowchart of the steps of a method for storing large files based on specifying Tracker Server in FastDFS provided by an embodiment of the present invention. Refer to Figure 2 , which shows the system architecture diagram of the FastDFS distributed file system provided by an embodiment of the present invention. Combining Figure 2 It can be seen that the architecture of FastDFS mainly consists of the following two parts, including the client Client and the server Server. Among them, the server includes the tracking server Tracker ServerCluster and the storage server Storage Server Cluster. Among them, Tracker Server: is responsible for the routing and load balancing of client Client requests; Storage Server: is responsible for the actual file storage and management; Client: is responsible for interacting with Tracker Server and Storage Server to upload and download files.

[0045] As Figure 1 shown, the method for storing large files based on specifying Tracker Server provided by the embodiment of the present application includes the following steps:

[0046] The client obtains the corresponding storage address from the database of the business system or other address configuration files;

[0047] It should be noted that there are databases or some data information storing storage addresses in the business system. The FastDFS client (Client) will first obtain relevant storage addresses from these locations in the business system to get the specified Tracker Server.

[0048] Call the method according to the storage address to obtain the specified Tracker Server address;

[0049] It should be noted that the address corresponding to the specified Tracker Server is determined according to the obtained storage address to achieve the purpose of specifying the Tracker Server, rather than randomly selecting a Tracker Server to perform the upload / download operation of large files. The file is uploaded and stored to the specified file server through the specified Tracker Server, and the files uploaded in different regions and at different levels can be isolated or hierarchically managed. Moreover, since FastDFS itself does not provide the function of permission control, in this way, the business system side can perform corresponding permission control on the stored files by specifying the storage address, and then can control which data files can be accessed by the corresponding users or roles to achieve the corresponding permission control function.

[0050] Determine the corresponding Tracker Server according to the above Tracker Server address;

[0051] It should be noted that there are multiple Tracker Servers in the Tracker Server Cluster, and each Tracker Server corresponds to a storage address to ensure query indexing. After obtaining the storage address of the specified Tracker Server, the corresponding Tracker Server connected can be found in the Tracker Server Cluster by using this storage address.

[0052] Obtain the corresponding Storage Server by using the determined Tracker Server above and return it to the client;

[0053] It should be noted that there are multiple Storage Servers in the Storage Server Cluster. The available Storage Server is queried through the specified Tracker Server and returned to the client (Client).

[0054] The client uses the returned Storage Server to perform corresponding file upload or download operations.

[0055] It should be noted that: after the client obtains an available Storage Server, it can establish a connection with the Storage Server to perform corresponding file upload or download operations.

[0056] Please refer to Figure 3 , which shows the flowchart of the file upload steps provided by an embodiment of the present invention. As Figure 3 shown, the file upload operation provided by the embodiment of the present application includes the following steps:

[0057] Call uploadAppenderFile(String groupName, InputStream inputStream, long fileSize, String fileExtName, String…args) provided by the FastDFS client, where String…args is the newly added variable parameter;

[0058] Judge whether the storage address is passed in through this variable parameter;

[0059] It should be noted that: in this embodiment, the storage address of the specified Tracker Server is obtained through the above storage address calling method. Among them, uploadAppenderFile() itself is a method provided by the FastDFS Client for uploading files that support resumeable upload. After such a type of file is created, it can be modified by appending content instead of overwriting the entire file. This is very useful for uploading large files or files that need to be updated frequently. Here, a new parameter String…args is added in the uploadAppenderFile() method to judge whether the storage address is passed in; it not only retains the original function but also expands other functions. Using this newly added variable parameter method not only improves the flexibility of the code but also ensures its readability. The specific implementation code is as follows:

[0060]

[0061] If there is a storage address, obtain the storage address corresponding to the specified Tracker Server from the Tracker Server list, and judge whether the corresponding Tracker Server is normal when calling canTryToConnect. If it is normal, return the connected Tracker Server;

[0062] Use the determined Tracker Server above to obtain the corresponding Storage Server for file upload operations.

[0063] It should be noted that: in this embodiment, the Tracker Server is stored in the form of a list List. Therefore, it is necessary to index the storage address of the specified Tracker Server in the Track Server Cluster in this List; at the same time, the canTryToConnect command is also called to determine whether the connected Tracker Server is valid; after determining the specified Tracker Server, use this specified Tracker Server to obtain an available Storage Server; the client uploads the file by connecting to this available Storage Server. At this time, the storage server Storage Server will return the file ID to the client, and this file ID is used as the index information for accessing the file later. The file index information may include: group name, virtual disk path, data two-level directory, file name, etc.

[0064] If there is no storage address, randomly generate a Tracker Server address, and determine the corresponding Tracker Server according to this Tracker Server address; use the determined Tracker Server above to obtain the corresponding StorageServer for file upload operations.

[0065] It should be noted that: in this embodiment, if the specified Tracker Server is not found, the traditional method will be used to determine a random Tracker Server to query the available Storage Server to complete the file upload operation.

[0066] Please refer to Figure 4 , which shows the step flowchart of file download provided by an embodiment of the present invention. As Figure 4 shown, the file download operation provided by the embodiment of the present application includes the following steps:

[0067] Call downloadfile(String groupName, String path, String…args) provided by the FastDFS client, where String…args is the newly added variable parameter;

[0068] Judge whether the storage address of the Tracker Server is passed in through this variable parameter;

[0069] It should be noted that in this embodiment, the storage address of the specified TrackerServer is obtained through the above storage address calling method. Among them, in FastDFS, downloadfile(String groupName, String path) is a function for downloading files from the FastDFS file system. This function is usually part of the API provided by the FastDFS client library and is used to download files from the FastDFS storage server according to the specified group name (groupName) and file path (path). Here, a new parameter String...args is added to the downloadfile(String groupName, String path) function to determine whether the storage address is passed in; it not only retains the original function but also extends other functions. Using the method of this newly added variable parameter not only improves the flexibility of the code but also ensures its readability.

[0070] If there is a storage address, the storage address corresponding to the specified Tracker Server is obtained from the Tracker Server list and returned; the corresponding Storage Server is obtained through the Tracker Server and returned to the client;

[0071] If there is no storage address, a random Tracker Server address is selected, and the corresponding Tracker Server is determined according to this Tracker Server address; the corresponding StorageServer is obtained by using the determined Tracker Server and returned to the client.

[0072] After the client obtains the corresponding Storage Server, it downloads and outputs the file.

[0073] It should be noted that in this implementation, the Tracker Server is stored in the form of a list List. Therefore, it is necessary to index the storage address of the specified Tracker Server in the Tracker Server Cluster in this List; after determining the specified Tracker Server, the available Storage Server is obtained by using this specified Tracker Server; the client downloads the file by connecting to this available Storage Server. If there is no valid Tracker Server address, a traditional method is used to randomly select a Tracker Server.

[0074] Further, in the embodiment, if the file is a large file, the file upload operation will call the open-source upload component webupload for chunked upload. The specific steps are as follows:

[0075] Perform a chunking operation on the large file data;

[0076] Call the above uploadAppenderFile(String groupName, InputStream inputStream, long fileSize, String fileExtName, String…args) to perform the upload operation for the first chunk;

[0077] Use the appendFile method to perform the upload of the remaining chunks until all chunk data is successfully uploaded;

[0078] Merge all successfully uploaded chunk data to obtain a complete file.

[0079] It should be noted that: This chunked upload method is particularly effective in the case of unstable network environment or file size exceeding the server limit. Also, by concurrently uploading multiple shards, network bandwidth and server resources can be fully utilized to speed up the upload completion time and effectively improve the file upload speed; in terms of resume from breakpoint, if a network interruption or program abnormal exit occurs during the upload process, only the failed shards need to be retransmitted, rather than the entire file.

[0080] Further, in the embodiment, the steps for obtaining the Tracker Server address specifically include:

[0081] Obtain the specified IP from getTrackerAddress(String…args);

[0082] Traverse all addresses and grab the currently valid address as the Tracker Server address.

[0083] It should be noted that: In this embodiment, the getTrackerAddress() function is used to obtain the address of the Tracker Server, and what it returns is the network address of the Tracker Server, usually a combination of an IP address and a port number. Then, the available Tracker Server address is obtained through traversal. The implementation code for obtaining TrackerAddress is as follows:

[0084]

[0085] See Figure 2, which shows the system architecture diagram of the FastDFS distributed file system provided by an embodiment of the present invention. As Figure 2 shown, in the FastDFS distributed file system, the Tracker Server Cluster and the Storage Server Cluster are respectively composed of one or more servers.

[0086] Refer to Figure 2 , which shows the system architecture diagram of the FastDFS distributed file system provided by an embodiment of the present invention. As Figure 2 shown, in the FastDFS distributed file system, the Storage Server takes the group as a unit. After each Storage Server starts, it will actively connect to the Tracker Server, inform the storage-related information of the group it belongs to, and maintain a periodic heartbeat.

[0087] It should be noted that: in this embodiment, adopting this mechanism enables FastDFS to effectively manage and schedule the storage and access of files, improving the reliability and performance of the system; at the same time, due to the heartbeat mechanism between Storage Servers, the faults of Storage Servers can be detected and processed in a timely manner, ensuring the stability of the system.

[0088] Based on the same inventive concept as the above method, an embodiment of the present invention further provides a large file storage system based on FastDFS specifying a Tracker Server, including a memory, a processor, and a computer program stored in the memory and running on the processor. When the processor executes the computer program, the steps of the above method for storing large files based on FastDFS specifying a Tracker Server are implemented.

[0089] Based on the same inventive concept as the above method, an embodiment of the present invention further provides an electronic device, which is characterized in that it includes a processor, a communication interface, a memory, and a communication bus. Among them, the processor, the communication interface, and the memory complete communication with each other through the communication bus; the memory is used to store a computer program; the processor is used to implement the steps of the above method for storing large files based on FastDFS specifying a Tracker Server when executing the computer program stored on the memory.

[0090] Based on the same inventive concept as the above method, an embodiment of the present invention further provides a computer-readable storage medium, characterized in that a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, the steps of the above method for storing large files based on specifying a Tracker Server in FastDFS are implemented.

[0091] It should be noted that: the above sequence of embodiments of the present invention is only for description and does not represent the superiority or inferiority of the embodiments. And the above specific embodiments of this specification have been described. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0092] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between each embodiment can be referred to each other. The key point of each embodiment is to illustrate the differences from other embodiments.

[0093] The above are only the preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent replacements, improvements, etc. made within the principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A large file storage method based on FastDFS specifying a Tracker Server, the FastDFS comprising a client and a server, wherein the server comprises a Tracker Server Cluster and a Storage Server Cluster, characterized in that the method comprises the following steps: The client obtains the corresponding storage address from the database of the business system or other address configuration files; Call the method based on the storage address to obtain the specified Tracker Server address; Determine the corresponding Tracker Server based on the above Tracker Server address; Use the above determined Tracker Server to obtain the corresponding Storage Server and return it to the client; The client uses the returned Storage Server to perform corresponding file upload or download operations.

2. The large file storage method based on FastDFS designated Tracker Server as claimed in claim 1, characterized in that: The file upload operation includes: Call the uploadAppenderFile(String groupName, InputStreaminputStream, long fileSize, String fileExtName, String…args) provided by the FastDFS client, where String…args is a newly added variable parameter; Determine whether to pass in the storage address through the variable parameter; If there is a storage address, obtain the storage address corresponding to the specified Tracker Server from the Tracker Server list, and determine whether the corresponding Tracker Server is normal when calling canTryToConnect. If normal, return the connected Tracker Server; use the above determined Tracker Server to obtain the corresponding StorageServer for file upload operation; If there is no storage address, a random Tracker Server address is obtained, and the corresponding Tracker Server is determined based on the Tracker Server address; the corresponding Storage Server is obtained using the above determined Tracker Server to perform the file upload operation.

3. The large file storage method based on FastDFS designated Tracker Server as claimed in claim 1, characterized in that: The file download operation includes: Call downloadfile(String groupName, String path, String ... args) provided by the FastDFS client, where String ... args is a newly added variable parameter; This variable parameter is used to determine whether to pass in the storage address of the Tracker Server; If there is a storage address, the storage address corresponding to the specified Tracker Server is obtained from the Tracker Server list and returned; the corresponding Storage Server is obtained through the Tracker Server and returned to the client; If there is no storage address, a random Tracker Server address is used to determine the corresponding Tracker Server. The corresponding Storage Server is obtained using the Tracker Server determined above and returned to the client. The client obtains the corresponding Storage Server and then downloads and outputs the file.

4. The large file storage method based on FastDFS designated Tracker Server as claimed in claim 2, characterized in that: If the file is a large file, the open source upload component webupload will be called to upload it in blocks. The specific steps are as follows: Divide large file data into blocks; Call the above uploadAppenderFile(String groupName, InputStreaminputStream, longfileSize, String fileExtName, String...args) to upload the first block; Use the appendFile method to upload the remaining blocks until all block data are uploaded successfully; Merge all successfully uploaded chunked data to get a complete file.

5. The large file storage method based on FastDFS designated Tracker Server according to any one of claims 2-3, characterized in that: The Tracker Server address acquisition step includes: Get the specified IP from getTrackerAddress(String...args); Traverse all addresses and grab the currently valid address as the Tracker Server address.

6. The large file storage method based on FastDFS designated Tracker Server as claimed in claim 1, characterized in that: The Tracker Server Cluster and Storage Server Cluster are respectively composed of one or more servers.

7. The large file storage method based on FastDFS designated Tracker Server as claimed in claim 1, characterized in that: The Storage Server is organized into groups. After startup, each Storage Server will actively connect to the Tracker Server to inform it of storage-related information of its group and maintain periodic heartbeats.

8. A large file storage system based on FastDFS designated Tracker Server, comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, the steps of a large file storage method based on FastDFS specifying a TrackerServer are implemented as described in any one of claims 1-7.

9. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; Memory for storing computer programs; The processor is used to implement the steps of a large file storage method based on FastDFS designated Tracker Server as described in any one of claims 1 to 7 when executing the computer program stored in the memory.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by the processor, the steps of a large file storage method based on FastDFS designated Tracker Server are implemented as described in any one of claims 1 to 7.

Images