Solution for constructing dynamic storage service based on network file system (NFS)

Abstract

The invention discloses a solution for constructing a dynamic storage service based on a network file system (NFS), which comprises the following steps of: S1, performing configuration and service connection on all main storage servers and backup storage servers to form an NFS storage cluster, and connecting the NFS storage cluster with a storage tuning calculator through a tuning device; s2, the Nginx storage query service node and the FTP mounting catalog are connected to an NFS storage cluster in a grid-connected mode, so that an Nginx catalog and an FTP catalog can obtain an NFS storage node catalog; s3, the NFS storage nodes of all the main storage servers in the NFS storage cluster are preferentially and completely used, dynamic storage is achieved through the tuner and the storage tuning calculator, and in the dynamic storage process, when resources of the used NFS storage nodes are exhausted, a new backup storage server is issued, and the NFS storage nodes of the backup storage server are added. According to the method and the system, dynamic storage with large data volume, high performance, high availability and expandability can be provided in a high-concurrency scene.

Description

Technical Field

[0001] The present invention relates to the technical field of dynamic storage services, and in particular to a solution for building a dynamic storage service based on a network file system (NFS). Background Art

[0002] With the rapid development of Internet technology, the user scale of network applications is constantly expanding, and the business complexity is increasing. Especially during peak hours, such as large-scale users' QR code display, decoding, data collaboration, etc., Internet applications are facing huge concurrent access pressure, and the corresponding storage requirements for unstructured data such as pictures, files, and dynamically extended data are increasing. In the case of high concurrent access, there are different degrees of freezes, losses, confusion, etc. All these situations require extremely high requirements for dynamic storage.

[0003] Especially during specific peak hours, such as holidays, large-scale events or emergencies, users access network applications in a concentrated manner to complete tasks such as QR code display (for identity authentication, payment, information acquisition, etc.), decoding and processing, and cross-platform data collaboration, which brings unprecedented concurrent access pressure to the system. These operations not only require the system to respond quickly and process massive requests, but also need to ensure the accuracy and consistency of data to meet users' expectations for efficient and stable service experience.

[0004] However, with the increase in business complexity, traditional storage architectures have gradually exposed their limitations. In particular, the storage and management of unstructured data (such as pictures, videos, audio files, documents, logs, and various dynamically expanding data sets) face huge challenges. Unstructured data is not only large in volume and grows rapidly, but also has diverse formats and is difficult to manage effectively with traditional relational databases. In the context of high concurrent access, the read and write efficiency, data consistency, and reliability of these data have become increasingly prominent, and it is easy to experience freezes, data loss, confusion, and other phenomena that seriously affect user experience and service quality. Summary of the invention

[0005] The technical problem to be solved by the present invention is to provide a solution for building a dynamic storage service based on the network file system NFS, which can realize dynamic storage with large data volume, high performance, high availability and scalability in high concurrency scenarios.

[0006] In order to solve the above technical problems, the technical solutions adopted by the present invention are as follows.

[0007] A solution for building a dynamic storage service based on a network file system NFS is implemented based on an Nginx storage query server using an Nginx storage query service, an FTP storage receiving / providing server using an FTP storage receiving / providing service, several main storage servers with built-in tuners, several backup storage servers with built-in tuners, and a storage tuning calculator; the method comprises the following steps:

[0008] S1. Configure and connect all primary storage servers and backup storage servers to form an NFS storage cluster, and connect them to the storage tuning calculator through the tuner.

[0009] S2. Connect the Nginx storage query service node and FTP mount directory to the NFS storage cluster so that the Nginx directory and FTP directory can obtain the NFS storage node directory;

[0010] S3. Prioritize and fully use the NFS storage nodes of all primary storage servers in the NFS storage cluster, and realize dynamic storage through the tuner and storage tuning calculator. During the dynamic storage process, when the resources of the used NFS storage nodes are exhausted, a new backup storage server is issued to join its NFS storage nodes.

[0011] Preferably, in step S3, dynamic storage is implemented by a tuner and a storage tuning calculator, which specifically includes the following steps:

[0012] S31 uses the tuner to obtain NFS storage node storage information and sends it to the storage tuning calculator;

[0013] S32. The storage tuning calculator uses a program to calculate the optimal storage node allocation according to the configured storage rules, and feeds back to the tuner in the form of instructions;

[0014] S33. The tuner sends the storage data of the current NFS storage node to the NFS storage node specified by the instruction according to the instruction information, thereby realizing dynamic storage.

[0015] Preferably, the storage tuning calculator adopts a server sharding mechanism, and stores another copy of the data in an NFS storage node other than the NFS storage node storing one copy of the data, which is automatically distributed by the tuner.

[0016] Preferably, the storage rule in step S32 includes: the storage tuning calculator receives the total storage amount and usage value of the NFS storage node sent by the tuner;

[0017] When the storage tuning calculator finds that the storage utilization rates of all used NFS storage nodes are not equal and are not higher than the set storage utilization rate threshold, it calculates the optimal storage node allocation so that the NFS storage nodes with higher storage utilization rates transfer part of their own storage to other NFS storage nodes with lower storage utilization rates, and ensures that the storage utilization rate of each used NFS storage node is equal after the transmission and reception are completed;

[0018] When the storage tuning calculator finds that the storage usage rate of a certain NFS storage node is higher than the set storage usage rate threshold, it automatically queries the NFS storage nodes whose storage usage rate is lower than the set storage usage rate threshold, and calculates the optimal storage node allocation, so that the NFS storage node whose storage usage rate is higher than the set storage usage rate threshold transfers part of its own storage to other NFS storage nodes whose storage usage rate is lower than the set storage usage rate threshold, and ensures that the storage usage rate of the NFS storage node receiving the transfer is lower than the set storage usage rate threshold after receiving the transfer, and ensures that the storage usage rate of each used NFS storage node is equal after the transmission and reception are completed.

[0019] Preferably, in step S3, when the resources of the used NFS storage nodes are exhausted, specifically: under the premise of ensuring that the storage utilization rate of each used NFS storage node is equal and lower than the set storage utilization rate threshold after the transmission and reception are completed, the optimal storage node allocation cannot be calculated; a new backup storage server is issued to add its NFS storage node, and the storage tuning calculator uses a program to calculate the optimal storage node allocation according to the configured storage rules; the storage utilization rate threshold is 80%.

[0020] Preferably, when a client requests to access a file, the request is forwarded to the primary storage server or the backup storage server through a load balancing strategy.

[0021] Due to the adoption of the above technical scheme, the technical progress achieved by the present invention is as follows.

[0022] The present invention is easy to implement: NFS, Nginx and FTP are all common technologies in data storage, concurrent query and data transmission. By combining the three innovatively through a small amount of code, dynamic storage and service of server nodes can be realized.

[0023] The present invention can improve the high availability and scalability of the system: by building a network file system NFS storage cluster, multiple servers jointly undertake the task of file storage and access, which significantly improves the fault tolerance and scalability of the system. When a server fails, the system can automatically forward the request to other servers that are running normally, ensuring the continuity and uninterruptedness of the service.

[0024] The present invention can enhance the flexibility and response speed of storage services: the Nginx storage query service built by introducing the Nginx proxy feature can efficiently process high-concurrency query requests and provide stable and fast responses. This provides great convenience for users to access stored data, especially in scenarios with large data access volume, significantly improving user experience.

[0025] The present invention can enrich the data storage interaction function: by utilizing the comprehensive functions of FTP (such as file upload, download, directory browsing, file renaming, deletion, breakpoint resume, etc.), the present technical solution provides a more complete data interaction mode for storage reception / service provision. At the same time, the user management function and multiple data transmission modes and security measures supported by FTP further guarantee the security and reliability of data.

[0026] The present invention can realize dynamic storage management and optimization: the storage tuning calculator and tuner are used in combination, and the optimal storage node allocation scheme is automatically calculated and the storage layout is automatically adjusted through real-time monitoring and intelligent analysis of the storage status of NFS storage nodes. This not only ensures the reasonable distribution and efficient use of data, but also effectively prevents the storage node from shutting down due to overload, reducing the risk of data loss.

[0027] The present invention can improve the automation and intelligence level of the overall system: the design of the entire solution focuses on automation and intelligence, and through the collaborative work of various components, the whole process from data storage, query to optimization is realized. This not only reduces the complexity and error rate of manual operation, but also improves the overall operation efficiency and stability of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 This is a schematic diagram of the present invention. DETAILED DESCRIPTION

[0029] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0030] A solution for building dynamic storage services based on the network file system NFS, combined with Figure 1 As shown, based on an Nginx storage query server using Nginx storage query service, an FTP storage receiving / providing server using FTP storage receiving / providing service, several main storage servers with built-in tuners, several backup storage servers with built-in tuners, and a storage tuning calculator, the present invention includes the following steps:

[0031] S1. Configure and connect all primary storage servers and backup storage servers to form an NFS storage cluster, and connect them to the storage tuning calculator through the tuner.

[0032] Multiple storage servers share files and directories through the network, and are configured and connected to services, so that the storage of all storage servers can be reasonably utilized, providing high-availability and scalable file sharing services. By sharing storage space, the file system can be mounted on multiple servers at the same time, which can achieve high data availability and load balancing. When a client requests to access a file, the request can be forwarded to the primary storage server or backup storage server through the load balancing strategy, and data synchronization and real-time interaction between them can be ensured, achieving the continuity and consistency of business transactions.

[0033] By connecting the tuner to the storage tuning calculator, the storage of all storage servers can be reasonably allocated through the storage tuning calculator.

[0034] S2. Connect the Nginx storage query service node and FTP mount directory to the NFS storage cluster so that the Nginx directory and FTP directory can obtain the NFS storage node directory.

[0035] Specifically, the Nginx storage query service node is connected to the NFS storage cluster, so that the Nginx directory can obtain the NFS storage node directory and thus obtain node information. The Nginx storage query server uses the Nginx proxy feature and high concurrency characteristics to enable storage files to provide external queries in the form of services, thereby providing query services under high concurrency, laying the computing foundation for storage reception / service provision, and accurately locating the request service to the corresponding NFS storage node.

[0036] Specifically, the FTP mount directory is connected to the NFS storage cluster so that the FTP directory can obtain the NFS storage node directory and thus obtain data information. The FTP storage receiving / providing server uses the FTP file upload, file download, directory browsing, file renaming, file deletion, and breakpoint resume functions to achieve data storage interaction. At the same time, FTP also provides user management functions and supports multiple data transmission modes and security measures to ensure the reliability and security of file transmission.

[0037] S3. Prioritize and fully use the NFS storage nodes of all primary storage servers in the NFS storage cluster, and realize dynamic storage through the tuner and storage tuning calculator. During the dynamic storage process, when the resources of the used NFS storage nodes are exhausted, a new backup storage server is issued to join its NFS storage nodes.

[0038] The storage tuning calculator uses the NFS features to obtain the storage status of the NFS storage nodes of each primary storage server and backup storage server, aggregates them on the calculator, and uses the program to calculate the optimal storage node allocation according to the configured storage rules, thereby ensuring that each NFS storage node (each storage size is different) can ensure reasonable and optimal storage under a reasonable storage ratio and prevent storage node downtime and data loss.

[0039] The tuner uses the NFS feature to obtain the storage status of the NFS storage nodes of each primary storage server and backup storage server, sends the data to the storage tuning calculator, and receives instructions from the storage tuning calculator to send the storage data of the current NFS storage node to the corresponding NFS storage node.

[0040] This step specifically includes the following steps:

[0041] S31. Use the tuner to obtain storage information of the NFS storage node and send it to the storage tuning calculator.

[0042] S32. The storage tuning calculator uses a program to calculate the optimal storage node allocation according to the configured storage rules, and feeds it back to the tuner in the form of instructions.

[0043] Specifically, the storage rules include: the storage tuning calculator receives the total storage amount and usage value of the NFS storage node sent by the tuner. When the storage tuning calculator finds that the storage usage rates of all used NFS storage nodes are not equal and are not higher than the set storage usage rate threshold, it calculates the optimal storage node allocation, so that the NFS storage nodes with higher storage usage rates transfer part of their own storage to other NFS storage nodes with lower storage usage rates, and ensures that the storage usage rate of each used NFS storage node is equal after the transmission and reception are completed.

[0044] For example: the storage of the main storage server A is 1000G, and 800G is configured as NFS storage node storage. The storage of the main storage server B is 500G, and 400G is configured as NFS storage node storage. Then the storage of the entire NFS storage cluster is 1200G. Assuming that there is currently a storage demand of 600G, after the storage tuning calculator calculates the optimal storage node allocation, the storage on the main storage server A is 600*800 / (800+400)G, and the storage on the main storage server B is 600*400 / (800+400)G. This example is only a simple example, and the actual adjustment should be made according to the server IO performance.

[0045] When the storage tuning calculator finds that the storage usage rate of a certain NFS storage node is higher than the set storage usage rate threshold, it automatically queries the NFS storage nodes whose storage usage rate is lower than the set storage usage rate threshold, and calculates the optimal storage node allocation, so that the NFS storage nodes whose storage usage rate is higher than the set storage usage rate threshold transfer part of their own storage to other NFS storage nodes whose storage usage rate is lower than the set storage usage rate threshold, and ensure that the storage usage rate of the NFS storage nodes receiving the transmission is lower than the set storage usage rate threshold after receiving the transmission, and ensure that the storage usage rate of each used NFS storage node is almost equal after the transmission and reception are completed.

[0046] When the resources of the used NFS storage nodes are exhausted, specifically: under the premise of ensuring that the storage utilization rate of each used NFS storage node is equal and lower than the set storage utilization rate threshold after the transmission and reception are completed, the optimal storage node allocation cannot be calculated, then a new backup storage server is sent to add its NFS storage node, and the storage tuning calculator uses the program to calculate the optimal storage node allocation according to the configured storage rules.

[0047] Taking the storage usage threshold of 80% as an example, the main code implementation is as follows:

[0048]

[0049] To ensure node failure and data loss, the storage tuning calculator also uses a server sharding mechanism, that is, the same data needs to be configured with shard nodes. For example, if the NFS storage node of storage server A stores one piece of data and two shards are configured, then another piece of data needs to be stored in the NFS storage node of the storage server other than storage server A, which will be automatically distributed by the tuner.

[0050] S33. The tuner sends the storage data of the current NFS storage node to the NFS storage node specified by the instruction according to the instruction information, thereby realizing dynamic storage.

[0051] The present invention has the following advantages:

[0052] (1) Easy to implement

[0053] NFS, Nginx and FTP are all common technologies for data storage, concurrent query and data transmission. By combining the three innovatively with a small amount of code, dynamic storage and service of server nodes can be realized.

[0054] (2) Improve system availability and scalability

[0055] By building a network file system NFS storage cluster, multiple servers share the task of file storage and access, significantly improving the system's fault tolerance and scalability. When a server fails, the system can automatically forward requests to other servers that are operating normally, ensuring service continuity and uninterrupted service.

[0056] (3) Enhance the flexibility and responsiveness of storage services

[0057] The Nginx storage query service built with the Nginx proxy feature can efficiently handle high-concurrency query requests and provide stable and fast responses. This greatly facilitates users to access stored data, and significantly improves the user experience, especially in scenarios with large amounts of data access.

[0058] (4) Enriching data storage interaction functions

[0059] By utilizing the comprehensive functions of FTP (such as file upload, download, directory browsing, file renaming, deletion, breakpoint resume, etc.), this technical solution provides a more complete data interaction method for storage reception / service provision. At the same time, the user management function and multiple data transmission modes and security measures supported by FTP further ensure the security and reliability of data.

[0060] (5) Implement dynamic storage management and optimization

[0061] The storage tuning calculator and tuner are used together to automatically calculate the optimal storage node allocation plan and automatically adjust the storage layout through real-time monitoring and intelligent analysis of the storage status of NFS storage nodes. This not only ensures the reasonable distribution and efficient use of data, but also effectively prevents storage nodes from crashing due to overload, reducing the risk of data loss.

[0062] (6) Improve the automation and intelligence level of the overall system

[0063] The design of the entire solution focuses on automation and intelligence. Through the collaborative work of various components, the entire process from data storage, query to optimization is automated. This not only reduces the complexity and error rate of manual operations, but also improves the overall operating efficiency and stability of the system.

Claims

1. A solution for building a dynamic storage service based on a network file system NFS, characterized by: Based on an Nginx storage query server using Nginx storage query service, an FTP storage receiving / providing server using FTP storage receiving / providing service, several main storage servers with built-in tuners, several backup storage servers with built-in tuners and a storage tuning calculator; the method comprises the following steps: S1. Configure and connect all primary storage servers and backup storage servers to form an NFS storage cluster, and connect them to the storage tuning calculator through the tuner. S2. Connect the Nginx storage query service node and FTP mount directory to the NFS storage cluster so that the Nginx directory and FTP directory can obtain the NFS storage node directory; S3. Prioritize and fully use the NFS storage nodes of all primary storage servers in the NFS storage cluster, and realize dynamic storage through the tuner and storage tuning calculator. During the dynamic storage process, when the resources of the used NFS storage nodes are exhausted, a new backup storage server is issued to join its NFS storage nodes.

2. A solution for building a dynamic storage service based on a network file system NFS according to claim 1, characterized in that: In step S3, dynamic storage is realized by using a tuner and a storage tuning calculator, which specifically includes the following steps: S31 uses the tuner to obtain NFS storage node storage information and sends it to the storage tuning calculator; S32. The storage tuning calculator uses a program to calculate the optimal storage node allocation according to the configured storage rules, and feeds back to the tuner in the form of instructions; S33. The tuner sends the storage data of the current NFS storage node to the NFS storage node specified by the instruction according to the instruction information, thereby realizing dynamic storage.

3. A solution for building a dynamic storage service based on a network file system NFS according to claim 2, characterized in that: The storage tuning calculator adopts a server sharding mechanism, and stores another copy of the data in an NFS storage node other than the NFS storage node storing the data, which is automatically distributed by the tuner.

4. A solution for building a dynamic storage service based on a network file system NFS according to claim 2, characterized in that: The storage rule in step S32 includes: the storage tuning calculator receives the total storage amount and usage value of the NFS storage node sent by the tuner; When the storage tuning calculator finds that the storage utilization rates of all used NFS storage nodes are not equal and are not higher than the set storage utilization rate threshold, it calculates the optimal storage node allocation so that the NFS storage nodes with higher storage utilization rates transfer part of their own storage to other NFS storage nodes with lower storage utilization rates, and ensures that the storage utilization rate of each used NFS storage node is equal after the transmission and reception are completed; When the storage tuning calculator finds that the storage usage rate of a certain NFS storage node is higher than the set storage usage rate threshold, it automatically queries the NFS storage nodes whose storage usage rate is lower than the set storage usage rate threshold, and calculates the optimal storage node allocation, so that the NFS storage node whose storage usage rate is higher than the set storage usage rate threshold transfers part of its own storage to other NFS storage nodes whose storage usage rate is lower than the set storage usage rate threshold, and ensures that the storage usage rate of the NFS storage node receiving the transfer is lower than the set storage usage rate threshold after receiving the transfer, and ensures that the storage usage rate of each used NFS storage node is equal after the transmission and reception are completed.

5. A solution for building a dynamic storage service based on a network file system NFS according to claim 4, characterized in that: When the used NFS storage node resources are exhausted in step S3, specifically: under the premise of ensuring that the storage utilization rate of each used NFS storage node is equal and lower than the set storage utilization rate threshold after the transmission and reception are completed, the optimal storage node allocation cannot be calculated; a new backup storage server is sent to add its NFS storage node, and the storage tuning calculator uses a program to calculate the optimal storage node allocation according to the configured storage rules; the storage utilization rate threshold is 80%.

6. A solution for constructing a dynamic storage service based on a network file system NFS according to claim 1, characterized in that: When a client requests to access a file, the request is forwarded to the primary storage server or the backup storage server through the load balancing strategy.

Images