Network disk storage method based on ceph

Abstract

The invention provides a network disk storage method based on Ceph, which utilizes the remarkable characteristics of high performance, high availability, decentration and the like of Ceph as storage to construct a set of distributed network disk system based on Ceph storage in a user enterprise. Along with the continuous increase of the enterprise data volume, the network disk system constructed by the method can easily deal with the data volume without frequently upgrading or expanding hardware, so that the operation and maintenance cost of the enterprise is greatly reduced. Meanwhile, due to the decentration design of the Ceph, even if part of nodes break down, the whole storage system can still keep normal operation, and continuous availability of data and continuity of services are ensured. Besides, the network disk system constructed by the method also remarkably reduces the hardware cost of an enterprise, and compared with a traditional storage solution, the Ceph provides higher cost performance, so that a large amount of funds can be saved when the enterprise constructs the network disk system.

Description

Technical Field

[0001] The present invention relates to the field of cloud storage technology, and in particular to a network disk storage method based on CEPH. Background Art

[0002] In the context of the Internet era, users' demand for storage is increasingly inclined towards fast response, large capacity and instant availability, which has prompted the development of the storage industry to gradually shift from a device procurement model to a service procurement model. Cloud storage technology has emerged as an innovative data storage solution that provides users with efficient, transparent and easy-to-manage storage services. The cloud storage platform encapsulates and abstracts the underlying storage devices, so that users do not need to worry about the complexity and specific implementation details of the underlying storage devices. Users can easily store, manage and access data through simple interfaces or platform interfaces. This transparent design not only lowers the user's usage threshold, but also improves the flexibility and scalability of storage services.

[0003] For users in China, the access speed of foreign network disk products is slow, which can hardly meet the needs of quickly obtaining and sharing files.

[0004] For enterprises and individual users, data security during storage and transmission is of vital importance. Unfortunately, many current online disk services cannot ensure this, which poses a serious threat to the leakage of personal privacy data and corporate trade secrets. Once the online disk service provider stops the service for some reason, or encounters force majeure that causes the storage service to be interrupted, the user's data will be at risk of being lost, which may cause immeasurable losses.

[0005] For enterprise users, file data management is an indispensable part of their daily production and operation activities. Whether it is the storage and backup of document data, or data analysis and mining, it is inseparable from an efficient and reliable file data management system. The current online disk applications on the market have indeed brought unprecedented convenience to users. However, although there are many storage service providers on the market, such as Alibaba Cloud Storage, Baidu Cloud Storage and Tencent Cloud Storage, their services cannot fully meet the diverse needs of enterprises. For example, Tencent Cloud Storage and Alibaba Cloud Storage mainly provide pure back-end storage services. Although they can meet basic storage needs, they have certain limitations in data management, security control and customized services. Although Baidu Cloud Storage provides some additional services, its service scope and functions cannot fully cover all the needs of enterprises.

[0006] For enterprise users, the existing public network disks are often unsatisfactory in terms of transmission speed. To increase the transmission speed, enterprises often need to pay high fees, which undoubtedly increases the operating costs of enterprises. More importantly, these network disk services often cannot be seamlessly connected with the business processes and organizational structures of enterprises, which not only cannot effectively improve work efficiency, but may also expose enterprises to greater operational risks. Therefore, for enterprises, the existing network disk services have undoubtedly become a heavy burden. Summary of the invention

[0007] The purpose of the technical solution of the present invention is to improve data security and ensure efficient collaborative work.

[0008] The technical solution of the present invention provides a network disk storage method based on ceph, comprising the following steps:

[0009] The client obtains the user's personal information for identity authentication. If the identity authentication is passed, the client displays the user's personal homepage and user storage data corresponding to the user's personal information, and obtains the user's various requests;

[0010] The Nginx load balancing server in the service layer evenly distributes each request to each service node in the file service module and group module in the service layer according to the preset load balancing algorithm. If it is verified that the pre-allocated service node is faulty, the faulty service node is proposed and an available service node is allocated;

[0011] Classify the requests assigned to the file service module according to file data requests and file data requests;

[0012] For a request to upload file data, the file data is segmented according to the segment size preset by the user to obtain multiple file segments, and the multiple file segments are regarded as an independent upload unit and uploaded one by one until the segment integrity and file integrity are verified, and the file metadata information and file data are obtained;

[0013] The storage layer includes a MySQL relational database and a Ceph storage cluster. The MySQL relational database stores file metadata information and sets a preset file name. The hardware environment of the Ceph storage cluster has 2n storage nodes, which are divided into a network disk storage cluster and a backup storage cluster. They also store file data and set a preset file name.

[0014] For a request to download file data, locate the file metadata information and the storage location of the file data in the MySQL relational database and the network disk storage cluster according to the preset file name. If the storage location is located and the attribute of the file data is a file, it is directly decrypted and transmitted to the client in combination with the file metadata information. If the storage location is not located, the client prompts the user to relocate from the backup storage cluster. If the storage location is located, the user is prompted to recover and parse the data.

[0015] If the storage location is located and the attribute of the file data is a folder, and multiple file fragments are saved, they are merged into file data with the attribute of a file in the order of the fragments, and the integrity is verified. If the verification passes, the file is decrypted and transmitted to the client in combination with the file metadata information. If the storage location is not located, the client prompts the user to relocate from the backup storage cluster. If the storage location is located, the user is prompted to restore the data and merge and parse it;

[0016] Classify the requests assigned to the group module into role management requests and sharing requests;

[0017] For role management requests, authentication is provided based on the permissions corresponding to the user's personal homepage, and external keys are accessed for user permission verification;

[0018] For sharing requests, based on the group sharing permissions in the user permission verification, create a sharing link and set the sharing expiration time, and grant corresponding sharing permissions, including viewing, downloading, and saving to the network disk.

[0019] Preferably, the preset load balancing algorithm includes polling and least connections.

[0020] Preferably, the even distribution of the requests is achieved by modifying the configuration file of the Nginx load balancing server.

[0021] Preferably, the configuration file includes specifying a listening port and setting an upstream server list.

[0022] Preferably, the storage layer is deployed on an independent physical or virtual server to ensure a loose coupling relationship between it and the business operation service.

[0023] Preferably, the user's personal homepage includes personal information, used space on the network disk, remaining space on the network disk, and percentage of available space on the network disk.

[0024] Preferably, the request for uploading file data further includes:

[0025] Before the file data is officially uploaded, obtain the preset file name corresponding to the file data;

[0026] Locate the preset file name in the Mysql relational database and the Ceph storage cluster, and if the storage location of the preset file name is not located, upload;

[0027] If the storage location of the preset file name is located, it is further determined whether it is in the group. If it is in the group, it is uploaded to the user's personal Mysql relational database and the Ceph storage cluster.

[0028] The technical solution of the present invention proposes a network disk storage method based on Ceph, which utilizes the significant characteristics of Ceph as storage, such as high performance, high availability and decentralization, to build a distributed network disk system based on Ceph storage within the user enterprise. With the continuous growth of the amount of enterprise data, the network disk system constructed by this method can easily cope with it without frequent hardware upgrades or expansions, thereby greatly reducing the operation and maintenance costs of the enterprise. At the same time, due to the decentralized design of Ceph, even if some nodes fail, the entire storage system can still maintain normal operation, ensuring the continuous availability of data and the continuity of business. In addition, the network disk system constructed by this method also significantly reduces the hardware cost of the enterprise. Compared with traditional storage solutions, Ceph provides a higher cost-effectiveness, allowing enterprises to save a lot of money when building a network disk system. More importantly, due to the outstanding performance of Ceph in security, enterprises can greatly reduce the possibility of security problems by using the network disk system constructed by this method, thereby providing more reliable data protection for the daily operation and development of the enterprise. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 Provides system architecture for microdisk storage services;

[0030] Figure 2 Upload process for files;

[0031] Figure 3 For file download process;

[0032] Figure 4 A flowchart for file sharing. DETAILED DESCRIPTION

[0033] The present invention will be further described below in conjunction with specific embodiments. It should be understood that these embodiments are only used to illustrate the present invention and are not intended to limit the scope of the present invention. In addition, it should be understood that after reading the content taught by the present invention, those skilled in the art can make various changes or modifications to the present invention, and these equivalent forms fall within the scope limited by the appended claims of the application equally.

[0034] like Figure 1 As shown, an embodiment of the present invention provides a network disk storage method based on ceph, comprising the following steps:

[0035] Users can access the distributed network disk system through the front-end interface of the Web browser. Since the network disk service is deployed in a distributed architecture, each user's request will first be intelligently distributed through the Nginx load balancing server to ensure that the request is evenly distributed. If the business request involves file read and write IO operations, the service node will establish communication with the Ceph cluster to collaborate to complete various specific operations on file resources in the network disk system. The system separates the storage cluster from the network disk server, realizes the decoupling of the storage cluster and the network disk operation service, improves the fault tolerance and availability of file data, and greatly facilitates data backup and migration.

[0036] Nginx load balancing configuration: First, you need to correctly configure the Nginx server to ensure that it can evenly distribute user requests to each service node according to the preset load balancing algorithm (such as polling, least connection, etc.). This can be achieved by modifying the Nginx configuration file, including specifying the listening port, setting the upstream server list, etc.

[0037] Integration of service nodes and Ceph cluster: Each service node needs to be configured with the corresponding client software or library to communicate with the Ceph cluster. This involves installing the Ceph client software package, configuring Ceph authentication information, and setting the file storage path.

[0038] Independent deployment of storage services: Deploy storage services (such as Ceph clusters) on independent physical or virtual servers to ensure loose coupling between them and business operation services. This improves the flexibility and scalability of the system.

[0039] Data backup and migration strategy: Develop a detailed data backup plan to regularly back up data in the Ceph cluster to other storage media or remote locations. At the same time, design a data migration strategy so that data can be quickly and safely migrated from one storage node to another when needed.

[0040] 1. Client

[0041] When users access the system through the web interface, they must first authenticate their identity. Once the user successfully passes the login authentication, the system will automatically jump to the user's personal homepage, where the user will be able to easily view and manage their personal information. In addition, the system also provides users with a detailed overview of the network disk usage. By accessing the network disk management page, users can clearly understand their network disk capacity information, including key data such as used space, remaining space, and available space percentage.

[0042] 2. Service Layer

[0043] (1) Load balancing: Use Nginx server to perform load balancing for network disk services deployed on multiple nodes. When a single node fails, the faulty node can be automatically removed to ensure service reliability and availability.

[0044] (2) File service module: The file service module implements operations on file resources, including uploading, downloading, quick transfer, copying and deleting files. It provides functions such as file directory operation and file query.

[0045] 1) Upload: In order to optimize the upload process of large files, the system allows users to customize the size of the file segment (for example, 10MB). This function is designed to solve potential problems that may arise from directly uploading large files, such as browser performance bottlenecks, excessive memory usage, and abnormal upload failures. When a user needs to upload a large file, the system will automatically cut the file into multiple smaller data blocks, namely file segments, according to the segment size set by the user. Each segment is regarded as an independent upload unit and can be uploaded one by one, such as Figure 2 shown.

[0046] 2) Download: The storage method of files on disk varies according to their size. Specifically, for unfragmented files, we follow the established storage strategy, keep their complete form on disk, and store them with the MD5 value of the file as the file name. This storage structure allows us to quickly locate the file through the specific path / {NameSpace} / {first two digits of MD5} / {MD5} / .

[0047] In this path structure:

[0048] {NameSpace} represents a namespace, which is used to distinguish data of different users or applications.

[0049] {The first two digits of MD5} are the first two characters of the file's MD5 value, which are used for directory classification to optimize file retrieval efficiency.

[0050] {MD5} is the complete MD5 value of the file, which is used as the file name to ensure the uniqueness of the file in the storage system.

[0051] When the system detects that the attribute corresponding to the {MD5} level is a file, it means that the stored file is a complete, unfragmented file. At this time, the system will decrypt the file and then directly transmit it to the front end in a streaming manner for users to download.

[0052] However, when the system finds that the {MD5} level corresponds to a folder, it indicates that a fragmented file is stored. This type of file is stored separately on the disk in the form of multiple fragments. In order to reorganize the fragmented files into a complete file for users to download, the system needs to read and merge them one by one in the order of the fragments, and finally generate a complete file. After the merger is completed, the system also transmits the file to the front-end browser in the form of a file stream for users to download and use, such as Figure 3 shown.

[0053] 3) Quick upload: Before the file is officially uploaded, the system will perform a pre-check step to confirm whether the file already exists in the user's storage space. The specific process is as follows:

[0054] a) File existence check:

[0055] The system first calculates the MD5 value of the file to be uploaded as a unique file identifier.

[0056] Next, the system uses this MD5 value as a search condition to query whether there is a matching record in the database.

[0057] If no matching record is found in the database, it means that the file is unique in the user's current storage space and needs to be further processed and uploaded.

[0058] If a matching record is found, it means that the file already exists in the system, and further judgment is required based on the storage location of the file.

[0059] b) File storage location determination and processing:

[0060] The file already exists in the user's own space:

[0061] The system saves the file's metadata information (such as file name, file size, file type, MD5 value, etc.) to the database and immediately returns a prompt to the user indicating that the upload was successful.

[0062] The file exists in another user or public space:

[0063] The system needs to copy the file from the current storage location to the user's personal storage space.

[0064] After the copy is completed, the system saves the metadata information of the newly copied file (including the new storage path, file name, file size, file type, MD5 value, etc.) to the database.

[0065] The system returns a successful upload prompt to the user, indicating that the file has been successfully added to the user's personal storage space.

[0066] (3) Group module: mainly provides role-based management functions and permission authentication for specific file resource operations.

[0067] 1) Role management: Provides authentication based on the user's permissions in the database, and supports access to external keys for user permission verification. This includes various permissions for users in the group and various permissions for shared files. Based on the database table, individual permissions in the group and individual permissions for specific resources in the group are implemented separately. At the same time, individual permissions for specific resources in the group can cover individual role permissions in the group.

[0068] 2) Sharing: used to share files / directories between users. In the process of file / directory sharing, if it is in the group space, you need to verify the user's permissions to the file / directory resource (whether there is sharing permission). Then create a sharing link, grant permissions to this sharing (view, download, save to network disk, etc.) and set the expiration time of the sharing, such as Figure 4 shown.

[0069] 3. Storage layer

[0070] The data storage layer integrates two key components: MySQL relational database and Ceph storage cluster.

[0071] The MySQL relational database is responsible for storing and managing various metadata information in the network disk system, including but not limited to file metadata information (such as file name, size, type, creation time, etc.), file directory structure (i.e., how files are organized in the system), user quota information (i.e., the upper limit of storage space available to each user), sharing records (i.e., detailed information about user shared files), group-related information (such as group name, member list, permission settings, etc.), and operation records (such as the history of file upload, download, deletion, etc.). The accurate storage and efficient retrieval of this information provide solid data support for the various functions of the network disk system.

[0072] As the core data storage component of the entire distributed network disk system, the Ceph storage cluster stores complete file data and shard information designed to improve read and write efficiency and fault tolerance. Through its own redundancy mechanism, Ceph can automatically distribute data to multiple nodes, and even if some nodes fail, data access can be quickly restored, thus ensuring business continuity and data integrity.

[0073] The system uses the Ceph storage system, which requires a hardware environment with 2n storage nodes, divided into two groups, one is the network disk storage cluster, and the other is the storage cluster of the backup module. This module mainly builds the storage cluster. The following are the basic process steps:

[0074] Prepare the machine environment to build a storage cluster. The cluster is divided into a network disk service storage cluster and a backup storage cluster.

[0075] Install the Ceph storage cluster;

[0076] Install the CephFS file system and use the mount tool to mount the CephFS file system.

[0077] The network disk system implements the core storage module based on the distributed file system. The roles that interact with it in the entire system include the client and the file service module. After passing the authentication, the client uses the CFSRPC protocol to mount the personal directory to the local computer, which is presented as an easy-to-operate file system in the client visual component. The file service module needs to request corresponding operations from the distributed storage client when executing routines such as initializing the user's personal space, querying storage resources, or setting network disk quotas.

[0078] The beneficial effects of the embodiments of the present invention are as follows:

[0079] Due to the urgent need for data security and efficient collaboration, a distributed network disk system based on Ceph storage is built within the user enterprise. This solution fully utilizes the significant features of Ceph as a storage solution, such as high performance, high availability and decentralization, and lays a solid foundation for the network disk system.

[0080] By adopting Ceph storage, the storage system at the bottom of the network disk has not only become extremely robust and highly available, but also achieved excellent scalability. As the amount of enterprise data continues to grow, the network disk system can easily cope with it without frequent hardware upgrades or expansions, thereby greatly reducing the enterprise's operation and maintenance costs. At the same time, due to Ceph's decentralized design, even if some nodes fail, the entire storage system can still maintain normal operation, ensuring the continuous availability of data and business continuity.

[0081] In addition, the Ceph-based distributed network disk system also significantly reduces the hardware costs of enterprises. Compared with traditional storage solutions, Ceph provides a higher cost-effectiveness, allowing enterprises to save a lot of money when building network disk systems. More importantly, due to Ceph's outstanding performance in security, enterprises using Ceph storage network disk systems can greatly reduce the possibility of security problems, thereby providing more reliable data protection for daily operations and development of enterprises.

Claims

1. A network disk storage method based on ceph, characterized in that: The following steps are involved: The client obtains the user's personal information for identity authentication. If the identity authentication is passed, the client displays the user's personal homepage and user storage data corresponding to the user's personal information, and obtains the user's various requests; The Nginx load balancing server in the service layer evenly distributes each request to each service node in the file service module and group module in the service layer according to the preset load balancing algorithm. If it is verified that the pre-allocated service node is faulty, the faulty service node is proposed and an available service node is allocated; Classify the requests assigned to the file service module according to file data requests and file data requests; For a request to upload file data, the file data is segmented according to the segment size preset by the user to obtain multiple file segments, and the multiple file segments are regarded as an independent upload unit and uploaded one by one until the segment integrity and file integrity are verified, and the file metadata information and file data are obtained; The storage layer includes a MySQL relational database and a Ceph storage cluster. The MySQL relational database stores file metadata information and sets a preset file name. The hardware environment of the Ceph storage cluster has 2n storage nodes, which are divided into a network disk storage cluster and a backup storage cluster. They also store file data and set a preset file name. For a request to download file data, locate the file metadata information and the storage location of the file data in the MySQL relational database and the network disk storage cluster according to the preset file name. If the storage location is located and the attribute of the file data is a file, it is directly decrypted and transmitted to the client in combination with the file metadata information. If the storage location is not located, the client prompts the user to relocate from the backup storage cluster. If the storage location is located, the user is prompted to recover and parse the data. If the storage location is located and the attribute of the file data is a folder, and multiple file fragments are saved, they are merged into file data with the attribute of a file in the order of the fragments, and the integrity is verified. If the verification passes, the file is decrypted and transmitted to the client in combination with the file metadata information. If the storage location is not located, the client prompts the user to relocate from the backup storage cluster. If the storage location is located, the user is prompted to restore the data and merge and parse it; Classify the requests assigned to the group module into role management requests and sharing requests; For role management requests, authentication is provided based on the permissions corresponding to the user's personal homepage, and external keys are accessed for user permission verification; For sharing requests, based on the group sharing permissions in the user permission verification, create a sharing link and set the sharing expiration time, and grant corresponding sharing permissions, including viewing, downloading, and saving to the network disk.

2. A ceph-based network disk storage method as claimed in claim 1, characterized in that: The preset load balancing algorithms include round-robin and least connections.

3. A ceph-based network disk storage method as claimed in claim 1, characterized in that: By modifying the configuration file of the Nginx load balancing server, the above-mentioned even distribution of various requests is achieved.

4. A ceph-based network disk storage method as claimed in claim 3, characterized in that: The configuration file includes specifying a listening port and setting an upstream server list.

5. A ceph-based network disk storage method as claimed in claim 1, characterized in that: The storage layer is deployed on an independent physical or virtual server to ensure a loose coupling relationship between it and the business operation service.

6. A ceph-based network disk storage method as claimed in claim 1, characterized in that: The user's personal homepage includes personal information, used space on the network disk, remaining space on the network disk, and the percentage of available space on the network disk.

7. A ceph-based network disk storage method as claimed in claim 1, characterized in that: The request for uploading file data also includes: Before the file data is officially uploaded, obtain the preset file name corresponding to the file data; Locate the preset file name in the Mysql relational database and the Ceph storage cluster, and if the storage location of the preset file name is not located, upload; If the storage location of the preset file name is located, it is further determined whether it is in the group. If it is in the group, it is uploaded to the user's personal Mysql relational database and the Ceph storage cluster.

Images