A method and apparatus for cloud platform file storage service

Abstract

The application relates to the field of information technology, and particularly provides a cloud platform file storage service method and device, which comprise a central file storage management unit, a plurality of storage type gateways and a plurality of storage backends, and jointly support data management and access under a cloud environment; the central file storage management unit communicates with each storage gateway through a dedicated control network. Compared with the prior art, the application can realize an efficient, flexible and cost-effective file storage solution according to storage requirements in different business scenarios.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of information technology, and specifically provides a cloud platform file storage service method and device. BACKGROUND

[0002] Under the background of rapid development of cloud computing technology, as the key support for digital transformation, cloud platforms are increasingly becoming the core infrastructure for enterprises and individuals to store, process and access data. With the diversification of cloud computing applications, the demand for file storage services is becoming increasingly complex and diverse. Traditional file storage services often focus on a single performance indicator or cost efficiency, making it difficult to fully meet the changing business scenarios in modern cloud environments.

[0003] Early cloud storage solutions are mostly based on centralized architecture, using a single storage technology such as hard disk drive (HDD)-based storage systems. Although such systems have relatively low costs, they have limitations in handling large-scale concurrent access, providing high-performance data transmission, and supporting dynamic expansion. Especially when dealing with data-intensive applications such as big data analysis and machine learning, traditional storage solutions often encounter performance bottlenecks, making it difficult to efficiently support rapid data processing and analysis. In addition, with the explosive growth of data volume, how to ensure data high availability while effectively controlling storage costs has become a major challenge for cloud service providers. In particular, for cold data that needs to be stored for a long time and has a low access frequency, traditional backup and recovery methods are often costly and inefficient, lacking effective data deduplication and compression mechanisms, resulting in serious waste of storage resources.

[0004] On the other hand, with the advancement of technology, distributed file systems such as CephFS have begun to emerge, which improve the reliability and scalability of the system by introducing data redundancy and load balancing mechanisms. However, these systems still face complexity problems in actual deployment and operation, especially in configuration, monitoring and fault recovery, requiring high professional knowledge of technical personnel, increasing operational costs. For high-performance computing and large-scale data processing scenarios, the market calls for more advanced parallel file storage solutions.

[0005] Traditional file systems, when faced with I / O-intensive applications, lack effective data parallel processing mechanisms, making it difficult to meet the extremely high data throughput requirements. In addition, how to ensure data consistency while achieving efficient cross-node collaboration is also a technical problem that needs to be solved. SUMMARY

[0006] The present application is aimed at the deficiencies of the prior art, and provides a cloud platform file storage service method with strong practicality.

[0007] The further technical task of the present application is to provide a cloud platform file storage service device which is reasonable in design and safe and applicable.

[0008] The technical solution adopted by the present application to solve its technical problems is:

[0009] A cloud platform file storage service method, comprising a central file storage management unit, a plurality of storage type gateways and a plurality of storage backends, which jointly support data management and access under a cloud environment, wherein the central file storage management unit communicates with each storage gateway through a dedicated control network.

[0010] Further, the gateway management program undertakes the task of receiving and executing instructions from the central file storage management unit, including configuration adjustment operations.

[0011] The gateway management program is encapsulated into a separate program package and stored in the file storage management unit. The gateway management program regularly inquires the file storage management unit whether there is a new program package. Once it is confirmed that there is a new version available, the file storage management unit will transmit the program package of the new version of the gateway management program to the target gateway by means of the SSH protocol, and automatically trigger the installation process to realize the non-sensing upgrade.

[0012] Further, the central file storage management unit comprises a message queue listener, a controller and a timing task trigger. The message queue listener focuses on real-time monitoring and processing of the message queue. The controller focuses on responding to HTTP requests to provide an interface for external interaction, together with the timing task trigger, which all depend on the business logic layer and are responsible for coordinating various operation logics, and also establish contact with other key modules such as the database access layer, the gateway interface layer and other application services of the cloud platform.

[0013] Further, for the standard file storage type, the storage backend relies on the block device, and the object of operation is all data blocks.

[0014] First, a data snapshot is generated, then based on this snapshot, a new temporary block device is cloned, and then the temporary block device is mounted in the specially designed backup unit.

[0015] Further, the backup unit reads the data of each block, converts it into an independent object, and uploads it to the object storage service one by one. The backup unit also records the backup metadata, i.e. the mapping table of block ID and corresponding data block MD5 checksum value, to provide a basis for incremental backup.

[0016] Further, when performing incremental backup, the backup unit compares the MD5 value of the current block data with the MD5 value of the previous backup recorded in the backup metadata. If the two values are consistent, it indicates that the data has not changed and does not need to be backed up repeatedly.

[0017] On the contrary, if the difference is found or the block ID does not exist in the metadata, the block is backed up;

[0018] During recovery, the target block device is mounted to the backup unit, the backup unit takes out the corresponding data block from the object storage according to the backup metadata, and writes back to the corresponding block position one by one.

[0019] Further, for the low-frequency file storage type, the storage backend is a CephFS distributed file system, and the backup and recovery granularity is a file;

[0020] When implementing backup, the principle of first creating a snapshot is also followed, and the CephFS snapshot mechanism provides a read-only view containing all file metadata;

[0021] First, group by performing modulo operation on the file inode, each group as a separate task, executed by an independent thread, integrated files in the form of tar packaging;

[0022] The backup metadata records the correspondence between the file name and the modification time, and in the incremental backup stage, the backup unit compares the modification timestamp of the current file with the timestamp recorded in the backup metadata, and only when there is a difference between the two, the file backup operation is triggered;

[0023] During recovery, the CephFS is mounted to the backup unit, and the backup unit writes the data in the object storage into the CephFS according to the backup metadata.

[0024] Further, for the parallel file storage type, the storage backend is a black box, and the snapshot content is directly read. In order to obtain snapshot data, an indirect strategy is adopted, that is, a new temporary file system is created through the snapshot;

[0025] Subsequently, the temporary file system is mounted to the backup unit, and the backup of the parallel file storage is at the file level.

[0026] An apparatus of a cloud platform file storage service, comprising: at least one memory and at least one processor;

[0027] The at least one memory is used to store a machine readable program;

[0028] The at least one processor is used to call the machine readable program to execute a method of a cloud platform file storage service.

[0029] Compared with the prior art, the method and the device of the cloud platform file storage service have the following outstanding beneficial effects:

[0030] The present application aims at the storage needs in different business scenarios, realizes an efficient, flexible and cost-effective file storage solution, and its beneficial effects mainly lie in the following aspects:

[0031] (1) High flexibility and wide applicability: By introducing multiple storage types such as standard, low frequency, parallel, etc., this system can provide the most suitable storage solution according to specific business needs. Whether it is a small-scale application sensitive to cost or a large data processing scenario with extremely high performance requirements, this technology can provide support to ensure the diversified construction of cloud platform application ecology.

[0032] (2) Enhanced data security and reliability: Innovative backup and recovery strategies are tailored for different storage types, such as snapshot integration of block devices and object storage, and intelligent file grouping backup of CephFS, ensuring data integrity and high availability. Through intelligent incremental backup mechanism, it effectively reduces the occupation of storage resources, while speeding up the data recovery speed and improving the business continuity;

[0033] (3) Improved system scalability and operation efficiency: The optimization of central file storage management unit and gateway design not only realizes efficient management and scheduling of resources, but also simplifies the operation complexity and reduces the maintenance cost through automatic update strategy and unified gateway interface management. Especially the introduction of parallel file storage breaks the expansion bottleneck of traditional storage systems, supporting seamless expansion of business;

[0034] (4) Enhanced network performance and security: Through careful design of network architecture, using Gigabit control network and Gigabit business network, combined with network card binding technology, it ensures the safe transmission of control commands and high speed of data access, while effectively alleviating the bandwidth bottleneck, providing a solid network foundation for data interaction in cloud environment.

[0035] This patent technology solution significantly improves the flexibility, performance, cost-effectiveness, data security and system management efficiency of file storage services, providing a comprehensive optimization solution for cloud platform data storage and management, and effectively promoting the in-depth application and development of cloud computing technology. BRIEF DESCRIPTION OF DRAWINGS

[0036] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiment or prior art description. Obviously, the drawings in the following description are some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor.

[0037] ATTACHMENT Figure 1A network architecture diagram in a method of a cloud platform file storage service;

[0038] Attached Figure 2 A central file storage management unit architecture diagram in a method of a cloud platform file storage service. DETAILED DESCRIPTION

[0039] In order to better understand the scheme of the present application, the present application will be further described in detail below in conjunction with specific embodiments. Obviously, the described embodiments are only some of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative labor fall within the scope of protection of the present application.

[0040] A best embodiment is given below:

[0041] File storage service, as an indispensable basic component of cloud platform, is crucial for building a diversified cloud application ecosystem. It not only gives users the convenience of accessing shared network file systems, but also provides a solid data storage foundation for various core services of cloud computing, such as computing, database, big data analysis, and machine learning.

[0042] Therefore, file storage service must have high flexibility to adapt to a wide range of application scenarios and accordingly introduce diversified storage types. For example, for lightweight application systems and smaller-scale file sharing needs, the service will adopt HDD-based standard file storage types to ensure economic efficiency; for video monitoring and other scenarios that are write-intensive, read less, involve massive data, and require strict cost control, low-frequency file storage types using erasure coding technology will be deployed; for machine learning and big data analysis and other fields with strict performance requirements, high-performance parallel file storage types based on SSD will be adopted to meet the needs of high-speed data interaction and processing. In summary, each file storage type needs to rely on a specially designed storage backend to balance access efficiency and usage cost in different business scenarios.

[0043] Specifically, the standard file storage type is based on block devices to build its storage backend, and the file storage gateway is deployed on the virtual machine of the tenant side. The gateway and the tenant virtual machine are in the same virtual private network.

[0044] In a specific implementation, the file storage gateway hosts tenant data by mounting and formatting the file system on the block device, while the gateway software running inside the gateway virtual machine provides clients with shared file storage services of NFS and SMB protocols. The core advantage of this architecture is that it directly manages the metadata of the shared file system using the block device file system, which significantly improves the metadata processing performance compared to the strategy of separating data and metadata, and effectively utilizes the local file system cache mechanism. However, this design has obvious limitations: since the block device is usually limited to a single mount point, the file storage gateway cannot be horizontally expanded, and its performance upper limit is determined by a single block storage device and a single gateway process. In summary, the standard file storage type that relies on block devices as a storage backend, although it performs well in improving metadata processing efficiency and utilizing cache, its scalability and performance upper limit make it more suitable for scenarios with relatively low file sharing requirements.

[0045] The low-frequency file storage type uses CephFS as its storage backend. As a high-performance distributed file system, CephFS achieves a good balance between storage cost and performance by integrating erasure code technology. The file storage gateway under this scheme is deployed on the management side's physical server, equipped with a special gateway application, and also provides clients with network file system sharing services of NFS and SMB protocols. The gateway software uses the libCephFS library to communicate seamlessly with CephFS, ensuring efficient execution of the service. The significant advantage of this architecture is that CephFS, as a distributed system, naturally supports multiple concurrent read and write operations, allowing the file storage gateway to easily scale horizontally according to demand, theoretically eliminating performance bottlenecks. However, the disadvantage of this scheme is that the CephFS component is complex, increasing the operation and management costs, and for small file sharing application scenarios, it may be difficult to maximize cost-effectiveness. Therefore, the low-frequency file storage type based on CephFS, with its excellent scalability and strong adaptability to large-scale file sharing scenarios (such as video surveillance), has become an ideal choice for such needs.

[0046] Parallel file storage systems are designed to meet extreme performance challenges, serving applications that have extremely high requirements for data throughput and access speed. Traditional block device-based or CephFS-based storage solutions, although each has its own advantages, often appear to be inadequate when faced with such high-intensity performance requirements. Therefore, in order to break through these performance bottlenecks, parallel file storage types usually rely on highly optimized cluster technologies such as GPFS or BeeGFS, which are equipped with self-developed high-performance shared file system protocols. These professional parallel file systems, through carefully designed architecture, not only make full use of every node resource in the cluster to realize parallel reading and writing of data, but also ensure the efficiency and consistency of data access through their unique file system protocols. They maximize the potential of the cluster through intelligent data distribution strategies, advanced caching mechanisms, and optimized I / O scheduling algorithms. Therefore, when providing parallel file storage products in the cloud platform, we directly choose professional parallel file storage clusters as the storage backend of the parallel file storage type.

[0047] As shown in Figure 1 , the method of a cloud platform file storage service in this embodiment includes a central file storage management unit, multiple storage type gateways, and multiple storage backends, which jointly support efficient data management and access in a cloud environment. In this architecture, the file storage management unit plays the role of a command center, communicating with each storage gateway through a dedicated control network.

[0048] The design of the control network focuses more on security than on bandwidth capacity, so we use Gigabit network standards and use network card bonding technology (bond1) to enhance the reliability and security of the network link, ensuring that all management instructions and monitoring information are safely and accurately transmitted. At the data access level, clients interact with storage gateways through a business network, which is designed to fully consider the speed requirements of the business. At the same time, the data flow between the storage gateway and its respective storage backend relies on the storage network, which also carries a large amount of data transmission tasks. Given the high bandwidth requirements of the business network and the storage network, we use Gigabit networks and combine network card bonding technology (bond4) to maximize network throughput, reduce potential bandwidth bottlenecks, and ensure the continuity and efficiency of data flow.

[0049] Standard file storage gateways and low-frequency file storage gateways are similar in architecture design, both integrating two core components: a gateway management program and a gateway service program. The gateway management program is responsible for receiving and executing instructions from the file storage management unit, including configuration adjustments and other operations.

[0050] On the other hand, the gateway service program focuses on providing stable and reliable shared network file system services to the client, ensuring efficient data access. This design separates the gateway management program from the service program, which has the advantage that when the gateway management program needs to be upgraded or maintained, the operation can be carried out independently without interrupting the operation of the gateway service program, thereby ensuring continuous access of the client and avoiding any service interruption caused by maintenance. More specifically, the gateway management program is encapsulated into a separate program package and stored in the file storage management unit.

[0051] The gateway management program regularly inquires whether there is a new program package in the file storage management unit. Once it confirms that there is a new version available, the file storage management unit will transmit the program package of the new version of the gateway management program to the target gateway through the SSH protocol and automatically trigger the installation process to achieve non-sensory upgrade. This automated update strategy greatly simplifies the operation and maintenance work, while ensuring the immediate application of the latest system functions, providing a strong guarantee for the long-term stable operation of the file storage service.

[0052] As shown in Figure 2 , the architecture of the file storage management unit, the main entrance of the unit includes a message queue listener, a controller and a timing task trigger, each part has its own function, the message queue listener focuses on real-time monitoring and processing of the message queue to ensure the timeliness of information flow; the controller focuses on responding to HTTP requests to provide an interface for external interaction. Both of them, together with the timing task trigger, depend on the business logic layer, which serves as the core hub responsible for coordinating various operation logics, and also establishes contact with other key modules such as the database access layer, the gateway interface layer, and other application services of the cloud platform. In order to better adapt and integrate various file storage gateways, the gateway interface layer plays a crucial role, it achieves unified management of different types of gateways by abstracting a series of standardized gateway management interfaces. This includes but is not limited to file system instance management, mount point management, access rule management, snapshot management, backup management and other core interfaces. These interface designs cover both standard and low-frequency file storage gateways, and are also applicable to parallel file storage clusters such as GPFS, BeeGFS, etc., ensuring the universality and ease of use of the interface, greatly simplifying the complexity of system integration and maintenance.

[0053] It is worth noting that the backup management interface needs to be customized according to different storage backends due to the diversity of storage backend technologies. In order to maintain low coupling and high cohesion between system components, an independent backup unit is added to focus on data backup and recovery functions. During the backup operation, this unit is responsible for extracting data from the storage backend and storing it in the object storage system. During the data recovery process, the backup data in the object storage is written back to the original storage backend, ensuring data integrity and recoverability.

[0054] For standard file storage types, the storage backend relies on block devices, so the objects of the backup and recovery operations are data blocks. To ensure data integrity and consistency during backup, the following strategies are adopted: First, generate a data snapshot to effectively avoid data damage caused by data changes; then, based on this snapshot, clone a new temporary block device; next, the temporary block device is mounted to the specially designed backup unit. The backup unit reads the data of each block, converts it into independent objects, and uploads them one by one to the object storage service. In this process, the backup unit also records the backup metadata, which is a mapping table of block ID and corresponding data block MD5 checksum value, providing a basis for incremental backup. When performing incremental backup, the backup unit compares the MD5 value of the current block data with the MD5 value recorded in the backup metadata of the previous backup. If the two values are consistent, it means that the data has not changed, and there is no need to repeat the backup, effectively reducing unnecessary storage consumption and processing time. On the contrary, if a difference is found, or the block ID does not exist in the metadata, the block is backed up to ensure that all updated content is saved, achieving precise synchronization of data.

[0055] During recovery, the target block device is mounted to the backup unit, and the backup unit retrieves the corresponding data blocks from the object storage based on the backup metadata and writes them back to the corresponding block positions one by one. Thanks to the linear structure of the block device and the independent characteristics between blocks, the backup and recovery processes can use multi-threading technology to further accelerate.

[0056] For low-frequency file storage type, its storage backend is CephFS distributed file system, and the granularity of backup and recovery is file. When implementing backup, the principle of creating snapshot first is also followed to ensure data stability during backup operation and avoid inconsistency risk caused by data changes. CephFS snapshot mechanism provides a read-only view containing all file metadata (including key inode information). Due to the complex characteristics of CephFS file system, such as non-linear arrangement of files, interlaced soft and hard links, the backup scheme adopts a different method from block device backup. First, the file inode is grouped by modulo operation, solving the problem of hard link file backup, ensuring the consistency of associated files, and each group is executed as a separate task by an independent thread. Tar packaging is used to integrate files to avoid redundant backup caused by soft links. Backup metadata records the correspondence between file name and modification time. In the incremental backup stage, the backup unit compares the current file modification timestamp with the timestamp recorded in the backup metadata, and only when there is a difference, the file backup operation is triggered to ensure efficient use of backup resources and avoid unnecessary duplication of work.

[0057] When recovering, CephFS is mounted to the backup unit, and the backup unit writes data in the object storage into CephFS according to the backup metadata. The entire backup and recovery system takes into account the complexity of the file system and fully utilizes its distributed characteristics, providing an efficient and reliable data protection solution for low-frequency file storage type through intelligent grouping, efficient multi-threaded backup strategy and accurate incremental backup judgment.

[0058] For parallel file storage type, its storage backend is a black box, and it is not easy to read snapshot content directly. Therefore, in order to obtain snapshot data, we adopt an indirect strategy: creating a new temporary file system through snapshot. Then, mount this temporary file system to the backup unit, and the backup core of parallel file storage also focuses on file level, which means that the backup process is similar to the former.

[0059] In summary, file storage service provides diversified storage types to meet the needs of different business scenarios, and through fine design of architecture and management strategy, it realizes the efficiency, flexibility and reliability of data storage. The division of standard, low-frequency and parallel file storage types, combined with intelligent backup and recovery mechanism, ensures the safety of data and the continuity of business. The design of central file storage management unit and gateway shows high system integration capability, and the customized backup strategy for various storage backends further improves the safety of data.

[0060] Based on the above method, the device of a cloud platform file storage service in the embodiment comprises at least one memory and at least one processor.

[0061] The at least one memory is configured to store machine readable programs.

[0062] The at least one processor is configured to invoke the machine readable programs to execute a method of a cloud platform file storage service.

[0063] The above specific embodiments are only specific cases of the present application, and the patent protection scope of the present application includes but is not limited to the above specific embodiments. Any technical solution meeting the above specific embodiments of the present application and any appropriate changes or replacements made by ordinary technical personnel in the art shall fall within the patent protection scope of the present application.

[0064] Although the embodiments of the present application have been shown and described, it will be understood by those skilled in the art that various changes, modifications, replacements and variations can be made to the embodiments without departing from the principles and spirits of the present application, and the scope of the present application is defined by the appended claims and their equivalents.

Claims

1. A method for providing file storage services on a cloud platform, characterized in that: It includes a central file storage management unit, multiple storage type gateways and multiple storage backends to jointly support data management and access in the cloud environment. The central file storage management unit communicates with each storage gateway through a dedicated control network. For standard file storage types, the storage backend relies on block devices, and the objects of operation are all data blocks; First, a data snapshot is generated. Then, a new temporary block device is cloned based on the snapshot. Then, the temporary block device is mounted to a specially designed backup unit. For low-frequency file storage, the storage backend is the CephFS distributed file system, and the backup and recovery granularity is file. When implementing backups, the same principle of creating a snapshot first is followed. The CephFS snapshot mechanism provides a read-only view of all file metadata. First, the files are grouped by performing a modulo operation on the file inode. Each group is treated as a separate task and executed by an independent thread. The files are then packaged in tar format. The backup metadata records the correspondence between the file name and the modification time. During the incremental backup phase, the backup unit compares the modification timestamp of the current file with the timestamp recorded in the backup metadata. The backup operation of the file is triggered only when there is a difference between the two. During recovery, CephFS is mounted to the backup unit, and the backup unit writes the data in the object storage to CephFS according to the backup metadata guidance; Parallel file storage: The storage backend is a black box that directly reads snapshot content. To obtain snapshot data, an indirect strategy is adopted to create a new temporary file system from the snapshot. Subsequently, this temporary file system is mounted to the backup unit, and the parallel file storage is backed up at the file level.

2. A cloud platform file storage service method according to claim 1, characterized in that: The gateway management program is responsible for receiving and executing instructions from the central file storage management unit, including configuration adjustment operations; The gateway management program is encapsulated into an independent program package and stored in the file storage management unit. The gateway management program periodically asks the file storage management unit whether there is a new version of the program package. Once it is confirmed that a new version is available, the file storage management unit will use the SSH protocol to transfer the new version of the gateway management program package to the target gateway and automatically trigger the installation process to achieve seamless upgrade.

3. A cloud platform file storage service method according to claim 2, characterized in that: The central file storage management unit includes a message queue listener, a controller and a scheduled task trigger. The message queue listener focuses on real-time monitoring and processing of the message queue, and the controller focuses on responding to HTTP requests and providing an interface for external interaction. Together with the scheduled task trigger, they all rely on the business logic layer and are responsible for coordinating various operation logics. At the same time, they also establish connections with other key modules such as the database access layer, the gateway interface layer, and other application services of the cloud platform.

4. A cloud platform file storage service method according to claim 3, characterized in that: The backup unit reads the data of each block, converts it into an independent object, and uploads it one by one to the object storage service. The backup unit also records the backup metadata, that is, the mapping table of the block ID and the corresponding data block MD5 checksum value, providing a basis for incremental backup.

5. A cloud platform file storage service method according to claim 4, characterized in that: When performing incremental backup, the backup unit compares the MD5 value of the current block data with the MD5 value of the previous backup recorded in the backup metadata. If the two values ​​are consistent, it indicates that the data has not changed and there is no need to back up again; On the contrary, if a difference is found, or the block ID does not exist in the metadata, the block is backed up; During recovery, the target block device will be mounted to the backup unit. The backup unit will retrieve the corresponding data blocks from the object storage based on the backup metadata and write them back to the corresponding block locations one by one.

6. A cloud platform file storage service device, characterized in that: include: at least one memory and at least one processor; The at least one memory is configured to store a machine-readable program; The at least one processor is configured to call the machine-readable program to execute the method according to any one of claims 1 to 5.

Images