Cloud platform file storage service method and device

Abstract

The invention relates to the technical field of information, and particularly provides a cloud platform file storage service method and device, which comprises a central file storage management unit, multiple storage type gateways and multiple storage rear ends, which jointly support data management and access in a cloud environment. And the central file storage management unit communicates with each storage gateway through a special control network. Compared with the prior art, the file storage method and the file storage device have the advantages that efficient and flexible file storage solutions with remarkable cost benefits can be realized according to storage requirements in different business scenes.

Description

Technical Field

[0001] The present invention relates to the field of information technology, and specifically provides a method and device for cloud platform file storage service. Background Art

[0002] With the rapid development of cloud computing technology, cloud platforms, as a key support for digital transformation, are increasingly becoming the core infrastructure for enterprises and individuals to store, process and access data. With the diversification of cloud computing applications, the demand for file storage services is becoming increasingly complex and diverse. Traditional file storage services often focus on a single performance indicator or cost efficiency, which makes it difficult to fully meet the changing business scenarios in the modern cloud environment.

[0003] Early cloud storage solutions were mostly based on a centralized architecture and used a single storage technology, such as hard disk drive (HDD)-based storage systems. Although such systems are relatively low-cost, they have limitations in handling large-scale concurrent access, providing high-performance data transmission, and supporting dynamic expansion. Especially when dealing with data-intensive applications such as big data analysis and machine learning, traditional storage solutions often encounter performance bottlenecks and cannot efficiently support the rapid processing and analysis of data. In addition, with the explosive growth of data volume, how to effectively control storage costs while ensuring high data availability has become a major challenge facing cloud service providers. In particular, for cold data that requires long-term storage and has a low access frequency, traditional backup and recovery methods are often costly and inefficient, lacking effective data deduplication and compression mechanisms, resulting in serious waste of storage resources.

[0004] On the other hand, with the advancement of technology, distributed file systems such as CephFS have begun to emerge. They have improved the reliability and scalability of the system by introducing data redundancy and load balancing mechanisms. However, these systems still face complexity issues in actual deployment and operation and maintenance, especially in configuration, monitoring and fault recovery, which require high professional knowledge of technicians and increase operating costs. For high-performance computing and large-scale data processing scenarios, the market calls for more advanced parallel file storage solutions.

[0005] Traditional file systems are unable to meet the extremely high data throughput requirements when facing I / O-intensive applications due to the lack of effective data parallel processing mechanisms. In addition, how to achieve efficient cross-node collaboration while ensuring data consistency is also a technical problem that needs to be solved urgently. Summary of the invention

[0006] The present invention aims to address the deficiencies of the above-mentioned prior art and provides a method for cloud platform file storage service with strong practicality.

[0007] A further technical task of the present invention is to provide a cloud platform file storage service device that is rationally designed, safe and applicable.

[0008] The technical solution adopted by the present invention to solve its technical problem is:

[0009] A method for cloud platform file storage service includes a central file storage management unit, multiple storage type gateways and multiple storage backends, which jointly support data management and access in a cloud environment. The central file storage management unit communicates with each storage gateway through a dedicated control network.

[0010] Furthermore, the gateway management program is tasked with receiving and executing instructions from the central file storage management unit, including configuration adjustment operations;

[0011] The gateway management program is encapsulated into an independent program package and saved in the file storage management unit. The gateway management program periodically asks the file storage management unit whether there is a new version of the program package. Once it is confirmed that a new version is available, the file storage management unit will use the SSH protocol to transfer the new version of the gateway management program package to the target gateway and automatically trigger the installation process to achieve seamless upgrade.

[0012] Furthermore, the central file storage management unit includes a message queue listener, a controller and a scheduled task trigger. The message queue listener focuses on real-time monitoring and processing of the message queue, and the controller focuses on responding to HTTP requests and providing an interface for external interaction. Together with the scheduled task trigger, they all rely on the business logic layer and are responsible for coordinating various operation logics. At the same time, they also establish connections with other key modules such as the database access layer, the gateway interface layer, and other application services of the cloud platform.

[0013] Furthermore, for standard file storage types, the storage backend relies on block devices, and the objects of operation are all data blocks;

[0014] First, a data snapshot is generated, and then a new temporary block device is cloned based on the snapshot. Then, the temporary block device is mounted to a specially designed backup unit.

[0015] Furthermore, the backup unit reads the data of each block, converts it into an independent object, and uploads it to the object storage service one by one. The backup unit also records the backup metadata, that is, a mapping table of block ID and corresponding data block MD5 checksum value, providing a basis for incremental backup.

[0016] Furthermore, when performing incremental backup, the backup unit compares the MD5 value of the current block data with the MD5 value of the previous backup recorded in the backup metadata. If the two values ​​are consistent, it indicates that the data has not changed and there is no need to repeat the backup;

[0017] On the contrary, if a difference is found, or the block ID does not exist in the metadata, the block is backed up;

[0018] During recovery, the target block device will be mounted to the backup unit. The backup unit will retrieve the corresponding data blocks from the object storage based on the backup metadata and write them back to the corresponding block locations one by one.

[0019] Furthermore, for low-frequency file storage types, the storage backend is the CephFS distributed file system, and the granularity of backup and recovery is files;

[0020] When implementing backup, the same principle of creating snapshots first is followed. The CephFS snapshot mechanism provides a read-only view containing all file metadata.

[0021] First, the files are grouped by performing a modulo operation on the file inode. Each group is treated as a separate task and executed by an independent thread. The files are packaged in tar format.

[0022] The backup metadata records the correspondence between the file name and the modification time. During the incremental backup phase, the backup unit compares the modification timestamp of the current file with the timestamp recorded in the backup metadata. The backup operation of the file is triggered only when there is a difference between the two.

[0023] During recovery, CephFS is mounted to the backup unit, and the backup unit writes the data in the object storage to CephFS according to the backup metadata guidance.

[0024] Furthermore, in the parallel file storage type, the storage backend is a black box that directly reads the snapshot content. In order to obtain the snapshot data, an indirect strategy is adopted to create a new temporary file system through the snapshot;

[0025] Subsequently, this temporary file system is mounted to the backup unit, and the parallel file storage is backed up at the file level.

[0026] A cloud platform file storage service device, comprising: at least one memory and at least one processor;

[0027] The at least one memory is used to store a machine-readable program;

[0028] The at least one processor is used to call the machine-readable program to execute a method for cloud platform file storage service.

[0029] Compared with the prior art, the cloud platform file storage service method and device of the present invention have the following outstanding beneficial effects:

[0030] The present invention realizes an efficient, flexible and cost-effective file storage solution for storage requirements in different business scenarios, and its beneficial effects are mainly reflected in the following aspects:

[0031] (1) High flexibility and wide applicability: By introducing multiple storage types such as standard, low-frequency, and parallel, the system can provide the most suitable storage solution according to specific business needs. Whether it is a small-scale application that is cost-sensitive or a big data processing scenario with extremely high performance requirements, this technology can provide support and ensure the diversified construction of the cloud platform application ecosystem.

[0032] (2) Enhanced data security and reliability: Innovative backup and recovery strategies are customized for different storage types, such as snapshots of block devices and object storage integration, as well as CephFS's intelligent file grouping backup, to ensure data integrity and high availability. The intelligent incremental backup mechanism effectively reduces storage resource usage, while speeding up data recovery and improving business continuity.

[0033] (3) Improved system scalability and operation and maintenance efficiency: The optimization of the central file storage management unit and gateway design not only realizes efficient management and scheduling of resources, but also simplifies the operation and maintenance complexity and reduces maintenance costs through automated update strategies and unified gateway interface management. In particular, the introduction of parallel file storage breaks the expansion bottleneck of traditional storage systems and supports seamless business expansion;

[0034] (4) Enhanced network performance and security: Through a carefully designed network architecture, the use of a Gigabit control network and a 10 Gigabit business network, combined with network card binding technology, ensures the secure transmission of control commands and high-speed data access, while effectively alleviating bandwidth bottlenecks and providing a solid network foundation for data interaction in a cloud environment.

[0035] Through a series of innovative designs, this patented technical solution significantly improves the flexibility, performance, cost-effectiveness, data security and system management efficiency of file storage services, provides a comprehensive optimization solution for data storage and management of cloud platforms, and effectively promotes the in-depth application and development of cloud computing technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0037] Attached Figure 1It is a network architecture diagram in a method of cloud platform file storage service;

[0038] Attached Figure 2 The present invention is an architectural diagram of a central file storage management unit in a method of a cloud platform file storage service. DETAILED DESCRIPTION

[0039] In order to enable those skilled in the art to better understand the solution of the present invention, the present invention is further described in detail below in conjunction with specific implementation methods. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0040] A best embodiment is given below:

[0041] As an indispensable basic component of the cloud platform, file storage service is crucial to building a diversified cloud application ecosystem. It not only provides users with the convenience of accessing shared network file systems, but also provides a solid data storage foundation for various core services of cloud computing, such as computing, databases, big data analysis, and machine learning.

[0042] In view of this, file storage services must be highly flexible to adapt to a wide range of application scenarios and launch a variety of storage types accordingly. For example, for lightweight application systems and smaller-scale file sharing needs, the service will use standard file storage types based on HDD to ensure economic efficiency; for scenarios such as video surveillance that are write-intensive, read-less, involve massive amounts of data, and have strict cost control, low-frequency file storage types using erasure coding technology will be deployed; as for fields such as machine learning and big data analysis that have strict performance requirements, high-performance parallel file storage types based on SSD will be adopted to meet the needs of high-speed data interaction and processing. In short, each file storage type needs to rely on a specially designed storage backend to balance access efficiency and usage costs in different business scenarios.

[0043] Specifically, the standard file storage type builds its storage backend based on block devices, while the file storage gateway is deployed on the virtual machine on the tenant side. The gateway and the tenant virtual machine are in the same virtual private network.

[0044] In the specific implementation, the file storage gateway hosts tenant data by mounting and formatting the file system on the block device. At the same time, the gateway software running inside the gateway virtual machine provides the client with shared file storage services of NFS and SMB protocols. The core advantage of this architecture is that it uses the block device file system to directly manage the metadata of the shared file system. Compared with the strategy of separating data from metadata, it significantly improves the metadata processing performance and can effectively utilize the local file system cache mechanism. However, this design also has obvious limitations: since block devices are usually limited to a single mount point, the file storage gateway cannot achieve horizontal expansion, and its performance ceiling is determined by a single block storage device and a single gateway process. In summary, although the standard file storage type that relies on block devices as the storage backend performs well in improving metadata processing efficiency and utilizing cache, its scalability and performance ceiling make it more suitable for scenarios with relatively low file sharing requirements.

[0045] The low-frequency file storage type uses CephFS as its storage backend. As a high-performance distributed file system, CephFS achieves a good balance between storage cost and performance by integrating erasure coding technology. The file storage gateway under this solution is deployed on the physical server on the management side, equipped with a proprietary gateway application, and also provides network file system sharing services of NFS and SMB protocols to clients. The gateway software uses the libCephFS library to communicate seamlessly with CephFS to ensure efficient execution of the service. The significant advantage of this architecture is that CephFS, as a distributed system, naturally supports multi-client concurrent read and write operations, so that the file storage gateway can be easily horizontally expanded according to demand, theoretically eliminating performance bottlenecks. However, the disadvantage of this solution is that the complexity of CephFS components is high, and the operation and maintenance and management costs increase accordingly. For small file sharing application scenarios, it may be difficult to maximize cost-effectiveness. Therefore, the low-frequency file storage type built on CephFS has become an ideal choice for such needs with its excellent scalability and strong adaptability to large-scale file sharing scenarios (such as video surveillance).

[0046] Parallel file storage systems are designed to meet extreme performance challenges. They serve application scenarios that have extremely high requirements for data throughput and access speed. Traditional storage solutions based on block devices or CephFS, although they have their own advantages, often seem to be unable to cope with such high-intensity performance requirements. Therefore, in order to break through these performance bottlenecks, parallel file storage types usually rely on highly optimized cluster technologies, such as GPFS or BeeGFS, which are equipped with self-developed high-performance shared file system protocols. These professional-level parallel file systems, through carefully designed architectures, can not only fully utilize the resources of each node in the cluster to achieve parallel reading and writing of data, but also ensure the efficiency and consistency of data access through their unique file system protocols. They maximize the potential of clusters through intelligent data distribution strategies, advanced caching mechanisms, and optimized I / O scheduling algorithms. Therefore, when the cloud platform provides parallel file storage product types, we directly choose professional parallel file storage clusters as the storage backend of parallel file storage types.

[0047] like Figure 1 As shown, a cloud platform file storage service method in this embodiment includes a central file storage management unit, multiple storage type gateways and multiple storage backends, which together support efficient data management and access in a cloud environment. In this architecture, the file storage management unit plays the role of a command center and communicates with each storage gateway through a dedicated control network.

[0048] The design of the control network focuses on security more than bandwidth capacity. Therefore, we use the Gigabit network standard and use network card bonding technology (bond1) to enhance the reliability and security of the network link to ensure that all management instructions and monitoring information are transmitted safely and accurately. At the data access level, the client interacts with the storage gateway through the business network. This network design fully considers the business's requirements for speed. At the same time, the data flow between the storage gateway and its respective storage backend depends on the storage network, which also carries a large amount of data transmission tasks. In view of the high bandwidth requirements of the business network and the storage network, we use a 10G network and combine it with network card bonding technology (bond4) to maximize network throughput, reduce potential bandwidth bottlenecks, and ensure the continuity and efficiency of data flow.

[0049] The standard file storage gateway and the low-frequency file storage gateway are similar in architecture design. Both integrate two core components: the gateway management program and the gateway service program. The gateway management program is responsible for receiving and executing instructions from the file storage management unit, including configuration adjustments and other operations.

[0050] On the other hand, the gateway service program focuses on providing stable and reliable shared network file system services to the client to ensure efficient data access. The advantage of this design of separating the gateway management program from the service program is that when the gateway management program needs to be upgraded or maintained, the operation can be performed independently without interrupting the operation of the gateway service program, thereby ensuring continuous access by the client and avoiding any service interruption caused by maintenance. More specifically, the gateway management program is encapsulated into an independent package and stored in the file storage management unit.

[0051] The gateway management program periodically asks the file storage management unit whether there is a new version of the program package. Once it is confirmed that a new version is available, the file storage management unit will use the SSH protocol to transfer the new version of the gateway management program package to the target gateway and automatically trigger the installation process to achieve a seamless upgrade. This automated update strategy greatly simplifies operation and maintenance work, while ensuring the immediate application of the latest system functions, providing a strong guarantee for the long-term stable operation of the file storage service.

[0052] like Figure 2 As shown in Figure 1, the architecture of the file storage management unit. The main entrance of the unit includes a message queue listener, a controller, and a scheduled task trigger. Each part has its own responsibilities. The message queue listener focuses on real-time monitoring and processing of the message queue to ensure the timeliness of information flow; the controller focuses on responding to HTTP requests and providing an interface for external interaction. Both of them, together with the scheduled task trigger, rely on the business logic layer. As the core hub, this layer is responsible for coordinating various operation logics and also establishes connections with other key modules such as the database access layer, the gateway interface layer, and other application services of the cloud platform. In order to better adapt to and integrate a variety of file storage gateways, the gateway interface layer plays a vital role. It realizes unified management of different types of gateways by abstracting a series of standardized gateway management interfaces. This includes but is not limited to core interfaces such as file system instance management, mount point management, access rule management, snapshot management, and backup management. These interface designs cover both standard and low-frequency file storage gateways, and are also applicable to parallel file storage clusters such as GPFS and BeeGFS, ensuring the universality and ease of use of the interface, greatly simplifying the complexity of system integration and maintenance.

[0053] It is worth noting that the backup management interface responds to the diversity of storage backend technologies, and its specific implementation strategy needs to be customized according to different storage backends. In order to maintain low coupling and high cohesion between system components, an independent backup unit is specially added to focus on data backup and recovery functions. During the backup operation, this unit is responsible for extracting data from the storage backend and storing it in the object storage system; in the data recovery stage, it reverses the operation and writes the backup data in the object storage back to the original storage backend, ensuring the integrity and recoverability of the data.

[0054] For standard file storage types, the storage backend relies on block devices, so in backup and recovery operations, the objects of operation are all data blocks. To ensure the integrity and consistency of data during the backup process, the following strategies are adopted: First, a data snapshot is generated to effectively avoid the problem of backup data corruption caused by data changes; then, based on this snapshot, a new temporary block device is cloned; then, the temporary block device is mounted to a specially designed backup unit. The backup unit reads the data of each block, converts it into an independent object, and uploads it one by one to the object storage service. During this process, the backup unit also records the backup metadata, that is, the mapping table between the block ID and the corresponding data block MD5 checksum value, which provides a basis for incremental backup. When performing incremental backup, the backup unit compares the MD5 value of the current block data with the MD5 value of the previous backup recorded in the backup metadata. If the two values ​​are consistent, it means that the data has not changed and there is no need to repeat the backup, which effectively reduces unnecessary storage consumption and processing time; on the contrary, if a difference is found or the block ID does not exist in the metadata, the block is backed up to ensure that all updated content is saved and accurate data synchronization is achieved.

[0055] During recovery, the target block device will be mounted to the backup unit, which will retrieve the corresponding data blocks from the object storage based on the backup metadata and write them back to the corresponding block locations one by one. Thanks to the linear structure of the block device and the lack of dependencies between blocks, both the backup and recovery processes can be further accelerated using multi-threading technology.

[0056] For low-frequency file storage types, the storage backend is the CephFS distributed file system, and the granularity of backup and recovery is files. When implementing backup, the principle of creating snapshots first is also followed to ensure data stability during the backup operation and avoid the risk of inconsistency caused by data changes. The CephFS snapshot mechanism provides a read-only view containing all file metadata (including key inode information). In view of the complex characteristics of the CephFS file system, such as non-linearly arranged files and the interlacing of soft links and hard links, the backup solution adopts a different method from block device backup. First, the file inode is grouped by performing modulo operations on the file inode, which solves the problem that hard-linked files need to be backed up at the same time and ensures the consistency of associated files. Each group is executed as a separate task by an independent thread, and the files are integrated by tar packaging, avoiding redundant backup caused by soft links. The backup metadata records the correspondence between the file name and the modification time. In the incremental backup stage, the backup unit compares the modification timestamp of the current file with the timestamp recorded in the backup metadata. Only when there is a difference between the two, the backup operation of the file is triggered, ensuring the effective use of backup resources and avoiding unnecessary duplication of work.

[0057] During recovery, CephFS is mounted to the backup unit, and the backup unit writes the data in the object storage to CephFS based on the backup metadata. The entire backup and recovery system not only takes into account the complexity of the file system, but also gives full play to its distributed characteristics. Through intelligent grouping, efficient multi-threaded backup strategy and accurate incremental backup judgment, it creates a set of high-efficiency and high-reliability data protection solutions for low-frequency file storage types.

[0058] The parallel file storage type has a black box storage backend, and it is not easy to read the snapshot content directly. Therefore, in order to obtain the snapshot data, we adopted an indirect strategy: create a new temporary file system through the snapshot. Then, mount this temporary file system to the backup unit. The backup core of the parallel file storage also focuses on the file level, which means that the backup process is similar to the former.

[0059] In summary, the file storage service not only meets the needs of different business scenarios by providing a variety of storage types, but also achieves high efficiency, flexibility and reliability of data storage through carefully designed architecture and management strategies. The division of standard, low-frequency and parallel file storage types, combined with intelligent backup and recovery mechanisms, ensures data security and business continuity. The central file storage management unit and gateway design demonstrate a high degree of system integration capabilities, and the backup strategies customized for various storage backends further enhance data security.

[0060] Based on the above method, a cloud platform file storage service device in this embodiment includes: at least one memory and at least one processor;

[0061] The at least one memory is used to store a machine-readable program;

[0062] The at least one processor is used to call the machine-readable program to execute a method for cloud platform file storage service.

[0063] The above-mentioned specific implementations are only specific cases of the present invention. The patent protection scope of the present invention includes but is not limited to the above-mentioned specific implementations. Any technical solutions that conform to the above-mentioned specific implementations of the present invention and any appropriate changes or substitutions made by ordinary technicians in the relevant technical field shall fall within the patent protection scope of the present invention.

[0064] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A method for cloud platform file storage service, characterized in that: It includes a central file storage management unit, multiple storage type gateways and multiple storage backends, which jointly support data management and access in a cloud environment. The central file storage management unit communicates with each storage gateway through a dedicated control network.

2. A method for cloud platform file storage service according to claim 1, characterized in that: The gateway management program is responsible for receiving and executing instructions from the central file storage management unit, including configuration adjustment operations; The gateway management program is encapsulated into an independent program package and saved in the file storage management unit. The gateway management program periodically asks the file storage management unit whether there is a new version of the program package. Once it is confirmed that a new version is available, the file storage management unit will use the SSH protocol to transfer the new version of the gateway management program package to the target gateway and automatically trigger the installation process to achieve seamless upgrade.

3. A method for cloud platform file storage service according to claim 2, characterized in that: The central file storage management unit includes a message queue listener, a controller and a scheduled task trigger. The message queue listener focuses on real-time monitoring and processing of the message queue, and the controller focuses on responding to HTTP requests and providing an interface for external interaction. Together with the scheduled task trigger, they all rely on the business logic layer and are responsible for coordinating various operation logics. At the same time, they also establish connections with other key modules such as the database access layer, the gateway interface layer, and other application services of the cloud platform.

4. A method for cloud platform file storage service according to claim 3, characterized in that: For standard file storage types, the storage backend relies on block devices, and the objects of operation are all data blocks; First, a data snapshot is generated, and then a new temporary block device is cloned based on the snapshot. Then, the temporary block device is mounted to a specially designed backup unit.

5. A method for cloud platform file storage service according to claim 4, characterized in that: The backup unit reads the data of each block, converts it into independent objects, and uploads them one by one to the object storage service. The backup unit also records the backup metadata, that is, the mapping table of block ID and corresponding data block MD5 checksum value, which provides the basis for incremental backup.

6. A method for cloud platform file storage service according to claim 5, characterized in that: When performing incremental backup, the backup unit compares the MD5 value of the current block data with the MD5 value of the previous backup recorded in the backup metadata. If the two values ​​are consistent, it indicates that the data has not changed and there is no need to back up again; On the contrary, if a difference is found, or the block ID does not exist in the metadata, the block is backed up; During recovery, the target block device will be mounted to the backup unit. The backup unit will retrieve the corresponding data blocks from the object storage based on the backup metadata and write them back to the corresponding block locations one by one.

7. A method for cloud platform file storage service according to claim 6, characterized in that: For low-frequency file storage types, the storage backend is the CephFS distributed file system, and the granularity of backup and recovery is files; When implementing backup, the same principle of creating snapshots first is followed. The CephFS snapshot mechanism provides a read-only view containing all file metadata. First, the files are grouped by performing a modulo operation on the file inode. Each group is treated as a separate task and executed by an independent thread. The files are packaged in tar format. The backup metadata records the correspondence between the file name and the modification time. During the incremental backup phase, the backup unit compares the modification timestamp of the current file with the timestamp recorded in the backup metadata. The backup operation of the file is triggered only when there is a difference between the two. During recovery, CephFS is mounted to the backup unit, and the backup unit writes the data in the object storage to CephFS according to the backup metadata guidance.

8. A method for cloud platform file storage service according to claim 7, characterized in that: Parallel file storage type. The storage backend is a black box that directly reads the snapshot content. In order to obtain the snapshot data, an indirect strategy is adopted to create a new temporary file system through the snapshot. Subsequently, this temporary file system is mounted to the backup unit, and the parallel file storage is backed up at the file level.

9. A cloud platform file storage service device, characterized in that: include: at least one memory and at least one processor; The at least one memory is used to store a machine-readable program; The at least one processor is configured to call the machine-readable program to execute the method according to any one of claims 1 to 8.

Images