An implementation method of distributed thick-provisioned storage volume based on capacity management

Abstract

The application discloses an implementation method of a distributed thick-provisioned storage volume based on capacity management, which comprises the following steps: when a resource pool is created, a capacity management module calculates the total capacity of available hard disks and records the total capacity in a database; when a storage volume or a snapshot is created, the specification capacity, the maximum occupied capacity of the snapshot calculated based on the written data state and the available capacity are checked respectively, and if the available capacity is sufficient, the database is updated and then the creation is performed; a bitmap structure is used to record the data unit state, and the bitmap structure supports the snapshot capacity calculation and the capacity update during the deletion; when the volume or the snapshot is deleted, the capacity is released, the capacity of the subsequent snapshot is recalculated according to the snapshot chain inheritance relationship and the available capacity is updated. The application can effectively solve the problem of low creation efficiency caused by the full-volume zero writing required by the traditional thick-provisioned storage volume. The method of the application is free of physical zero writing, can guarantee the exclusivity of the storage space, realize the quick supply of the thick-provisioned storage volume, and improve the system efficiency and consistency.

Description

TECHNICAL FIELD

[0001] The present application relates to distributed storage technology, in particular to a method for implementing a distributed thick-provisioned storage volume based on capacity management. BACKGROUND

[0002] A distributed storage system integrates the resources of multiple storage servers to form a unified management storage resource pool. The system is usually composed of three core modules: a front-end business interface process, a distributed management process, and a back-end storage process. It uses distributed data consistency algorithms to calculate and store business data to different back-end storage nodes. The business model and service architecture are as shown in Figure 1 Data stored in the cluster is backed up through redundancy strategies such as replication or erasure coding, ensuring that data can be recovered from backup nodes when a single storage node fails, thus ensuring service reliability and data persistence.

[0003] As an important form of distributed storage, distributed block storage (Distributed Block Storage) stores data on multiple independent nodes and integrates these resources into a unified, highly available, and scalable block storage pool through a network. This technology has the advantages of high availability, easy scalability, high performance, flexibility, and cost-effectiveness, and is widely used in cloud computing, databases, big data analysis, virtual desktop infrastructure, and container storage scenarios. It supports two storage allocation modes: thick provisioning and thin provisioning. Thick provisioning mode pre-allocates all capacities when creating a volume, which simplifies management but may lead to low storage resource utilization due to insufficient actual usage.

[0004] Block storage snapshot (Block Storage Snapshot) is an instant and recoverable copy of the data state of a block storage volume (LUN) at a specific point in time. It is not a complete data copy in the traditional sense, but a more efficient and space-saving "point-in-time view". Point-in-time copy (Point-in-Time Copy): The snapshot captures the state of all data blocks of the storage volume at the moment the snapshot is created. Not a full copy (Not a Full Copy): A snapshot is usually not a complete copy of the entire volume's data. If a complete copy is made every time, it will be very time-consuming and occupy a large amount of storage space. Instead, it uses technologies such as copy-on-write (Copy-on-Write, COW) or redirect-on-write (Redirect-on-Write, ROW).

[0005] In the prior art, as recorded in the patent document with publication number CN112631951A, a storage space allocation method is provided. When creating a thick configuration storage volume, the method directly allocates space in the storage engine layer of the backend storage process by the block service interface layer. However, when dealing with requests such as VMware thick configuration virtual machine disk, the method still needs to perform a complete data zero write operation to occupy storage resources, which is time-consuming and consumes a large amount of system input and output resources, affecting the creation efficiency of the storage volume.

[0006] It should be noted that the information disclosed in the above background section is only for understanding the background of the present application, and therefore can include information that does not constitute prior art known to those of ordinary skill in the art. SUMMARY

[0007] The main purpose of the present application is to overcome the defects existing in the background art, and to provide an implementation method of distributed thick backup storage volume based on capacity management.

[0008] To achieve the above-mentioned purpose, the present application adopts the following technical solutions:

[0009] An implementation method of distributed thick backup storage volume based on capacity management, comprising:

[0010] In the resource pool creation process of the distributed storage system, the capacity management module calculates the total available hard disk capacity of the resource pool, and records the capacity information in the database;

[0011] When creating a storage volume in the resource pool, the specification capacity of the storage volume is obtained and compared with the current available capacity of the resource pool. If the capacity is sufficient, the specification capacity is deducted from the resource pool and the database is updated, and then the storage volume is created, otherwise the creation is refused;

[0012] When creating a snapshot for a storage volume, the maximum occupied capacity of the snapshot is calculated based on the written data state of the storage volume, and compared with the current available capacity of the resource pool. If the capacity is sufficient, the maximum occupied capacity is deducted from the resource pool and the database is updated, and then the snapshot is created, otherwise the creation is refused;

[0013] The state of each data unit in the storage volume is recorded using a bitmap structure. The bitmap state is used to determine the maximum occupied capacity of the snapshot when creating the snapshot, and to update the capacity occupation information when deleting the snapshot;

[0014] When deleting a storage volume, its occupied capacity is released and the available capacity of the resource pool is updated; when deleting a snapshot, the capacity occupation of the subsequent snapshot is recalculated according to the inheritance relationship of the snapshot chain, and the available capacity of the resource pool is updated.

[0015] The computer program product comprises a computer program which, when executed by a processor, implements the implementation method of the distributed thick-provisioned storage volume based on capacity management.

[0016] The present application has the following advantages:

[0017] The present application provides an implementation method of a distributed thick-provisioned storage volume based on capacity management, and the core is to implement a method for quickly creating a thick-provisioned storage volume. By introducing a centralized capacity resource management mechanism, the system can pre-allocate and reserve capacity when creating a thick-provisioned volume, thereby greatly shortening the creation time of the volume while ensuring the same space exclusive effect as the traditional thick-provisioned volume. This method completely avoids the time-consuming full-disk zero-writing operation that must be performed in the traditional implementation, significantly improving the efficiency of storage supply.

[0018] Specifically, the core of the method of the present application is to establish a global, resource pool-based capacity view and management mechanism. To achieve this, the system can use the cluster management function inherent in the distributed storage system, which can provide a global view of the cluster. On this basis, the capacity management module can directly count and obtain the complete hard disk capacity information at the resource pool creation stage. Based on this accurate and global capacity view, the system establishes a unified capacity management strategy: when creating each storage volume or storage volume snapshot, the corresponding quota is deducted from the total capacity of the resource pool in real time; conversely, when deleting a storage volume or snapshot, the corresponding capacity is released back to the resource pool. This mechanism not only makes the allocation and recovery of capacity clear and controllable, but also provides a reliable and efficient calculation and management method for the creation, deletion, and capacity inheritance of snapshot chains.

[0019] Further, by saving the capacity information, the occupation records of storage volumes and snapshots in the database, the system obtains important management advantages. The most prominent benefit is that when performing creation or deletion operations, there is no need to traverse and calculate the capacity occupation of all existing objects in the resource pool, but only the capacity deduction, release, or inheritance relationship update of the target volume or snapshot of the current operation is needed. This design greatly simplifies the complexity of capacity calculation, reduces the computational overhead of the system, and at the same time, through database transactions, ensures the consistency of capacity data in concurrent scenarios, making the management of storage resources more efficient and accurate.

[0020] Other advantages of the embodiments of the present application will be further described below. BRIEF DESCRIPTION OF DRAWINGS

[0021] Figure 1 is a distributed storage system business model and service architecture diagram.

[0022] Figure 2is a resource pool capacity control and storage volume and snapshot creation process logic diagram of an embodiment of the present application.

[0023] Figure 3 is a storage volume or snapshot creation time capacity judgment logic diagram of an embodiment of the present application.

[0024] Figure 4 is a data table structure design diagram for recording capacity information of an embodiment of the present application.

[0025] Figure 5 is a storage volume data unit bitmap state machine switching diagram of an embodiment of the present application.

[0026] Figure 6 is a storage volume and snapshot data unit bitmap state change example diagram of an embodiment of the present application.

[0027] Figure 7 is a snapshot maximum occupied capacity calculation principle schematic diagram of an embodiment of the present application.

[0028] Figure 8 is a storage volume and snapshot deletion time capacity update process logic diagram of an embodiment of the present application. DETAILED DESCRIPTION

[0029] The embodiments of the present application will be described in detail below. It should be emphasized that the following description is only exemplary and is not intended to limit the scope of the present application and its applications.

[0030] In addition, the terms "first", "second" are only for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features. Therefore, the features defined with "first", "second" can explicitly or implicitly include one or more of the features. In the description of the embodiments of the present application, the meaning of "a plurality of" is two or more, unless otherwise explicitly specified.

[0031] The present application aims to solve the problem of low creation efficiency of traditional thick-provisioned storage volumes due to the need to perform full-disk write-zero operations, and proposes an implementation method based on global capacity view and pre-allocation management. This method reserves and checks capacity at the resource pool level, and introduces a bitmap structure to fine-grainedly manage snapshot capacity, replacing the time-consuming physical write-zero process. Finally, while ensuring the exclusivity of storage space (thick-provisioned effect), fast and lightweight storage volume supply is achieved.

[0032] Referring to Figures 1 to 8 , the embodiment of the present application provides an implementation method of distributed thick-provisioned storage volume based on capacity management, comprising:

[0033] In the resource pool creation process of the distributed storage system, the available hard disk capacity of the resource pool is calculated by the capacity management module, and the capacity information is recorded in the database;

[0034] When creating a storage volume in the resource pool, the specification capacity of the storage volume is obtained and compared with the current available capacity of the resource pool. If the capacity is sufficient, the specification capacity is deducted from the resource pool and the database is updated, and then the storage volume is created. Otherwise, the creation is refused.

[0035] When creating a snapshot for a storage volume, the maximum occupied capacity of the snapshot is calculated based on the written data state of the storage volume, and compared with the current available capacity of the resource pool. If the capacity is sufficient, the maximum occupied capacity is deducted from the resource pool and the database is updated, and then the snapshot is created. Otherwise, the creation is refused.

[0036] The state of each data unit in the storage volume is recorded using a bitmap structure. The bitmap state is used to determine the maximum occupied capacity of the snapshot when creating the snapshot, and to update the capacity occupation information when deleting the snapshot.

[0037] When deleting a storage volume, its occupied capacity is released and the available capacity of the resource pool is updated. When deleting a snapshot, the capacity occupation of subsequent snapshots is recalculated according to the inheritance relationship of the snapshot chain, and the available capacity of the resource pool is updated.

[0038] It is worth noting that the traditional thick backup storage volume creation needs to perform full-disk zero writing operation, which is time-consuming and resource-consuming. The present application controls the capacity at the resource pool level and uses the bitmap structure, which ensures the exclusivity of storage space and eliminates the physical zero writing process, achieving fast supply.

[0039] In some embodiments, referring to Figure 2 and Figure 3 , the specific process of creating a snapshot includes: receiving a snapshot creation request and obtaining the bitmap structure of the target storage volume to determine its written data state; based on the bitmap structure and the snapshot chain relationship, calculating the maximum occupied capacity of the snapshot; querying the current available capacity of the resource pool from the database; determining whether the maximum occupied capacity is not greater than the current available capacity; if yes, performing the following operations in the database transaction: subtracting the maximum occupied capacity from the current available capacity for deduction and updating, and creating the snapshot; if no, refusing to create and returning a capacity insufficient indication.

[0040] In some embodiments, the recording and updating of the capacity information is implemented through the database, and the query and update operations of the database are encapsulated in the database transaction to avoid capacity counting abnormalities in concurrent scenarios. The data table structure for recording capacity information is as follows: Figure 4As shown, the bitmap structure includes a capacity count table and a storage volume and snapshot information table, and the data consistency is ensured by the primary key and foreign key association.

[0041] In some embodiments, referring to Figure 5 , the bitmap structure uses two bits to represent the state of each data unit, wherein: the first state represents no data written, the second state represents data written, the third state represents to be deleted, and the fourth state represents protected by snapshot; the state transition is driven by data operation, specifically including: performing write data operation on the data unit in the first state to convert it to the second state; converting the bitmap of the data unit in the second state in the source storage volume to the fourth state when creating a snapshot; performing deletion operation on the data unit in any state to mark its bitmap as the third state; and resetting the bitmap of the data unit in the third state to the first state after completing data recovery.

[0042] In some embodiments, referring to Figure 7 , the maximum occupied capacity of the snapshot includes: obtaining the bitmap state of the storage volume corresponding to the current snapshot; comparing the bitmap of the current snapshot with the bitmap of the previous snapshot; determining the number of incremental data units of the current snapshot relative to the previous snapshot according to the difference in bitmap state; and calculating the maximum occupied capacity of the snapshot based on the number of incremental data units and the redundancy strategy of the storage volume. An example of the change in the bitmap state of the data unit of the storage volume and snapshot is shown in FIG. 6.

[0043] In some embodiments, the rule of comparing the bitmap of the current snapshot with the bitmap of the previous snapshot to determine the capacity occupation specifically includes: comparing the bitmap state of the current snapshot with the bitmap state of the previous snapshot data unit by data unit; if the bitmap state of the current snapshot represents that data exists, and the corresponding bitmap state of the previous snapshot does not represent that data exists or is protected by snapshot, it is determined that the current snapshot generates capacity occupation for the data unit; if the bitmap state of the current snapshot represents that it is protected by snapshot, and the corresponding bitmap state of the previous snapshot represents that data exists, it is determined that the current snapshot does not generate capacity occupation for the data unit; and based on the above determination rule, the capacities of all data units generating capacity occupation are accumulated.

[0044] In some embodiments, referring to Figure 8 , the updating of the available capacity of the resource pool when deleting the snapshot includes: determining the next snapshot of the deleted snapshot in the snapshot chain; merging the interval representing data protection in the bitmap structure of the deleted snapshot into the bitmap structure of the next snapshot, and discarding the overlapping interval; recalculating the capacity occupation of the next snapshot based on the updated bitmap structure; and updating the available capacity of the resource pool and the related capacity record in the database according to the recalculated result.

[0045] In some embodiments, when calculating the capacity occupation of a storage volume or snapshot, the logical capacity is converted into actual physical data occupation according to the redundancy strategy of the storage volume, so as to accurately reflect the capacity consumption in the resource pool.

[0046] In some embodiments, the specific process of creating a storage volume includes: receiving a request for creating a storage volume and a specification capacity; querying the current available capacity of the resource pool from the database; determining whether the specification capacity is not greater than the current available capacity; if yes, performing in a database transaction: subtracting the specification capacity from the current available capacity for deduction update, and creating the storage volume; if no, rejecting the creation and returning a capacity shortage indication.

[0047] The present application effectively solves the problem of low efficiency caused by the need to perform full-disk zero writing operation in the creation process of traditional thick-provisioned storage volumes. The technical advantages mainly include: first, the system relies on cluster global information to realize real-time reservation, deduction and release of capacity at the resource pool level, thereby guaranteeing the exclusive effect of thick-provisioned space while completely eliminating the time-consuming physical zero writing process, significantly improving the volume creation speed; second, by designing a bitmap structure for each storage volume to record the data unit state, and combining the incremental inheritance relationship between snapshot chains, the maximum occupied capacity of the snapshot can be quickly and accurately calculated, ensuring the efficiency and reliability of the snapshot operation; third, with the centralized storage of capacity information and object occupation records in the database, the creation and deletion of any volume or snapshot only needs to update the capacity data of the current object and its associated snapshots, without the need for global traversal calculation, greatly simplifying the complexity of capacity management and enhancing the maintainability and data consistency of the system in a concurrent scenario. Ultimately, this method not only realizes the rapid supply of thick-provisioned volumes, but also provides fine-grained capacity management of the snapshot life cycle, overall improving the resource utilization efficiency and management flexibility of the distributed block storage system.

[0048] The implementation process and advantages of the specific embodiments of the present application are further described below.

[0049] The present application realizes global resource view management of the distributed storage cluster through the cluster management module. Based on this global perspective, the system can directly obtain and count the total available hard disk capacity when creating a resource pool, and establish a centralized capacity management mechanism based on this. Specifically, when creating a storage volume or a snapshot of a storage volume, the system will pre-deduct the corresponding capacity quota from the total capacity of the resource pool; conversely, when deleting a storage volume or a snapshot, the occupied capacity is released back to the resource pool. To ensure the accuracy and reliability of capacity calculation during snapshot creation and deletion, a corresponding fast calculation method also needs to be designed. The overall scheme can be summarized as follows:

[0050] Firstly, in the resource pool creation stage of the distributed storage system, the total available hard disk capacity of the resource pool is calculated and recorded. Secondly, when a storage volume is created in the resource pool, the corresponding deduction is made according to the application capacity; if the remaining capacity of the resource pool is insufficient, the creation is refused. Thirdly, when a snapshot is created for a storage volume, according to the snapshot protection principle, the capacity occupied by the data currently written into the storage volume is calculated and used as the maximum possible occupied capacity of the snapshot to make a reservation deduction; if the capacity is insufficient, the snapshot is not created. In order to support fast and accurate snapshot capacity calculation, the system maintains a bitmap structure for each storage volume to record the real-time state of each data block (Block), thereby significantly improving the capacity determination efficiency in the snapshot creation process.

[0051] In some embodiments of the application, a capacity management based thick backup implementation method for a distributed system is provided, which is mainly focused on centralized capacity management of storage resources. The method mainly includes the following steps: firstly, the total available capacity of the resource pool is obtained by aggregating and calculating the capacity of all hard disks in the resource pool, and the capacity information is recorded persistently. Secondly, when a new storage volume needs to be created or a snapshot is created for a storage volume, the system calculates the capacity size occupied by the operation, and compares the predicted occupied capacity with the available capacity information recorded in the resource pool, which is used as the access judgment basis for whether to allow the creation operation (if the capacity is insufficient, the corresponding creation operation is refused). Further, in the above embodiment, the capacity information and the capacity occupation information of the storage volume and the snapshot are preferably stored and managed by a database to ensure data consistency and queryability. At the same time, in order to quickly and accurately determine the maximum capacity that may be occupied by the snapshot when creating the snapshot, the system introduces a bitmap structure for each storage volume, which records the real-time state of the data unit, thereby efficiently calculating the upper limit of the space reserved for the snapshot.

[0052] The following will be described in detail through specific embodiments, how to implement the core process of capacity management in the resource pool management and the storage volume and snapshot creation process.

[0053] Resource pool capacity control and storage volume and snapshot creation process analysis:

[0054] The process mainly includes the following steps, and the logical relationship is as shown in Figure 2

[0055] ​First, when creating a resource pool, the system obtains the total capacity information of the resource pool and records it in the designated data table. When receiving a request to create a storage volume, the process enters the storage volume creation branch: the system obtains the specifications (i.e., capacity size) of the storage volume and calculates its maximum possible space occupancy. Then, it is determined whether the remaining capacity is still sufficient after adding the storage volume to the resource pool. If the capacity is sufficient, the storage volume's specification information is recorded to the database, and the creation of the storage volume is completed, returning a successful response; if the capacity is insufficient, an error message of insufficient resources is returned to the requester.

[0056] When the request is to create a snapshot for a certain storage volume, the process enters the snapshot creation branch: the system first calculates the maximum capacity that the snapshot can occupy (the specific calculation method will be described later). Then, it is determined whether the remaining capacity of the resource pool meets the requirements after adding the snapshot capacity. If it meets the requirements, the expected capacity information of the snapshot is recorded to the database, and the snapshot is successfully created; if it does not meet the requirements, the snapshot creation operation is cancelled, and an error message of insufficient resources is returned.

[0057] In this process, there are several key points worth noting: first, all queries and updates of capacity information must be encapsulated in database transactions to ensure the accuracy of capacity counting in concurrent access. Second, when determining the capacity occupancy of a storage volume or snapshot, its redundancy strategy (such as replication or erasure coding) needs to be considered to accurately reflect its actual consumption on physical storage. Third, the capacity occupancy between snapshots has a chain inheritance relationship, and the associated influence of upper and lower snapshots must be considered when calculating and updating.

[0058] The logic of the system's capacity judgment is as follows (see Figure 3 ): obtain the capacity A of the current storage volume to be created, query the total allocated capacity B and the total available capacity C of the resource pool; if A + B <= C, it is determined that the capacity is sufficient.

[0059] The data representation example (e.g., field design) for recording capacity information can be seen in Figure 4 . The core advantage of using a database for centralized recording is that when creating or deleting a single storage volume or snapshot, there is no need to globally traverse and calculate the capacity occupancy of all objects. Only the capacity deduction or release of the current operation object needs to be performed, and the data consistency is guaranteed through transactions, which greatly improves the management efficiency. The principles of capacity inheritance and update involved in deleting snapshots will be further explained in the subsequent sections.

[0060] Example of storage volume snapshot maximum capacity calculation method:

[0061] To calculate the maximum capacity reserved for snapshots accurately, the system introduces a bitmap structure for each storage volume to record the state of each data unit (ITEM). The state of each data unit is represented by two bits, which are defined as follows:

[0062] "00" means that the unit has never been written data;

[0063] "01" means that the unit currently has valid data;

[0064] "10" means that the unit is in a state to be deleted;

[0065] "11" means that the unit is protected by the snapshot.

[0066] The state machine switching of each data state is shown in Figure 5 .

[0067] Referring to Figure 6 , the state change process can be clearly shown by a set of actual operation sequences:

[0068] Assume that at the initial time T0, the bitmap state sequence of a certain storage volume is: 01-00-01-00-00-01-01-00. At this time, if a snapshot is created for the volume, all units in the source storage volume whose state is "01" (there is data) will have their bitmaps converted to "11" (protected by the snapshot). Therefore, after the snapshot is created, the state sequence of the source volume becomes: 11-00-11-00-00-11-11-00, and the newly created snapshot inherits and records the data state of the source volume at the snapshot creation time, and its bitmap sequence is: 01-00-01-00-00-01-01-00.

[0069] If new data is written to the volume at time T1, the bitmap state may be updated to: 01-01-11-01-00-01-11-00. At this time, if a snapshot is created again, all "01" state units in the source volume will be marked as "11" again, and the new snapshot will record the "01" and "11" state distribution at this time.

[0070] Based on the above bitmap mechanism, the logic for calculating the maximum protection capacity of the snapshot is as follows, and the principle diagram can be referred to Figure 7 .

[0071] Taking the first snapshot (snap1) as an example, its bitmap state is: 01-00-01-00-00-01-01-00. Among them, the number of units in the state "01" is 4. Since there is no upper snapshot before snap1, the maximum capacity it occupies is the sum of the capacities of these 4 data units.

[0072] For the second snapshot (snap2), its capacity calculation needs to refer to the bitmap of its upper snapshot (snap1), and make a cell-by-cell judgment according to the following rules:

[0073] Suppose the bitmap value of a cell in the upper snapshot is a, and the bitmap value of the corresponding cell in the current snapshot is b, then:

[0074] A. If b = 00, then the cell has no capacity occupation in the current snapshot.

[0075] B. If b = 01, then the cell may have capacity occupation in the current snapshot, and the maximum occupation is one data cell capacity.

[0076] C. If a and b are not equal, but a = 01 and b = 11, it means that the data of the cell is protected by the upper snapshot (snap1), and the current snapshot (snap2) has no new capacity occupation on this cell.

[0077] D. Except the above cases, if a and b are not equal, it means that the state of the current snapshot on the cell is different from that of the upper snapshot, and the current snapshot may need to occupy capacity for this cell, and the maximum occupation is also one data cell capacity.

[0078] According to this rule, the capacity occupation of snap2 can be calculated from multiple data cells in the bitmap that meet certain conditions, and the total capacity is the sum of the capacities of these target cells (see the cells marked in red in Figure 7 ).

[0079] Example of storage volume snapshot deletion capacity update method:

[0080] Further, since the essence of each snapshot in the snapshot chain is the incremental change of data, therefore, when deleting a snapshot, its capacity release and inheritance logic needs to be based on the incremental relationship between snapshots. The following is illustrated by a specific example.

[0081] Suppose at T0, a data block consisting of 100 repeated characters 'x' is written to the logical address interval [0-100] of a storage volume, and a data block consisting of 50 repeated characters 'y' is written to the interval [400-450]. At T1, a snapshot snap1 is created based on the volume state, and the data interval protected by snap1 is the 'x' data in [0-100] and the 'y' data in [400-450]. At T2, a new data block consisting of 100 repeated characters 'z' is written to the interval [50-150] of the volume. Then at T3, a snapshot snap2 is created again. At this time, because the new data covers part of the original interval, the data state protected by snap2 becomes: the interval [0-50] is the original 'x' data, the interval [50-150] is the newly added 'z' data, and the interval [400-450] is still the original 'y' data.

[0082] From the perspective of the increment of data changes, the data interval that snap2 really newly protects is only [50-150]. However, it must ensure that the data in snap1 that is not covered (that is, the intervals [0-50] and [400-450]) is continued to be retained. Therefore, when snap1 is deleted, the interval [50-100] that it originally protects (this part overlaps with the protection interval of snap2) will be released; and the data interval that is unique to snap1 and still needs to be protected (that is, [0-50] and [400-450]) must be inherited by snap2.

[0083] The above logic shows that after a snapshot is deleted, the capacity it occupies is not simply released back to the resource pool in its entirety, but the data protection interval unique to it is transferred to the next snapshot, and the part overlapping with the next snapshot is discarded. Accordingly, when a snapshot is deleted, the system needs to perform the following operations to maintain the accuracy of capacity management (see Figure 8 ): first, find the next snapshot (snap_next) of the snapshot in the chain; second, merge the protection interval in the bitmap structure of the snapshot into the bitmap structure of snap_next, while removing the overlapping part; then, based on the updated bitmap, recalculate the maximum protection capacity of snap_next; finally, update the available capacity of the resource pool and the related records in the database according to the recalculation result. This process ensures that during the dynamic changes of the snapshot chain, the capacity statistics always remain consistent with the actual needs of data protection.

[0084] In summary, the application can efficiently pre-allocate capacity for thick-provisioned storage volumes by the implementation method based on capacity management. The method obtains a global capacity view of the distributed cluster, and establishes a centralized capacity reservation, deduction and release mechanism based on the global capacity view, and combines a bitmap structure to realize fast calculation and inheritance management of snapshot capacity. The method ensures that the thick-provisioning technical effect (i.e. pre-occupying storage space) is achieved, and significantly improves the creation speed of the storage volume. The core advantage is that the capacity reservation and verification mechanism at the resource pool level replaces the full-volume zero writing operation that must be performed on the physical disk in the traditional thick-provisioning process, thereby avoiding long I / O waiting and resource consumption caused thereby, and realizing fast and lightweight thick-provisioning volume supply.

[0085] The embodiment of the application further provides a storage medium for storing a computer program, which is executed to perform at least the method described above.

[0086] The embodiment of the application further provides a control device, which comprises a processor and a storage medium for storing a computer program; wherein the processor is configured to execute the computer program to perform at least the method described above.

[0087] The embodiment of the application further provides a processor, which executes a computer program to perform at least the method described above.

[0088] The storage medium can be implemented by any type of non-volatile storage device, or a combination thereof. The non-volatile storage device can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a ferromagnetic random access memory (FRAM), a flash memory, a magnetic surface memory, an optical disc or a compact disc read-only memory (CD-ROM). The magnetic surface memory can be a disk memory or a tape memory. The storage medium described in the embodiment of the application is intended to include but is not limited to these and any other suitable type of memory.

[0089] In several embodiments provided by the present application, it should be understood that the disclosed system and method can be implemented in other manners. The described device embodiments are merely illustrative. For example, the division of the units is merely logical function division, and there can be other division manners in actual implementation. For example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed coupling or direct coupling or communication connection between the components can be indirect coupling or communication connection through some interfaces, devices or units, and can be electrical, mechanical or other forms.

[0090] The units described as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, that is, can be located in one place or distributed on a plurality of network units; and some or all of the units can be selected according to actual needs to achieve the purpose of the embodiment.

[0091] In addition, each functional unit in each embodiment of the present application can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; and the integrated unit can be implemented in the form of hardware or hardware plus software functional unit.

[0092] Those skilled in the art can understand that all or part of the steps of the above-mentioned method embodiments can be completed by program instruction related hardware, and the foregoing program can be stored in a computer readable storage medium, and the program is executed to perform the steps of the above-mentioned method embodiments; and the foregoing storage medium includes mobile storage equipment, read-only memory (ROM), random access memory (RAM), magnetic disk or optical disk and various storage medium capable of storing program codes.

[0093] Alternatively, the integrated unit of the present application, if implemented in the form of a software function module and sold or used as an independent product, can also be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the embodiments of the present application can be embodied in the form of a software product, and the computer software product is stored in a storage medium, and includes several instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute all or part of the methods described in the embodiments of the present application. The foregoing storage medium includes mobile storage equipment, ROM, RAM, magnetic disk or optical disk and various storage medium capable of storing program codes.

[0094] The methods disclosed in the several method embodiments of the present application can be combined arbitrarily without conflict to obtain new method embodiments.

[0095] The features disclosed in the several product embodiments of the present application can be combined arbitrarily without conflict to obtain new product embodiments.

[0096] The features disclosed in the several method or device embodiments of the present application can be combined arbitrarily without conflict to obtain new method embodiments or device embodiments.

[0097] The above is a further detailed description of the present application in combination with specific preferred embodiments, and the specific implementation of the present application should not be limited to these descriptions. For those skilled in the art to which the present application belongs, without departing from the concept of the present application, a number of equivalent substitutions or obvious modifications can be made, and the performance or use is the same, which should be regarded as belonging to the protection scope of the present application.

Claims

1. A method for implementing a distributed thick provisioned storage volume based on capacity management, characterized in that, The application relates to a method for managing the capacity of a resource pool in a distributed storage system. In the resource pool creation process of the distributed storage system, a capacity management module is used to calculate the total available hard disk capacity of the resource pool and record the capacity information in a database; When a storage volume is created in the resource pool, the specification capacity of the storage volume is obtained and compared with the current available capacity of the resource pool; if the capacity is sufficient, the specification capacity is deducted from the resource pool and the database is updated, and then the storage volume is created; otherwise, the creation is rejected; When a snapshot is created for the storage volume, the maximum occupied capacity of the snapshot is calculated based on the written data state of the storage volume, and compared with the current available capacity of the resource pool; if the capacity is sufficient, the maximum occupied capacity is deducted from the resource pool and the database is updated, and then the snapshot is created; otherwise, the creation is rejected; A bitmap structure is used to record the state of each data unit in the storage volume; the bitmap state is used to determine the maximum occupied capacity of the snapshot when the snapshot is created, and to update the capacity occupation information when the snapshot is deleted; the state of each data unit is as follows: the first state represents no data writing, the second state represents data writing, the third state represents deletion, and the fourth state represents snapshot protection; the state conversion is driven by data operation, and specifically includes: performing a write data operation on the data unit in the first state to convert it into the second state; converting the bitmap of the data unit in the second state in the source storage volume into the fourth state when the snapshot is created; performing a deletion operation on the data unit in any state to mark the bitmap of the data unit into the third state; and resetting the bitmap of the data unit in the third state into the first state after data recovery is completed; When the storage volume is deleted, the occupied capacity is released and the available capacity of the resource pool is updated; when the snapshot is deleted, the capacity occupation of the subsequent snapshot is recalculated according to the inheritance relationship of the snapshot chain, and the available capacity of the resource pool is updated.

2. The method of claim 1, wherein, The recording and updating of the capacity information are realized through a database, and the query and updating operations of the database are encapsulated in a database transaction to avoid capacity counting abnormities in a concurrent scenario.

3. The method of claim 1, wherein, The bitmap structure uses two bits to represent the state of each data unit.

4. The method of claim 1, wherein, The calculation of the maximum occupied capacity of the snapshot includes: obtaining the bitmap state of the storage volume corresponding to the current snapshot; comparing the bitmap of the current snapshot with the bitmap of the previous snapshot; determining the number of incremental data units of the current snapshot relative to the previous snapshot according to the difference between the bitmap states; calculating the maximum occupied capacity of the snapshot based on the number of incremental data units and the redundancy strategy of the storage volume.

5. The method of claim 4, wherein, The rules for comparing the bitmap of the current snapshot with the bitmap of the previous snapshot to determine the capacity occupation include: comparing the bitmap state of the current snapshot with the bitmap state of the previous snapshot data unit by data unit; if the bitmap state of the current snapshot represents that data exists and the corresponding bitmap state of the previous snapshot does not represent that data exists or is protected by the snapshot, it is determined that the current snapshot produces capacity occupation for the data unit; if the bitmap state of the current snapshot represents that the data is protected by the snapshot and the corresponding bitmap state of the previous snapshot represents that data exists, it is determined that the current snapshot does not produce capacity occupation for the data unit; Based on the above determination rule, the capacity of all data units that generate capacity occupation is accumulated.

6. The method of claim 1, wherein, Updating the available capacity of the resource pool when deleting a snapshot includes: Determining the next snapshot of the deleted snapshot in the snapshot chain; Merging the interval representing data protection in the bitmap structure of the deleted snapshot into the bitmap structure of the next snapshot, and discarding the overlapping interval; Based on the updated bitmap structure, recalculating the capacity occupation of the next snapshot; According to the recalculated result, updating the available capacity of the resource pool and the related capacity record in the database.

7. The method of claim 1, wherein, When calculating the capacity occupation of a storage volume or a snapshot, the logical capacity is converted into actual physical data occupation according to the redundancy strategy of the storage volume, so as to accurately reflect the capacity consumption in the resource pool.

8. The method of claim 1, wherein, The specific process of creating a storage volume includes: Receiving a request for creating a storage volume and a specification capacity; Querying the current available capacity of the resource pool from the database; Determining whether the specification capacity is not greater than the current available capacity; If yes, performing the following in a database transaction: subtracting the specification capacity from the current available capacity for deduction update, and creating the storage volume; If no, refusing to create and returning a capacity insufficient indication.

9. The method of claim 1, wherein, The specific process of creating a snapshot includes: Receiving a request for creating a snapshot and obtaining the bitmap structure of the target storage volume to determine its written data state; Based on the bitmap structure and the snapshot chain relationship, calculating the maximum occupation capacity of the snapshot; Querying the current available capacity of the resource pool from the database; Determining whether the maximum occupation capacity is not greater than the current available capacity; If yes, performing the following in a database transaction: subtracting the maximum occupation capacity from the current available capacity for deduction update, and creating the snapshot; If no, refusing to create and returning a capacity insufficient indication.

10. A computer program product comprising a computer program, characterized in that, The computer program, when executed by a processor, implements the implementation method of the distributed thick backup storage volume based on capacity management according to any one of claims 1 to 9.

Images