Distributed storage systems and their capacity statistics methods

Abstract

This invention discloses a distributed storage system and its usage capacity statistics method, relating to the field of storage technology. The method includes determining the logical scope of newly added data and the logical scope of deleted data for an object before and after a business operation request is executed during the process of updating object metadata. It also involves determining the overlap between the object's snapshot duplicate reference range and the deleted data logical scope, subtracting the overlap range from the deleted data logical scope, whereby the snapshot duplicate reference range represents the shared data range between the object and associated snapshot objects. The subtraction result and the newly added data logical scope are used as usage capacity change data caused by the business operation to update the usage capacity statistics of the distributed storage system. This invention solves the problem that related technologies cannot accurately determine the actual exclusive capacity of user data, enabling precise statistics on the usage capacity of user data written to the front end, thus improving the reliability and availability of the distributed storage system.

Description

Technical Field

[0001] This invention relates to the field of storage technology, and in particular to a distributed storage system and a method for calculating its usage capacity. Background Technology

[0002] Distributed storage systems provide high capacity, high availability, and elastic scalability by distributing data across multiple physical nodes. The usage capacity impacts resource scheduling and system health monitoring. Related technologies rely on block reference counters to calculate the usage capacity of user data written to the front end. This method cannot accurately identify partial sharing within data blocks, leading to redundancy in the statistical results when small-scale data modifications occur, failing to accurately reflect the actual exclusive capacity of user data. Furthermore, maintaining independent counting metadata for each data block results in significant additional storage and computational overhead.

[0003] It should be noted that the information disclosed in the background section above is only used to enhance the understanding of the background of this application, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0004] This invention provides a distributed storage system and a method for calculating its usage capacity, which can accurately calculate the usage capacity of user data written to the front end, effectively improving the reliability and availability of the distributed storage system.

[0005] To solve the above-mentioned technical problems, the present invention provides the following technical solution: This invention provides a method for calculating the usage capacity of a distributed storage system, comprising: In response to the business operation request of the target object, during the process of updating the target object's metadata, determine the scope of newly added data and the scope of deleted data logic of the target object before and after the execution of the business operation request; Obtain the snapshot duplicate reference range corresponding to the target object, and determine the overlapping data range between the data deletion logical range and the snapshot duplicate reference range; the snapshot duplicate reference range is used to characterize the data range shared between the target object and at least one associated snapshot object; Based on the logical range of deleted data and the logical range of added data after deducting overlapping data ranges, determine the data on the changes in the usage capacity of the distributed storage system caused by business operation requests, so as to update the usage capacity statistics of the distributed storage system.

[0006] Finally, this invention provides a distributed storage system, including multiple storage nodes, a metadata manager, and a capacity statistics processor; Each storage node includes at least one physical storage device for storing user data; The metadata manager is used to manage the metadata information of various objects; The capacity statistics processor responds to the business operation request of the target object, triggers the metadata manager to update the metadata information of the target object, and executes the computer program during the metadata manager's update of the metadata information to implement the steps of the capacity statistics method of the distributed storage system described above.

[0007] The advantages of the technical solution provided by this invention are as follows: During the process of updating the target object's metadata, the logical scope of newly added data and the logical scope of deleted data caused by business operations are determined. The snapshot duplicate reference range, which characterizes the shared data range between the target object and the snapshot object, is used. After deducting the overlapping portion of the deleted data logical scope with the snapshot duplicate reference range, the usage capacity change data is calculated in conjunction with the newly added data logical scope. This binds capacity statistics with metadata update transactions, ensuring consistency between statistical results and data operations. Simultaneously, because the portion shared with the snapshot is precisely separated in the deleted data logical scope, capacity statistics only reflect changes in the original object's exclusive data, effectively avoiding duplicate counting problems caused by snapshot sharing. This achieves accurate statistics on the user's actual written exclusive capacity, providing an accurate data foundation for resource scheduling, billing management, and garbage collection in the storage system, thus improving the reliability and availability of the distributed storage system. Furthermore, this invention also provides a corresponding distributed storage system with corresponding advantages for the capacity statistics method used in distributed storage systems. Attached Figure Description

[0008] To more clearly illustrate the technical solutions of the present invention or related technologies, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0009] Figure 1 A flowchart illustrating a method for calculating the usage capacity of a distributed storage system provided by the present invention; Figure 2 This invention provides a schematic diagram of a write request using a capacity update process. Figure 3 This invention provides a schematic diagram of a snapshot deletion request using a capacity update process; Figure 4 This invention provides a schematic diagram of a snapshot rollback request using a capacity update process; Figure 5 A structural framework diagram of an exemplary embodiment of the capacity statistics device for a distributed storage system provided by the present invention; Figure 6This is a structural diagram of an exemplary embodiment of the distributed storage system provided by the present invention. Detailed Implementation

[0010] To enable those skilled in the art to better understand the technical solutions of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. In this specification and the aforementioned drawings, the terms "first," "second," "third," "fourth," etc., are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. The term "exemplary" means "serving as an example, embodiment, or illustration." Any embodiment described herein as "exemplary" is not necessarily to be construed as superior to or better than other embodiments.

[0011] Distributed storage systems achieve high capacity, high availability, and elastic scalability by distributing data across multiple physical nodes, and are widely used in cloud computing, big data, and other scenarios. Capacity statistics directly impact resource scheduling, billing management, and cluster health monitoring. To ensure reliable system operation and service quality, it is essential to accurately calculate the actual amount of data written by users.

[0012] To optimize write performance, distributed storage systems employ append-only write mode. In this mode, all data write operations are sequentially appended to disk, effectively avoiding the seek and latency overhead caused by random write operations in traditional storage systems. Simultaneously, to achieve efficient snapshot functionality, copy-on-write technology is used. This method does not immediately copy all data when creating a snapshot; instead, the snapshot shares the same data blocks with the original object. Only when data is subsequently modified is the modified data block copied, thus greatly improving snapshot creation efficiency and saving storage space. Append-only write and copy-on-write technologies mean that a single physical data block may be referenced by multiple logical objects, such as the current object and its multiple historical snapshots. This multi-reference relationship differs from the simple model of one data point corresponding to one logical user in traditional capacity statistics. When accurate statistics of the net capacity actually written by a user are needed, the data portion shared by multiple logical objects must be accurately extracted from the total physical occupancy to avoid double counting.

[0013] Related technologies maintain an independent reference counter for each data block or data fragment when calculating usage capacity. The counter increments when a data block is created or referenced by a new object or snapshot; it decrements when a reference is deleted. Only when the counter reaches zero is the data block considered invalid, and its occupied capacity is deducted from the statistics. This method directly reflects the degree of data block sharing through the counter value. However, this method introduces significant metadata overhead. Even in scenarios without snapshot functionality, the system still needs to maintain counters for massive amounts of data blocks. Reading, writing, and storing this metadata consumes significant system resources, impacting overall performance. Secondly, this method can only track sharing relationships at the data block level and cannot handle partial sharing within a data block. For example, when some bytes in a data block are shared by a snapshot, while another part is overwritten by new data, this statistical method cannot distinguish between them and will count the entire data block as shared. This leads to statistical redundancy for small-scale modification operations and fails to accurately reflect changes in actual exclusive data capacity.

[0014] Therefore, this invention updates the usage capacity of each object in real time based on changes in the logical range length of the object data when a business request transaction is committed, and records the duplicate reference range of the snapshot for objects that have been snapshotted. This method can accurately count the usage capacity of user data written by the front end, improving the reliability and availability of the distributed storage system. After introducing the technical solution of this invention, various non-limiting embodiments of this invention will be described in detail below with reference to the accompanying drawings and specific embodiments. Please refer to [link to previous text] first. Figure 1 According to the present invention, a method for calculating the usage capacity of a distributed storage system can be implemented as a computer program product. For example, it can be installed and run on storage device nodes, metadata managers, or independent capacity statistics servers within a distributed storage system. This method is used to accurately calculate the exclusive capacity actually written by the user based on changes in the length of the data logical range and in conjunction with the snapshot's repeated reference range when business operations trigger metadata updates. It can be applied to system-level usage capacity statistics as well as to capacity statistics for logical volumes and their snapshots. In some embodiments of this method, the method includes the following steps: S101: In response to the business operation request of the target object, during the process of updating the target object's metadata, determine the scope of newly added data and the scope of deleted data logic of the target object before and after the execution of the business operation request.

[0015] Distributed storage systems organize user-written data into logical entities called objects. Each object has a unique identifier, metadata (such as name, creation time, snapshot information, etc.), and its contained data content. An object is the logical unit that carries and organizes data; the data is the actual content contained within the object and represents the actual load of object storage. Metadata records various information about the object. When data is written, it actually modifies the data content associated with the object and its logical scope. When object data is written to disk, the forward relationship is updated through the metadata L2P (Logical-to-Physical) update process. This update maps the logical scope of the object data to its actual location on the disk. The distributed storage system needs this mapping to determine the location of the data within the object in the data pool. L2P is stored in a Map (mapping table (data structure)) in the form of key-value pairs. The key stores the offset and length of the logical scope of the data, and the value stores the actual offset and length on the disk. The target object is the object corresponding to the data to be processed by the currently received business operation request. The business operation request is a request to perform data processing, snapshot management, or other operations on the target object in the distributed storage system. Business operation requests can be write operations that modify the data of the target object, such as `write`, `truncate`, `zero`, `delete`, and `remove`. These operations change the data content and logical scope of the object, thus affecting the system's capacity usage statistics. The newly added data logical scope is the newly added data logical interval that did not exist before the operation, representing the new data coverage brought about by the operation. The deleted data logical scope is the data logical interval that existed before the operation and is no longer retained after the execution of the business operation request, representing the range of old data removed by the operation. Together, these constitute the information about the change in the target object's data logical scope caused by the execution of the business operation request.

[0016] In this step, the distributed storage system organizes user data into target objects for management. A target object is an independent logical storage unit, corresponding to a file or a contiguous block of data. The system maintains metadata information for the target objects, including a logical-to-physical mapping table that records the correspondence between the target object's logical data scope and its physical storage location. When the distributed storage system receives a business operation request for a target object, such as a write request, truncation request, or deletion request, the execution of this business operation triggers an update to the target object's metadata. During the metadata update process, the system first determines the logical data scope occupied by the target object before and after the business operation. By comparing these two logical data scopes, the system obtains the newly added and deleted logical data scopes.

[0017] S102: Obtain the snapshot duplicate reference range corresponding to the target object, and determine the overlapping data range between the data deletion logic range and the snapshot duplicate reference range.

[0018] Under the copy-on-write snapshot mechanism, an original object can have multiple snapshots created sequentially over time; this chain is called a snapshot chain. Each snapshot object records the data state of the original object at the moment of its creation and shares unmodified data with its predecessor or the original object. Snapshot reuse scope refers to the scope of snapshots referenced by the snapshot identifier carried in the business operation request. Snapshot reuse scope characterizes the range of data shared between the target object and at least one associated snapshot object. For example, reuse scope refers to the logical range of data shared between a snapshot object and its predecessor or the original object. For example, if the original object's data logical range is 0 to 100 bytes, and after snapshot 1 is created, the original object is modified within the range of 0-20 bytes, then the duplicate reference range between snapshot 1 and the previous object (i.e., the original object) is 20-100 bytes (because the data in the range of 20-100 bytes was not modified and remains the same as when it was created); if snapshot 2 is subsequently created, and the original object is modified again within the range of 50-60 bytes, then the duplicate reference range between snapshot 2 and the previous snapshot (i.e., snapshot 1) is 0-49 bytes and 61-100 bytes (i.e., the unmodified portion shared by snapshot 2 and snapshot 1).

[0019] In the previous step, the logical scopes for adding and deleting data were determined. The system then retrieves the snapshot repeat reference scope corresponding to the target object. This information reflects which logical scopes are shared by the target object and its snapshot object. Subsequently, the system determines the overlapping data scope between the logical scope for deleting data and the snapshot repeat reference scope. This overlapping data scope represents the data portions that are about to be deleted from the target object but are shared by the snapshot object.

[0020] S103: Based on the logical range of deleted data and the logical range of added data after deducting overlapping data ranges, determine the data on the changes in the usage capacity of the distributed storage system caused by the business operation request, so as to update the usage capacity statistics of the distributed storage system.

[0021] Among them, the overlapping data range is the overlapping data logical interval between the deleted data logical range and the snapshot repeated reference range; the capacity change data is the value of the actual exclusive storage capacity change of the target object in the distributed storage system triggered by business operation requests. Capacity statistics are essentially the measurement of the exclusive data portion associated with the object.

[0022] In this step, when calculating the capacity change data, the distributed storage system includes all newly added data logical ranges in the capacity increase. For deleted data logical ranges, it first deducts the overlap with the snapshot's duplicate reference range, and then uses the result after deduction as the capacity decrease. Finally, the system updates the distributed storage system's capacity usage statistics based on the calculated capacity change data.

[0023] In the technical solution provided in this embodiment, by determining the new data logical scope and the deleted data logical scope caused by business operations during the process of updating the target object's metadata, and using the snapshot duplicate reference scope, which represents the shared data scope between the target object and the snapshot object, the overlapping part of the deleted data logical scope with the snapshot duplicate reference scope is deducted, and then the used capacity change data is calculated in combination with the new data logical scope. This binds capacity statistics with metadata update transactions, ensuring the consistency between statistical results and data operations. At the same time, since the part shared with the snapshot is accurately separated in the deleted data logical scope, the capacity statistics only reflect the changes in the original object's exclusive data, effectively avoiding the duplicate counting problem caused by snapshot sharing. This achieves accurate statistics on the user's actual written exclusive capacity, providing an accurate data foundation for the storage system's resource scheduling, billing management, and garbage collection, and improving the reliability and availability of the distributed storage system.

[0024] Based on the above embodiments, this embodiment further defines the implementation method for determining the logical scope of newly added data and the logical scope of deleted data, and solves the technical problem of how to accurately identify changes in the logical scope of data caused by business operations, which may include the following: Obtain the metadata information of the target object before executing the business operation request, and determine the second data logical range occupied by the target object after executing the business operation request; the metadata information at least records the first data logical range currently occupied by the target object; the data logical range that exists in the first data logical range but does not exist in the second data logical range is used as the deletion data logical range corresponding to the old data to be deleted; the data logical range that exists in the second data logical range but does not exist in the first data logical range is used as the addition data logical range corresponding to the data to be added.

[0025] In addition to containing the basic attributes of the target object, the metadata information also records at least the first data logical range currently occupied by the target object before executing the business operation request, which is the data reflecting the object's data status; the first data logical range is the total data logical range actually occupied by the target object before executing the business operation request, which is the benchmark for determining changes in the data range; the second data logical range is the total data logical range actually occupied by the target object after executing the business operation request, which is the basis for reflecting the data status after the operation.

[0026] In this embodiment, the first data logical range currently occupied by the target object, as recorded in metadata information, is used to determine all data ranges allocated to the target object before the business operation. Then, the system simulates or calculates the second data logical range that the target object will occupy after the business operation, based on the content of the business operation request. For example, for a write operation, if the write position exceeds the original range, the second data logical range will be expanded by adding a new range to the first data logical range; for a truncation operation, the second data logical range may be smaller than the first data logical range. Next, the first and second data logical ranges are compared. Data logical ranges existing in the first data logical range but not in the second data logical range represent data ranges that will be released after the business operation, while data logical ranges existing in the second data logical range but not in the first data logical range represent newly allocated data ranges after the business operation.

[0027] As can be seen from the above, this embodiment achieves accurate and quantitative definition of the logical range of added and deleted data by performing difference operations based on the logical range of data before and after the operation. This provides accurate basic data for the calculation of subsequent capacity change data, avoids capacity statistics errors caused by ambiguous data range definition, and further improves the accuracy of capacity statistics.

[0028] Based on the above embodiments, this embodiment further defines how to calculate the usage capacity change data using the snapshot repeated reference range, and solves the technical problem of how to quantify the impact of snapshot sharing on capacity statistics, which may include the following: The snapshot repeat reference range is the repeat reference range of the latest snapshot object corresponding to the target snapshot identifier carried in the business operation request. The total length of the data to be deleted is calculated as the length to be deleted, the total length of the data to be added is calculated as the length to be added, and the total length of the overlapping data range is calculated as the overlap length. The first length difference between the overlap length and the length to be deleted is used as the exclusive data length of the target object, and the second length difference between the length to be added and the exclusive data length is used as the usage capacity change data. The snapshot repeat reference range is updated according to the new data logical range, and the new data logical range is deducted from the snapshot repeat reference range.

[0029] Among them, the target snapshot identifier is used to uniquely identify the characteristic information of the snapshot object, carried in the business operation request, and is the basis for the system to identify the target snapshot; the latest snapshot object is the snapshot object with the latest creation time or the largest identifier number among all snapshot objects corresponding to the target object; the length to be deleted is the total data length corresponding to the logical range of data to be deleted, representing the overall scale of the data to be deleted; the length to be added is the total data length corresponding to the logical range of data to be added, representing the overall scale of the data to be added; the overlap length is the total data length corresponding to the overlapping data range, representing the scale of the part of the deleted data shared with the snapshot; the first length difference is the difference between the length to be deleted and the overlap length, representing the length of exclusive data actually deleted by the target object in this operation; the exclusive data length is the data length occupied by the target object alone and not shared with any snapshot object, which is an indicator reflecting the actual capacity occupancy; the second length difference is the difference between the length to be added and the exclusive data length, representing the data of the change in usage capacity caused by this business operation.

[0030] In this embodiment, the target snapshot identifier carried in the business operation request is obtained. This target snapshot identifier indicates the snapshot version associated with the current operation. Based on the target snapshot identifier, the duplicate reference range of the corresponding latest snapshot object is determined. This duplicate reference range records the shared data logical range between the latest snapshot object and the target object. Then, length calculations are performed: the total length of the deleted data logical range is calculated and recorded as the length to be deleted; the total length of the newly added data logical range is calculated and recorded as the length to be added; the total length of the overlapping data range between the deleted data logical range and the snapshot duplicate reference range is calculated and recorded as the overlap length. A first length difference between the length to be deleted and the overlap length is calculated, which represents the length of data in the deleted data logical range that is not shared by any snapshot and is exclusively owned by the target object. Next, a second length difference between the length to be added and the exclusive data length is calculated, which is the net capacity change data caused by this business operation. Since the length to be added is fully included, and the exclusive data length is the actual reduction obtained by stripping the shared portion from the length to be deleted, the second length difference accurately reflects the net capacity change caused by this operation. Because the newly added data is newly occupied by the target object and is no longer shared with the latest snapshot object, the logical scope of the newly added data is finally deducted from the snapshot's repeated reference scope.

[0031] As can be seen from the above, this embodiment achieves accurate quantitative statistics using capacity change data by converting the logical range of data into length for quantitative calculation, and establishes a unified capacity calculation standard. At the same time, it ensures the relevance of the snapshot reference range by associating the latest snapshot object with the target snapshot identifier, and synchronizes the snapshot reference range with the data status of the target object by dynamically updating the snapshot reference range, thus providing an accurate snapshot data foundation for capacity statistics of subsequent business operations, and further improving the continuity and accuracy of statistics.

[0032] Based on the above embodiments, this embodiment provides a method for creating a snapshot object and initializing the snapshot repeat reference range when a write operation is triggered. This solves the technical problems of how to achieve delayed creation of copy-on-write snapshots and how to initialize snapshot sharing information, and may include the following: The target object includes a snapshot repeat reference mapping table that records the association between snapshot identifiers and their corresponding repeat reference ranges. When a business operation request for the target object is received, the target snapshot identifier carried in the business operation request is compared with the latest snapshot identifier in the snapshot repeat reference mapping table. If the target snapshot identifier carried in the business operation request is greater than the latest snapshot identifier in the snapshot repeat reference mapping table, it means that a new snapshot has been created before this business operation, but the snapshot object corresponding to that snapshot has not yet been actually created. In this case, a new snapshot object corresponding to the target snapshot identifier is created. The snapshot object creation operation includes: copying the target object's logical-to-physical mapping table to the new snapshot object. This logical-to-physical mapping table records the mapping relationship between all current data logical ranges of the target object and their physical storage locations. After copying this mapping table, the new snapshot object shares all the current data of the target object. The interval covering all data logical ranges of the target object is used as the repeat reference range corresponding to the new snapshot object. In the target object's snapshot repeat reference mapping table, data is inserted using the target snapshot identifier as the key and the repeat reference range of the new snapshot object as the value. If the target snapshot identifier carried in the business operation request matches the latest snapshot identifier in the snapshot repeat reference map, it means that the business operation is being performed with an existing snapshot object, and there is no need to create a new snapshot object. Then, the snapshot repeat reference map for the target object is read. This map stores snapshot identifiers as keys and repeat reference ranges as values. Each repeat reference range represents the data sharing interval between the corresponding snapshot object and the previous snapshot object or the target object. Using the target snapshot identifier as the retrieval key, the corresponding repeat reference range is found in the snapshot repeat reference map and used as the snapshot repeat reference range for the target object.

[0033] The snapshot repeatable reference mapping table is a mapping table in the target object used to record the association between snapshot identifiers and corresponding repeatable reference ranges. It stores data with snapshot identifiers as keys and repeatable reference ranges as values, and is the core data structure for managing snapshot repeatable reference ranges. The latest snapshot identifier is the identifier with the largest number and the latest creation time of the corresponding snapshot object among all snapshot identifiers recorded in the snapshot repeatable reference mapping table. The new snapshot object is a snapshot copy created for a new target snapshot identifier, reflecting the current state of the target object. The logical-to-physical mapping table records the mapping relationship between the data logical range and physical storage location of the target object, and is the basis for the system to locate the physical storage location of the data. The data sharing interval is the essence of each repeatable reference range in the snapshot repeatable reference mapping table. It is the actual shared data logical interval between the corresponding snapshot object and the previous snapshot object or the target object, and is the basis for identifying shared data.

[0034] In this embodiment, for a distributed system using copy-on-write, a snapshot object is not created when a user creates a snapshot. Only during the first write operation after snapshot creation will the latest snapshot ID be included in the write OP (Operation Request). If the snapshot ID in the OP is found to be larger than the latest snapshot ID recorded in the object's metadata, a cloning operation is performed on the object. A new snapshot object with the snapshot ID in the OP is created, and the original object's metadata is copied to the new snapshot object, while simultaneously updating the original object's metadata. Since the new snapshot object is created at the current moment and shares all data with the target object, the system uses the interval covering the entire logical range of the target object's data as the repeatable reference range of the new snapshot object. The starting position of this interval is 0, and the ending position is the offset from the end position of the target object's data logical range. The system inserts data into the target object's snapshot repeatable reference map table, using the target snapshot identifier as the key and the repeatable reference range of the new snapshot object as the value. In this way, the snapshot repeatable reference map table records the shared information of the new snapshot object. For cases where a new snapshot object does not need to be created, the target object's snapshot repeatable reference map table is read directly. In this mapping table, each repeated reference range represents the data-sharing interval between the corresponding snapshot object and the previous snapshot object or the target object. For example, for a non-latest snapshot object in the snapshot chain, its repeated reference range records the data interval shared by that snapshot object and the previous snapshot object; for the latest snapshot object, its repeated reference range records the data interval shared by that snapshot object and the target object. The system uses the target snapshot identifier carried in the business operation request as the search key to look up the snapshot repeated reference map. Because this mapping table is stored with the snapshot identifier as the key, the system can quickly locate the corresponding entry and obtain the repeated reference range recorded in that entry.

[0035] The system uses the found duplicate reference range as the snapshot duplicate reference range corresponding to the target object for subsequent capacity calculation.

[0036] As can be seen from the above, in this embodiment, the creation of the snapshot object is delayed until the first write operation, avoiding the performance overhead caused by immediately copying data when creating a snapshot. Simultaneously, by initializing the duplicate reference range, accurate shared information is provided for subsequent capacity statistics. Furthermore, even when a snapshot object already exists, the snapshot duplicate reference range required for the current operation can be obtained efficiently and accurately, providing accurate shared information for capacity statistics.

[0037] Based on the above embodiments, this embodiment further defines how to ensure the consistency between capacity statistics and business operation transactions, and solves the technical problem of how to avoid capacity statistics deviation when anomalies occur during business operation execution. This can include the following: During the process of updating the metadata of the target object, a temporary capacity variable is created to record the capacity changes during the transaction processing. When the business operation request is started, the temporary capacity variable is initialized to a first preset value, such as 0, and the capacity change data is recorded in the temporary capacity variable. When the transaction of the business operation request is successfully committed, the data in the temporary capacity variable is added to the capacity usage statistics of the distributed storage system.

[0038] The temporary capacity variable is a temporary variable created during the process of updating the target object's metadata. It is specifically used to record the capacity changes during the transaction processing and is a variable that ensures transaction consistency. A successful transaction commit means that all operations corresponding to the business operation request, such as metadata updates and data reading and writing, have been completed without any exceptions, and the system confirms that the business operation has been successfully executed.

[0039] In this embodiment, after the system responds to the business operation request, a temporary capacity variable is created in the context of updating the target object's metadata. This variable is specifically used to record all capacity changes during the current transaction processing. After calculating the capacity change data caused by the current business operation request, the data is promptly recorded in the temporary capacity variable for temporary storage. Subsequently, the system executes all transaction operations corresponding to this business operation. If all transaction operations are completed without exception, the transaction is successfully committed. At this time, the system adds the capacity change data recorded in the temporary capacity variable to the capacity usage statistics of the distributed storage system, thus updating the statistics. If the transaction is interrupted midway, the data in the temporary capacity variable will be cleared and will not be included in the statistics.

[0040] As can be seen from the above, this embodiment provides a dedicated temporary storage medium for capacity change data by creating a temporary capacity variable, thus avoiding data loss or confusion. At the same time, it binds the update of capacity statistics results to business operation transactions, and only updates the statistics results after the transaction is successfully committed. This effectively avoids the problem of statistical result distortion caused by interruption in business operations, ensures the consistency between capacity statistics results and business operations, and improves the reliability of statistical results.

[0041] Furthermore, this embodiment also addresses the scenario of a distributed storage system organizing data at the data placement group level. It accumulates the data of temporary capacity variables into the capacity record variable of the corresponding data placement group, solving the technical problem of how to perform capacity statistical management at a finer granularity. This can include the following: The distributed storage system organizes target objects into data placement groups. Each data placement group includes a capacity record variable that records the storage capacity currently occupied by the target object. This variable is initialized to a second preset value, such as 0. The temporary capacity variable is used as the amount of change in the capacity record variable of the data placement group to which the target object belongs due to the business operation request. When the transaction of the business operation request is committed, the data of the temporary capacity variable is added to the capacity record variable.

[0042] In this context, a data placement group is a unit in a distributed storage system for organizing and managing target objects. It groups multiple target objects with the same characteristics together, facilitating batch capacity management and resource scheduling. The capacity record variable is a variable specifically used in a data placement group to record the current storage capacity occupied by the target objects within that group. It is an indicator reflecting the overall capacity occupancy status of the data placement group. The change amount refers to the capacity change value brought about by the current business operation request to the capacity record variable of the data placement group to which the target object belongs, i.e., the usage capacity change data recorded in the temporary capacity variable.

[0043] In this embodiment, the distributed storage system organizes target objects into multiple data placement groups, with each target object belonging to one data placement group. The system maintains a capacity record variable for each data placement group, initialized to zero and continuously updated as data is written and deleted. When the system calculates the capacity change data and records it in a temporary capacity variable, it needs to add the data from the temporary capacity variable to the correct capacity statistics when the transaction is committed. To do this, the system first determines which data placement group the target object operated on by the current business operation request belongs to. Then, the system uses the data recorded in the temporary capacity variable as the change in the capacity record variable of the data placement group to which the target object belongs due to the business operation request. Since the temporary capacity variable can be positive (indicating capacity increase) or negative (indicating capacity decrease), this change directly reflects the increase or decrease in the capacity of the data placement group. Finally, when the business operation request transaction is successfully committed, the system adds the data from the temporary capacity variable to the capacity record variable of the data placement group. In this way, the capacity record variable of each data placement group accurately reflects the actual capacity occupancy of all target objects within it. This embodiment achieves refined capacity statistics at the data placement group level, providing an accurate data foundation for resource scheduling and capacity monitoring at the storage pool level.

[0044] To facilitate understanding by those skilled in the art, this embodiment also provides an exemplary implementation, which may include the following: This embodiment maintains a Map clone_overlap (i.e., a snapshot duplicate reference map) in the original object. Map clone_overlap is a mapping structure that records the range of duplicate references between objects in the snapshot chain. The key of this map is the snapshot ID, and the value records the range of duplicate references between the snapshot ID and the previous snapshot ID object. For the latest snapshot ID, its value records the range of duplicate references with the original object data. Through this mechanism, the latest snapshot entry in clone_overlap always directly reflects the sharing relationship between the current original object and the latest snapshot, facilitating rapid calculation of changes in exclusive capacity during each write operation. clone_overlap records the range of duplicate references between each snapshot and the previous snapshot in the snapshot chain. This map is updated in real-time in the business operations. That is, when the system processes front-end business requests, such as each write, snapshot creation, snapshot deletion, or snapshot rollback operation, the records in clone_overlap are updated synchronously according to the type and scope of the operation to accurately reflect changes in the sharing relationship between snapshots. This provides an accurate basis for the sharing range for capacity statistics, ensuring that capacity statistics can isolate shared data and only calculate exclusive capacity. When an object is operated on, its metadata, including the L2P Map (Logical to Physical Map), needs to be modified. Therefore, the capacity usage of the business object can be statistically analyzed during the metadata update process. First, a variable `used_bytes` (i.e., a capacity record variable) is added to the data placement group to record the current storage capacity occupied by a data placement group (PG). This variable is initialized to 0. Subsequently, in each metadata update process that updates the mapping relationship between user objects and internal objects, the current capacity usage of the data placement group is calculated and updated. A new temporary capacity usage variable `tmp_used_bytes` (i.e., a temporary capacity variable) is created in the metadata update operation context and initialized to 0 during OP processing. This variable is used to record the changed capacity usage during the processing flow. In business scenarios with snapshots, capacity statistics are calculated by stripping shared data from the scope of repeated references involved in the request, and only the exclusive scope is counted.

[0045] Taking the front-end's write operation on a business object as an example, the processing flow is as follows: Figure 2 As shown, it may include the following: A1: Compare the snapshot ID carried in the OP with the latest snapshot ID recorded in clone_overlap. If the snapshot ID carried in the OP is updated, then create a new snapshot; otherwise, no snapshot needs to be created.

[0046] A2: For OPs that need to create new snapshot objects, insert the latest snapshot object's overlap in clone_overlap, that is, the key is the snapshot ID in the OP, the value is the range 0~end, and end is the offset of the end position of the original object's data range.

[0047] A3: Get the old L2P of the object, calculate the range old_range and its length remove_length that need to be deleted from the old L2P, and calculate the intersection length overlap_length of old_range and the latest snapshot of the object, tmp_used_bytes minus (remove_length-overlap_length).

[0048] Here, `remove_length - overlap_length` represents the portion of the old data to be deleted that is not shared by any snapshots and is exclusively owned by the original object. When a write operation overwrites old data, only the capacity of the portion of data exclusively owned by the original object is deducted; the portion shared by snapshots is not deducted. In other words, capacity is only reduced when data exclusively owned by the original object is deleted; even if data shared by snapshots is deleted from the original object, its capacity should not be deducted because the snapshot still holds it.

[0049] A4: Calculate the length of the new L2P required for the object, add_length, and increase tmp_used_bytes by add_length. Subtract the range of the new L2P from the overlap of the latest snapshot in clone_overlap.

[0050] Once newly written data is written to disk, it occupies physical storage space, therefore the system's capacity should increase by an amount equal to the logical length of the newly written data. Formula A3 only deducts the portion of old data exclusively used by the original object, leaving the shared portion untouched. Combined with formula A4, all newly written data is included in the capacity, and `tmp_used_bytes` records the net capacity change caused by this operation. That is, net capacity change = length of newly added exclusive data - length of deleted exclusive old data. The range of newly written data (`add_length`) is new data occupied by the original object and is no longer shared with the latest snapshot; therefore, it needs to be deducted from the repeatable reference range of the latest snapshot.

[0051] A5: Complete other OP processing, add tmp_used_bytes to used_bytes, and commit the transaction.

[0052] As can be seen from the above, this embodiment updates the usage capacity of each data placement group in real time according to the change of the logical range length of the object data when the business request transaction is committed, and records the repeated reference range of the snapshot for the object that has been snapshotted. This can accurately count the usage capacity of user data written by the front end and improve the reliability and availability of the distributed storage system.

[0053] Furthermore, this embodiment also provides a method for performing capacity statistics during snapshot deletion operations, solving the technical problem of how to accurately calculate the exclusive capacity to be released in snapshot deletion scenarios and avoid accidentally releasing shared data. This method may include the following: When the business operation request is a snapshot deletion operation, based on the snapshot identifier to be deleted, the first duplicate reference range of the snapshot object to be deleted and the second duplicate reference range of the previous snapshot object are obtained; based on the union of the first and second duplicate reference ranges, the changed duplicate reference range is obtained; the total logical range of the full data occupied by the snapshot object to be deleted is obtained, and the total length of the full logical range of the full data is calculated as the total length of the snapshot to be deleted; the snapshot overlap range between the full logical range of the full data and the changed duplicate reference range is calculated, and the total length of the snapshot overlap range is calculated as the snapshot overlap length; the length difference between the total length of the snapshot to be deleted and the snapshot overlap length is used as the usage capacity change data caused by the snapshot deletion operation; the intersection of the second duplicate reference range and the first duplicate reference range is used as the new duplicate reference range of the previous snapshot object, and the record corresponding to the snapshot object to be deleted is deleted to update the corresponding snapshot duplicate reference mapping table.

[0054] Among them, the snapshot identifier to be deleted is a unique identifier of the snapshot object to be deleted, carried in the business operation request; the snapshot object to be deleted is the snapshot object corresponding to the snapshot identifier to be deleted, and is the target of this deletion operation; the preceding snapshot object is the snapshot object in the snapshot chain that precedes the snapshot object to be deleted and has an earlier creation time. If the snapshot object to be deleted is the first snapshot object, then the preceding snapshot object is the target object; the first duplicate reference range is the duplicate reference range corresponding to the snapshot object to be deleted, that is, the shared data logical range between the snapshot object to be deleted and the preceding snapshot object; the second duplicate reference range is the duplicate reference range corresponding to the preceding snapshot object, that is, the shared data logical range between the preceding snapshot object and the parent snapshot object or the target object; the changing duplicate reference range is the first duplicate reference range and the second duplicate reference range. The logical range corresponding to the union of the repeated reference ranges represents the overall shared data range between the snapshot object to be deleted and the previous snapshot object; the full data logical range is the total data logical interval actually occupied by the snapshot object to be deleted, representing the overall data scale of the snapshot object to be deleted; the total length of the snapshot to be deleted is the total length corresponding to the full data logical range of the snapshot object to be deleted; the snapshot overlap range is the overlapping data logical range between the full data logical range of the snapshot object to be deleted and the changed repeated reference range; the snapshot overlap length is the total length corresponding to the snapshot overlap range, representing the scale of the part shared with other objects in the snapshot object to be deleted; the new repeated reference range is the intersection of the second repeated reference range of the previous snapshot object and the first repeated reference range of the snapshot object to be deleted, which is the repeated reference range after the previous snapshot object is updated.

[0055] In this embodiment, for a snapshot deletion request, two snapshot objects need to be considered: the snapshot object to be deleted and the previous snapshot object. The system obtains the duplicate reference ranges corresponding to these two snapshot objects based on the snapshot identifier to be deleted, and takes the union of the first and second duplicate reference ranges. The union represents the data range shared by the snapshot object to be deleted and the previous snapshot object. This part of the data will still be shared by the previous snapshot object even after the snapshot object to be deleted is deleted, and therefore should not be released. Next, the system obtains the total logical range of the full data occupied by the snapshot object to be deleted and calculates the total length of this full logical range. This total length represents the total physical storage capacity occupied by the snapshot object to be deleted before deletion. The system calculates the total length of the overlapping part of the full logical range and the changed duplicate reference range. This length represents the data part shared by the snapshot object to be deleted and the previous snapshot object. This part of the data will still be held by the previous snapshot object after the snapshot object to be deleted is deleted, and should not be deducted from the capacity statistics. The difference between the total length of the snapshot to be deleted and the overlap length of the snapshots is calculated. This difference represents the data length unique to the snapshot object to be deleted. Only this part of the data will actually release storage space after deletion. In other words, when deleting a forward relationship, the capacity of the portion that intersects with the union of the overlap of two snapshot objects is not reduced. Here, the overlap of the snapshot itself to be deleted represents a duplicate reference between this snapshot and the previous snapshot object (or the original object). The overlap of the previous snapshot object represents a duplicate reference between the previous snapshot object and this snapshot object. Neither of these duplicate references is exclusively occupied by the snapshot object to be deleted; the data capacity unique to the snapshot object to be deleted needs to be subtracted when deleting the snapshot object. Finally, the system updates the snapshot duplicate reference mapping table. Specifically, the intersection of the second duplicate reference range and the first duplicate reference range is taken as the new duplicate reference range of the previous snapshot object, and the record corresponding to the snapshot object to be deleted is deleted. In this way, the shared relationships in the snapshot chain are correctly updated.

[0056] An exemplary snapshot deletion service request execution process is as follows: Figure 3 As shown, it may include the following: B1: Based on the snapshot ID carried in the snapshot deletion OP, obtain the overlap(S) of the same snapshot ID and the overlap(P) of the previous snapshot ID in the original object clone_overlap, and take the union of the two overlaps to obtain delta_overlap.

[0057] B2: Obtain the full L2P Map of the snapshot object to be deleted, with a total data range length of remove_length. Calculate the intersection of the L2P Map and delta_overlap, with the data range length of the intersection being overlap_length.

[0058] B3: Delete all L2Ps of the snapshot object. Subtract (remove_length - overlap_length) from tmp_used_bytes.

[0059] B4: Update clone_overlap, take the intersection of the overlap of the previous snapshot object to be deleted and the overlap of the snapshot to be deleted, and delete the overlap of that snapshot from clone_overlap.

[0060] B5: Complete other OP processing, add tmp_used_bytes to used_bytes, and commit the transaction.

[0061] As can be seen from the above, this embodiment calculates and deducts the capacity of the shared data portion by considering the association between the snapshot object to be deleted and the previous snapshot object in the snapshot chain, thus avoiding the problem of mistakenly including the shared data capacity in the deletion amount and achieving accurate statistics of capacity change data when deleting snapshots. At the same time, by updating the duplicate reference range of the previous snapshot object and deleting the record of the snapshot object to be deleted, the snapshot duplicate reference mapping table is kept consistent with the actual state of the snapshot chain, providing accurate snapshot association data for subsequent capacity statistics and ensuring the continuity of statistical results.

[0062] Furthermore, this embodiment also provides a method for performing capacity statistics during snapshot rollback operations, solving the technical problem of how to accurately calculate capacity changes when rolling back a target object to a historical snapshot state. This method may include the following: When the business operation request is a snapshot rollback operation, the following steps are taken: First, obtain the first logical-to-physical mapping table currently occupied by the target object; calculate the total length of the first full logical range recorded in the first logical-to-physical mapping table as the first rollback total length; second, obtain the latest snapshot duplicate reference range corresponding to the target object; calculate the first rollback overlap range between the first full logical range and the latest snapshot duplicate reference range, and its corresponding first rollback overlap length; third, deduct the difference between the first rollback total length and the first rollback overlap length from the temporary capacity variable, and delete all logical-to-physical mapping tables currently occupied by the target object; fourth, obtain the snapshot identifier to be rolled back, and determine the appropriate snapshot identifier to be rolled back. The intersection of the repeated reference ranges of all snapshot objects between the snapshot object to be rolled back and the target object is used as the comprehensive repeated reference range; the second logical-to-physical mapping table of the snapshot object corresponding to the snapshot identifier to be rolled back is obtained, and the total length of the second full logical range recorded in the second logical-to-physical mapping table is calculated as the second total rollback length; the second rollback overlap range of the second full logical range and the comprehensive repeated reference range and its corresponding second rollback overlap length are calculated; the difference between the second total rollback length and the second rollback overlap length is included in the temporary capacity variable; the second logical-to-physical mapping table of the snapshot object corresponding to the snapshot identifier to be rolled back is copied to the target object.

[0063] The snapshot rollback operation is a business operation that restores the current data state of the target object to the data state of a certain historical snapshot object. The first logical-to-physical mapping table is the logical-to-physical mapping table currently occupied by the target object before the snapshot rollback operation, reflecting the data storage mapping relationship of the target object before rollback. The first full logical scope is the entire data logical scope of the target object recorded in the first logical-to-physical mapping table, representing the data scale of the target object before rollback. The first rollback total length is the total length corresponding to the first full logical scope. The first rollback overlap range is the overlapping data logical scope between the first full logical scope and the latest snapshot repeated reference range corresponding to the target object. The first rollback overlap length is the total length corresponding to the first rollback overlap range. The snapshot identifier to be rolled back is a unique identifier of the historical snapshot object to which the target object needs to be restored, carried in the business operation request. The snapshot object to be rolled back is the snapshot object corresponding to the snapshot identifier to be rolled back, which is the baseline state for the rollback of the target object; the comprehensive duplicate reference range is the intersection of the duplicate reference ranges of all snapshot objects from the snapshot object to be rolled back to the target object, representing the common shared data logical range in this snapshot chain; the second logical-to-physical mapping table is the logical-to-physical mapping table corresponding to the snapshot object to be rolled back, reflecting the data storage mapping relationship of the snapshot object to be rolled back; the second full logical range is the total data logical range of the snapshot object to be rolled back recorded in the second logical-to-physical mapping table, representing the target data scale after the target object is rolled back; the second total rollback length is the total length corresponding to the second full logical range; the second overlapping rollback range is the overlapping data logical range between the second full logical range and the comprehensive duplicate reference range; the second overlapping rollback length is the total length corresponding to the second overlapping rollback range.

[0064] In this embodiment, for snapshot rollback requests, if the original object and the snapshot object to be rolled back have overlapping parts, these parts are not modified, and the remaining rollback parts are considered as new writes. The new write parts are updated according to the modified write capacity, that is, the intersection of the overlap with the latest snapshot object is calculated, and the capacity of this part needs to be increased. First, the logical-to-physical mapping table currently occupied by the target object is obtained. This mapping table records the correspondence between all the current data logical ranges of the target object and their physical storage locations. The system calculates the total length of the full logical range recorded in this mapping table. At the same time, the system obtains the latest snapshot repeated reference range corresponding to the target object, and calculates the overlap range between the first full logical range and the latest snapshot repeated reference range, and then calculates the total length of the overlap range to obtain the part of the current data of the target object that is shared with the latest snapshot. The exclusive part of the current data of the target object that is not shared by any snapshot will be deleted during the rollback process. Therefore, the difference between the first rollback total length and the first rollback overlap length needs to be deducted from the capacity. Subsequently, the system deletes all logical-to-physical mapping tables currently occupied by the target object. Next, the system obtains the identifier of the snapshot to be rolled back and determines the intersection of the repeated reference ranges of all snapshot objects from the snapshot object to be rolled back corresponding to this identifier to the target object, thus obtaining the data range shared by all snapshots from the snapshot to be rolled back to the target object. The system then obtains the logical-to-physical mapping table of the snapshot object corresponding to the snapshot identifier, calculates the total length of the full logical range recorded in this mapping table, and obtains the total data capacity pointed to by the snapshot object to be rolled back. The system also calculates the total length of the overlapping range between the second full logical range and the comprehensive repeated reference range, obtaining the portion of the data in the snapshot object to be rolled back that is shared with other snapshots. The portion of the data in the snapshot object to be rolled back that is unique will become new data in the target object after the rollback; therefore, the difference between the second total rollback length and the second rollback overlap length needs to be included in the capacity. Finally, the second logical-to-physical mapping table of the snapshot object corresponding to the snapshot identifier to be rolled back is copied to the target object, completing the rollback operation.

[0065] An exemplary snapshot rollback service request execution process is as follows: Figure 4 As shown, it may include the following: C1: Get the current L2P Map of the original object, whose total data range length is remove_length, and calculate the intersection of the L2P Map and the latest snapshot overlap, whose length is remove_overlap.

[0066] C2: Delete all current L2p of the original object, and subtract (remove_length-overlap_length) from tmp_used_bytes.

[0067] C3: Based on the snapshot ID carried in the snapshot rollback OP, calculate the duplicate reference range between the snapshot ID to be rolled back and the current original object, i.e., the intersection of the overlaps of all snapshots from the snapshot ID to be rolled back to the original object, rollback_overlap. Copy the L2P Map of the snapshot object to be rolled back to the original object, with a length of rollback_length.

[0068] C4: Calculate the intersection of the L2P Map and rollback_overlap of the snapshot object to be rolled back, with a length of rollback_overlap_length. tmp_used_bytes is incremented by (rollback_length - rollback_overlap_length).

[0069] The introduction of the temporary capacity variable tmp_used_bytes and its updating to used_bytes when the metadata update operation is committed to the database is to ensure the transactionality of capacity statistics and IO operations. If the current metadata update operation is not completed but the cluster encounters an anomaly that causes the process to be interrupted, the update of the current capacity usage should also fail. Otherwise, it will lead to deviations in the capacity usage statistics.

[0070] As can be seen from the above, this embodiment accurately calculates the actual exclusive capacity change brought about by the snapshot rollback operation by deducting the shared portion between the target object's data before rollback and the snapshot, as well as the shared portion between the rollback baseline data and the snapshot chain, thus avoiding statistical errors caused by not considering the sharing relationship. At the same time, by clearing the mapping relationship before rollback and copying the rollback baseline mapping relationship, the consistency between the target object's data state and the rollback requirements is ensured, achieving the accuracy of capacity statistics during snapshot rollback operations and providing accurate data support for system resource management.

[0071] It should be noted that there is no strict order of execution between the steps in this invention. As long as they conform to the logical order, these steps can be executed simultaneously or in a certain preset order. Figures 1-4 This is just an illustrative example and does not mean that this is the only possible execution order.

[0072] This invention also provides a corresponding apparatus for the distributed storage system capacity statistics method, further enhancing the practicality of the method. The apparatus can be described from both a functional module perspective and a hardware perspective. The following describes the distributed storage system capacity statistics apparatus provided by this invention. This apparatus is used to implement the distributed storage system capacity statistics method provided by this invention. In this embodiment, the distributed storage system capacity statistics apparatus may include or be divided into one or more program modules. These one or more program modules are stored in a storage medium and executed by one or more processors to complete the distributed storage system capacity statistics method disclosed in Embodiment 1. The program module referred to in this embodiment refers to a series of computer program instruction segments capable of performing specific functions, which are more suitable than the program itself for describing the execution process of the distributed storage system capacity statistics apparatus in the storage medium. The following description will specifically introduce the functions of each program module in this embodiment. The distributed storage system capacity statistics apparatus described below corresponds to the distributed storage system capacity statistics method described above.

[0073] From the perspective of functional modules, see Figure 5 , Figure 5 A structural diagram of the capacity statistics device for the distributed storage system provided in this embodiment, in one specific implementation, shows that the device may include: The change range determination module 501 is used to respond to the business operation request of the target object and determine the logical range of newly added data and the logical range of deleted data of the target object before and after the execution of the business operation request during the process of updating the target object's metadata. The overlapping data determination module 502 is used to obtain the snapshot duplicate reference range corresponding to the target object, and determine the overlapping data range of the deleted data logical range and the snapshot duplicate reference range; the snapshot duplicate reference range is used to characterize the data range shared between the target object and at least one associated snapshot object; The capacity statistics module 503 is used to determine the changes in the usage capacity of the distributed storage system caused by business operation requests based on the logical range of deleted data and the logical range of added data after deducting overlapping data ranges, so as to update the usage capacity statistics results of the distributed storage system.

[0074] The capacity statistics device for a distributed storage system mentioned above is described from a functional module perspective. Furthermore, this invention also provides an electronic device, described from a hardware perspective. This electronic device includes a memory and a processor. The memory stores a computer program, and the processor is configured to run the computer program to execute the steps in any of the above-described embodiments of the capacity statistics method for a distributed storage system.

[0075] Embodiments of this application also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the steps in any of the above embodiments of the distributed storage system usage capacity statistics method when running.

[0076] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0077] Embodiments of this application also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps in any of the above embodiments of the distributed storage system usage capacity statistics method.

[0078] Embodiments of this application also provide another computer program product, including a non-volatile computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps in any of the above embodiments of the distributed storage system usage capacity statistics method.

[0079] Finally, this invention also provides a distributed storage system, please refer to [link / reference]. Figure 6 The system may include multiple storage nodes 601, a metadata manager 602, and a capacity statistics processor 603. Each storage node 601 includes at least one physical storage device for storing user data; the metadata manager 602 manages the metadata information of each object; the capacity statistics processor 603 responds to business operation requests from target objects, triggers the metadata manager to update the metadata information of the target objects, and executes the steps of the capacity statistics method for the distributed storage system described in any of the above embodiments while executing a computer program during the metadata manager's metadata information update process.

[0080] In this embodiment, multiple storage nodes 601 constitute the system's data storage layer. Each storage node includes at least one physical storage device, such as a solid-state drive (SSD) or a hard disk drive (HDD), which is used to actually store data written by users. Multiple storage nodes are interconnected via a network to jointly provide high-capacity, highly available storage services. The metadata manager 602 is an independent hardware component or logical unit, which may include a first processor and a first memory. The first memory stores a first computer program. When the first processor executes the first computer program, it manages the metadata information of each target object. The metadata information includes a logical-to-physical mapping table that records the mapping relationship between the logical range of target object data and its physical storage location, and a snapshot repeat reference mapping table that records the association relationship between snapshot identifiers and repeat reference ranges. The repeat reference range is used to characterize the data range shared between the target object and at least one associated snapshot object. The capacity statistics processor 603 is also an independent hardware component or logical unit, including a second processor and a second memory. The second memory stores a second computer program. When the second processor executes the second computer program, it implements the steps in the capacity statistics method embodiment of any of the above-described distributed storage systems. During system operation, the capacity statistics processor 603 responds to the business operation request of the target object, triggers the metadata manager 602 to update the metadata information of the target object, and executes a second computer program to calculate the capacity change data during the metadata manager's update of the metadata information, and feeds back the calculation results to the metadata manager or directly updates the system's capacity statistics records.

[0081] As shown above, this embodiment decouples the capacity statistics function from the metadata management function. Simultaneously, the capacity statistics processor executes statistical methods synchronously during metadata updates, ensuring the real-time nature and accuracy of the statistics. Multiple storage nodes provide ample data storage capacity, the metadata manager ensures the correctness of data location information, and the capacity statistics processor ensures the accuracy of capacity statistics, together forming a distributed storage system capable of accurately calculating the user's dedicated capacity. The distributed storage system's capacity statistics method updates the used capacity in real time based on changes in the length of the object's logical scope and introduces overlap statistical snapshots of capacity occupancy. This allows for accurate calculation of the actual used capacity at the business layer and provides queries at the storage pool level or system level. On one hand, it provides customers with accurate capacity usage statistics, improving the user experience; on the other hand, it provides the backend garbage collection module with statistics on the effective data volume. The backend garbage collection module can use this to calculate the ratio of effective data to garbage data for adjusting the garbage collection speed, improving the availability and reliability of the storage system.

[0082] The above provides a detailed description of a distributed storage system and its capacity statistics method provided by the present invention. The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. Whether the units and algorithm steps of the various examples described in the disclosed embodiments are executed by electronic hardware or computer software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, and such implementations should not be considered beyond the scope of the present invention. Several improvements and modifications can be made to the present invention without departing from its principles, and these improvements and modifications also fall within the protection scope of the present invention.

Claims

1. A method for calculating the usage capacity of a distributed storage system, characterized in that, include: In response to a business operation request from a target object, during the process of updating the target object's metadata, the scope of newly added data and the scope of deleted data for the target object before and after the execution of the business operation request are determined. Obtain the snapshot duplicate reference range corresponding to the target object, and determine the overlapping data range of the deleted data logical range and the snapshot duplicate reference range; the snapshot duplicate reference range is used to characterize the data range shared between the target object and at least one associated snapshot object; Based on the deleted data logical range after deducting the overlapping data range and the newly added data logical range, the data on the change in the usage capacity of the distributed storage system caused by the business operation request is determined, so as to update the usage capacity statistics of the distributed storage system.

2. The method for calculating the usage capacity of a distributed storage system according to claim 1, characterized in that, Determining the logical scope of newly added data and the logical scope of deleted data for the target object before and after the execution of the business operation request includes: Obtain metadata information of the target object before executing the business operation request, and determine the second data logical range occupied by the target object after executing the business operation request; the metadata information at least records the first data logical range currently occupied by the target object; The data logic range that exists in the first data logic range but not in the second data logic range shall be used as the deletion data logic range corresponding to the old data to be deleted. The data logic range that exists in the second data logic range but not in the first data logic range is taken as the new data logic range corresponding to the data to be added.

3. The method for calculating the usage capacity of a distributed storage system according to claim 2, characterized in that, The snapshot duplication reference range is the duplication reference range of the latest snapshot object corresponding to the target snapshot identifier carried in the business operation request. Based on the deleted data logical range (excluding the overlapping data range) and the added data logical range, the data determining the capacity change data of the distributed storage system caused by the business operation request includes: The total length of the logical range of the data to be deleted is calculated as the length to be deleted; the total length of the logical range of the data to be added is calculated as the length to be added; and the total length of the overlapping data range is calculated as the overlap length. The first length difference between the overlap length and the length to be deleted is used as the exclusive data length of the target object, and the second length difference between the length to be added and the exclusive data length is used as the usage capacity change data. Based on the newly added data logical range, update the snapshot repeat reference range, and deduct the newly added data logical range from the snapshot repeat reference range.

4. The method for calculating the usage capacity of a distributed storage system according to claim 1, characterized in that, The target object includes a snapshot repeat reference mapping table for recording the association between snapshot identifiers and corresponding repeat reference ranges, and responds to the target object's business operation requests, including: If the target snapshot identifier carried in the business operation request is greater than the latest snapshot identifier in the snapshot duplicate reference mapping table, then a new snapshot object corresponding to the target snapshot identifier is created; Copy the logical-to-physical mapping table of the target object to the new snapshot object, and use the interval that covers the entire data logical range of the target object as the repeat reference range corresponding to the new snapshot object; In the snapshot duplicate reference mapping table of the target object, data is inserted using the target snapshot identifier as the key and the duplicate reference range of the new snapshot object as the value.

5. The method for calculating the usage capacity of a distributed storage system according to claim 4, characterized in that, The target snapshot identifier carried in the business operation request is consistent with the latest snapshot identifier in the snapshot duplicate reference mapping table. The snapshot duplicate reference range corresponding to the target object is obtained, including: Read the snapshot duplicate reference mapping table of the target object. The snapshot duplicate reference mapping table is stored with snapshot identifier as key and duplicate reference range as value. Each duplicate reference range represents the data sharing interval between the corresponding snapshot object and the previous snapshot object or the target object. Using the target snapshot identifier as the retrieval key, the corresponding duplicate reference range is searched from the snapshot duplicate reference mapping table and used as the snapshot duplicate reference range corresponding to the target object.

6. The method for calculating the usage capacity of a distributed storage system according to claim 1, characterized in that, Update the usage capacity statistics of the distributed storage system, including: During the process of updating the metadata of the target object, a temporary capacity variable is created to record capacity changes during transaction processing; The data on the change in usage capacity is recorded in the temporary capacity variable; When the transaction of the business operation request is successfully committed, the data in the temporary capacity variable is added to the usage capacity statistics of the distributed storage system.

7. The method for calculating the usage capacity of a distributed storage system according to claim 6, characterized in that, The distributed storage system organizes target objects into data placement groups. Each data placement group includes a capacity recording variable that records the current storage capacity occupied by the target object. The data in the temporary capacity variable is accumulated into the usage capacity statistics of the distributed storage system, including: The temporary capacity variable is used as the amount of change in the capacity record variable of the data placement group to which the target object belongs caused by the business operation request; When the transaction of the business operation request is committed, the data of the temporary capacity variable is added to the capacity record variable.

8. The method for calculating the usage capacity of a distributed storage system according to any one of claims 1 to 7, characterized in that, When the business operation request is a snapshot deletion operation, it also includes: Based on the snapshot identifier to be deleted, obtain the first duplicate reference range of the snapshot object to be deleted and the second duplicate reference range of the previous snapshot object of the snapshot object to be deleted; The variable repeated reference range is obtained based on the union of the first repeated reference range and the second repeated reference range. Obtain the total logical range of the full data occupied by the snapshot object to be deleted, and calculate the total length of the full logical range of the data as the total length of the snapshot to be deleted; Calculate the snapshot overlap range between the full data logical range and the changed repeated reference range, and calculate the total length of the snapshot overlap range as the snapshot overlap length; The length difference between the total length of the snapshot to be deleted and the overlap length of the snapshots is used as the data of the change in usage capacity caused by the snapshot deletion operation. The intersection of the second duplicate reference range and the first duplicate reference range is taken as the new duplicate reference range of the previous snapshot object, and the record corresponding to the snapshot object to be deleted is deleted to update the corresponding snapshot duplicate reference mapping table.

9. The method for calculating the usage capacity of a distributed storage system according to any one of claims 1 to 7, characterized in that, When the business operation request is a snapshot rollback operation, it also includes: Obtain the first logical-to-physical mapping table currently occupied by the target object, and calculate the total length of the first full logical range recorded in the first logical-to-physical mapping table as the first rollback total length; Obtain the latest snapshot repeated reference range corresponding to the target object, and calculate the first rollback overlap range and the corresponding first rollback overlap length of the first full logical range and the latest snapshot repeated reference range. The difference between the first total rollback length and the first rollback overlap length is deducted from the temporary capacity variable, and all logical-to-physical mapping tables currently occupied by the target object are deleted. Obtain the snapshot identifier to be rolled back, and determine the intersection of the repeated reference ranges of all snapshot objects from the snapshot object to be rolled back corresponding to the snapshot identifier to the target object, as the comprehensive repeated reference range; Obtain the second logical-to-physical mapping table of the snapshot object corresponding to the snapshot identifier to be rolled back, and calculate the total length of the second full logical range recorded in the second logical-to-physical mapping table as the second total rollback length; Calculate the second rollback overlap range of the second full logic range and the comprehensive repeated reference range, and its corresponding second rollback overlap length; The difference between the second total rollback length and the second rollback overlap length is included in the temporary capacity variable; Copy the second logical-to-physical mapping table of the snapshot object corresponding to the snapshot identifier to be rolled back to the target object.

10. A distributed storage system, characterized in that, Includes multiple storage nodes, a metadata manager, and a capacity statistics processor; Each storage node includes at least one physical storage device for storing user data; The metadata manager is used to manage the metadata information of each object; The capacity statistics processor responds to the business operation request of the target object, triggers the metadata manager to update the metadata information of the target object, and executes the computer program during the metadata manager's update of the metadata information, thereby implementing the steps of the capacity statistics method for the distributed storage system as described in any one of claims 1 to 9.

Images