A quota calibration method, device, equipment and readable storage medium

Abstract

The specification provides a quota calibration method, device and equipment and readable storage medium, the method comprises: in response to the quota adjustment request initiated by the quota slave unit, the quota adjustment request comprises the current allocated capacity upper limit recorded by the quota slave unit and the adjustment value to be adjusted; according to the current allocated capacity upper limit recorded by the quota slave unit, the adjustment value to be adjusted and the capacity upper limit allocated by the quota management unit to the quota slave unit, a preset rule is used to generate a calibration value; according to the calibration value, the current allocated capacity upper limit recorded by the quota slave unit associated with the quota data misalignment event and the capacity upper limit allocated by the quota management unit to the quota slave unit are calibrated. Through the technical scheme of the specification, the calibration logic is embedded in the conventional quota adjustment request interaction process, when the master-slave data inconsistency is detected, the calibration value is dynamically generated and applied by using the preset rule, and high-precision, real-time embedded self-healing is realized.

Description

Technical Field

[0001] This specification relates to the field of communication technology, and in particular to a quota calibration method, apparatus, device and readable storage medium. Background Technology

[0002] The generation and processing demands of massive amounts of data have driven the widespread adoption of distributed file systems. These systems interconnect multiple independent devices via a network to build large and scalable storage clusters to address the challenges of storing massive amounts of unstructured data. In such a shared storage environment, to prevent individual users, user groups, or specific applications from uncontrollably consuming excessive storage space and thus crowding out the resources of other entities, and to ensure the fair allocation and efficient utilization of storage resources, implementing precise quota management has become one of the core functions of distributed file systems.

[0003] Quota management not only prevents excessive resource consumption by setting space usage limits, but also helps system administrators clearly understand the allocation and consumption of storage resources, enabling more effective capacity planning and adjustments. To strictly enforce quota limits while minimizing the impact on file system input / output performance, modern distributed file systems commonly employ quota caching mechanisms.

[0004] This mechanism typically employs a master-slave architecture. One master node (e.g., a quota management unit) maintains the total upper limit of the quota policy and aggregates global usage, while multiple slave nodes (e.g., quota slave units configured on various data servers) cache a portion of the allocated quota in their local memory to accelerate local quota checks and deductions. Maintaining consistency of master-slave quota data in a complex distributed environment composed of multiple potentially faulty independent nodes presents a significant challenge.

[0005] Taking the Ceph file system as an example, it employs a quota calibration scheme based on scheduled tasks to address inconsistency issues. This scheme persists all quota information in a centralized storage object, while only maintaining non-persistent caches on individual metadata servers. When a potential inconsistency is detected, the system triggers a calibration process, which mainly includes two steps: first, clearing the quota caches of all metadata server nodes; and then, recalculating and updating the persistent quota data in the centralized storage object based on the directory's metadata information. This existing technical solution has several inherent flaws, limiting its reliability and effectiveness in production environments. First, its calibration process cannot guarantee atomicity. In scenarios with multiple concurrent operations, the time window between clearing the cache and reconstructing the data, as well as potential race conditions, can still lead to momentary or continuous inaccuracies in quota statistics. Second, this calibration scheme heavily relies on the file system's directory metadata as the data source for reconstructing quotas. However, the metadata itself may be corrupted or incomplete due to metadata server failures, software errors, or network partitions. Reconstructing based on unreliable metadata may introduce new, more difficult-to-detect data inconsistencies, or even exacerbate system chaos. Furthermore, the solution employs a periodic timed task triggering mechanism, which lacks real-time capability and cannot respond immediately when an anomaly occurs, resulting in a long inconsistency window. At the same time, its method of clearing the cache and rebuilding it is crude and time-consuming. During the calibration period, the quota function may be unavailable or unstable for a long time, which seriously affects the continuity of upper-layer services. Summary of the Invention

[0006] In view of this, this specification provides a quota calibration method, apparatus, device, and readable storage medium to improve the problem of poor quota data calibration results mentioned above.

[0007] The specific technical solution is as follows: This specification provides a quota calibration method applied to a quota management unit of a distributed storage system. The distributed storage system further includes several quota slave units configured and associated with each MDS. The quota management unit also records the maximum capacity allocated to each MDS's associated quota slave unit. Each quota slave unit records the maximum capacity allocated to the associated storage quota on its associated MDS and the amount already used. The method includes: responding to a quota adjustment request initiated by a quota slave unit, the quota adjustment request including the currently allocated capacity recorded by the quota slave unit. Limit and adjustment value; in response to a quota data misalignment event, a calibration value is generated using preset rules based on the currently allocated capacity limit recorded by the quota slave unit, the adjustment value, and the allocated capacity limit recorded by the quota management unit for that quota slave unit. The quota data misalignment event includes situations where the currently allocated capacity limit recorded by the quota slave unit is less than the allocated capacity limit recorded by the quota management unit for that quota slave unit. The calibration value is used to calibrate the currently allocated capacity limit recorded by the quota slave unit and the allocated capacity limit recorded by the quota management unit for that quota slave unit associated with the quota data misalignment event.

[0008] As a technical solution, the step of generating a calibration value using preset rules includes: if the quota adjustment request is a quota increase request, and based on the case where the current allocated capacity limit recorded by the quota slave unit plus the value to be adjusted is less than or equal to the allocated capacity limit recorded by the quota management unit for the quota slave unit, the allocated capacity limit recorded by the quota management unit is used as the calibration value.

[0009] As a technical solution, the step of generating a calibration value using preset rules includes: if the quota adjustment request is a quota increase request, and the current allocated capacity limit recorded by the quota slave unit plus the value to be adjusted is greater than the current allocated capacity limit recorded by the quota management unit for that quota slave unit, the current allocated capacity limit recorded by the quota slave unit plus the value to be adjusted is used as the calibration value.

[0010] As a technical solution, the step of calibrating the quota data misalignment event associated with the quota slave unit's currently allocated capacity limit and the quota management unit's recorded allocated capacity limit for the quota slave unit, including: updating the quota management unit's recorded allocated capacity limit for the quota slave unit with the calibration value, and sending the calibration value to the quota slave unit so that the quota slave unit updates the quota slave unit's recorded currently allocated capacity limit with the calibration value.

[0011] As a technical solution, the step of generating a calibration value using preset rules includes: if the quota adjustment request is a quota reclamation request, subtracting the value to be adjusted from the currently allocated capacity limit recorded by the quota slave unit as the calibration value; calibrating the currently allocated capacity limit recorded by the quota slave unit and the allocated capacity limit recorded by the quota management unit for the quota data inaccuracy event associated with the calibration value, including: updating the allocated capacity limit recorded by the quota management unit for the quota slave unit using the calibration value.

[0012] This specification also provides a quota calibration device for a quota management unit in a distributed storage system. The distributed storage system further includes several quota slave units configured and associated with each MDS. The quota management unit also records the maximum capacity allocated to each MDS's associated quota slave unit. Each quota slave unit records the maximum capacity allocated to the associated storage quota on its associated MDS and the amount already used. The device includes: a first module, configured to respond to a quota adjustment request initiated by a quota slave unit, the quota adjustment request including the currently allocated maximum capacity and the amount already used by the quota slave unit. The first module is used to generate a calibration value based on a preset rule in response to a quota data misalignment event. This calibration value is determined by the quota slave unit's currently allocated capacity limit, the value to be adjusted, and the quota management unit's recorded capacity limit for that quota slave unit. The quota data misalignment event includes situations where the quota slave unit's currently allocated capacity limit is less than the quota management unit's recorded capacity limit for that quota slave unit. The second module is used to calibrate the quota slave unit's currently allocated capacity limit and the quota management unit's recorded capacity limit for that quota slave unit in relation to the quota data misalignment event based on the calibration value.

[0013] As a technical solution, the step of generating a calibration value using preset rules includes: if the quota adjustment request is a quota increase request, and based on the case where the current allocated capacity limit recorded by the quota slave unit plus the value to be adjusted is less than or equal to the allocated capacity limit recorded by the quota management unit for the quota slave unit, the allocated capacity limit recorded by the quota management unit is used as the calibration value.

[0014] As a technical solution, the step of generating a calibration value using preset rules includes: if the quota adjustment request is a quota increase request, and the current allocated capacity limit recorded by the quota slave unit plus the value to be adjusted is greater than the current allocated capacity limit recorded by the quota management unit for that quota slave unit, the current allocated capacity limit recorded by the quota slave unit plus the value to be adjusted is used as the calibration value.

[0015] As a technical solution, the step of calibrating the quota data misalignment event associated with the quota slave unit's currently allocated capacity limit and the quota management unit's recorded allocated capacity limit for the quota slave unit, including: updating the quota management unit's recorded allocated capacity limit for the quota slave unit with the calibration value, and sending the calibration value to the quota slave unit so that the quota slave unit updates the quota slave unit's recorded currently allocated capacity limit with the calibration value.

[0016] As a technical solution, the step of generating a calibration value using preset rules includes: if the quota adjustment request is a quota reclamation request, subtracting the value to be adjusted from the currently allocated capacity limit recorded by the quota slave unit as the calibration value; calibrating the currently allocated capacity limit recorded by the quota slave unit and the allocated capacity limit recorded by the quota management unit for the quota data inaccuracy event associated with the calibration value, including: updating the allocated capacity limit recorded by the quota management unit for the quota slave unit using the calibration value.

[0017] This specification also provides an electronic device including a processor and a readable storage medium storing machine-executable instructions that can be executed by the processor to implement the aforementioned quota calibration method.

[0018] This specification also provides a readable storage medium storing machine-executable instructions that, when invoked and executed by a processor, cause the processor to implement the aforementioned quota calibration method.

[0019] The technical solutions provided in this specification offer at least the following beneficial effects: By embedding calibration logic into the regular quota adjustment request interaction process, and dynamically generating and applying calibration values ​​using preset rules when master-slave data inconsistency is detected, high-precision, real-time embedded self-healing is achieved. This method replaces polling with triggered calibration, avoiding additional overhead, and uses a lightweight, single-stage protocol to correct data while ensuring no significant impact on normal IO performance, significantly improving the reliability, availability, and system resource efficiency of quota management. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of this specification or the prior art, the drawings used in the description of the embodiments of this specification or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this specification. For those skilled in the art, other drawings can be obtained based on these drawings of the embodiments of this specification.

[0021] Figure 1 This is a flowchart of a quota calibration method in one embodiment of this specification; Figure 2 This is a schematic diagram of the misalignment judgment process in one embodiment of this specification; Figure 3 This is a calibration flowchart of one embodiment of this specification; Figure 4 This is a calibration flowchart of one embodiment of this specification; Figure 5 This is a calibration flowchart of one embodiment of this specification; Figure 6 This is a calibration flowchart of one embodiment of this specification; Figure 7 This is a structural diagram of a quota calibration device according to one embodiment of this specification; Figure 8 This is a hardware structure diagram of an electronic device according to one embodiment of this specification.

[0022] Reference numerals: Module 1 21, Module 22, Module 3 23. Detailed Implementation

[0023] The terminology used in the embodiments described herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of this specification. The singular forms “a,” “described,” and “the” as used in this specification and claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to any and all possible combinations comprising one or more of the associated listed items.

[0024] It should be understood that although the terms first, second, third, etc., may be used to describe various information in embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this specification, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" may also be interpreted as "when," "when," or "in response to a determination."

[0025] This specification provides a quota calibration method, apparatus, device, and readable storage medium to at least improve one of the above-mentioned technical problems.

[0026] The specific technical solution is described below.

[0027] In one embodiment, this specification provides a quota calibration method applied to a quota management unit of a distributed storage system. The distributed storage system further includes several quota slave units configured and associated with each MDS. The quota management unit also records the maximum capacity allocated to each MDS's associated quota slave unit. The quota slave unit records the maximum capacity allocated to the associated storage quota on the associated MDS and the amount already used. The method includes: responding to a quota adjustment request initiated by a quota slave unit, the quota adjustment request including the currently allocated quota... The quota data includes the allocated capacity limit and the value to be adjusted; in response to a quota data misalignment event, a calibration value is generated using preset rules based on the currently allocated capacity limit and the value to be adjusted recorded by the quota slave unit and the allocated capacity limit recorded by the quota management unit for that quota slave unit. The quota data misalignment event includes situations where the currently allocated capacity limit recorded by the quota slave unit is less than the allocated capacity limit recorded by the quota management unit for that quota slave unit; the calibration value is used to calibrate the currently allocated capacity limit recorded by the quota slave unit and the allocated capacity limit recorded by the quota management unit for that quota slave unit associated with the quota data misalignment event.

[0028] In a specific scenario, the quota management unit can exist in the form of quota_local, and the quota slave unit can also exist in the form of quota_local. The quota management unit also records the maximum capacity of the associated storage quota allocated to each MDS associated quota slave unit, which can be recorded in several data items such as quota_leader_allocate.quota_local_0, quota_leader_allocate.quota_local_1, quota_leader_allocate.quota_local_2, etc. under quota_leader_allocate. The total capacity allocated to all quota slave units is recorded in quota_leader_total_allocate. The maximum capacity and the amount used by the quota slave unit on the associated MDS associated with the storage quota can be recorded in quota_local_hard and quota_local_used.

[0029] The distributed storage system is built on a multi-MDS (Multi-Data Service) architecture. The system is also configured with several quota slave units. Each quota slave unit is uniquely associated with its corresponding MDS. Each MDS corresponds to one and only one quota slave unit. The quota slave unit serves as the dedicated carrier of the MDS in the quota management system, undertaking all quota-related operations and data records of that MDS. The quota management unit, as the master node for quota management in the distributed storage system, coordinates and manages the allocation and calibration of storage quotas for the entire system. It is the core unit for achieving unified management and control of data from all quota slave units.

[0030] In the quota management system of a distributed storage system, the quota management unit establishes an independent quota allocation record for each quota slave unit, accurately recording the capacity limit of the associated storage quota allocated to each MDS associated quota slave unit. This capacity limit is the highest threshold for quota usage set by the quota management unit for the corresponding quota slave unit, and is the core basis for the quota slave unit to carry out storage resource usage. Correspondingly, each quota slave unit synchronously records the capacity limit allocated to the associated MDS associated with the storage quota, and also calculates and records the actual usage of the quota on the corresponding MDS in real time. This usage is dynamically updated with storage read and write operations on the MDS. Based on its own recorded capacity limit and usage, the quota slave unit will perform real-time control over the storage resource usage of the corresponding MDS. When the usage approaches or reaches the capacity limit, it will trigger operations such as quota application or write restriction. The quota management unit and each quota slave unit will communicate with each other through network communication to ensure the synchronization of quota allocation and usage information.

[0031] Specifically, such as Figure 1 This includes the following steps, the order of which can be changed depending on the needs of the actual application scenario: Step S11, in response to a quota adjustment request initiated by the quota slave unit, the quota adjustment request includes the currently allocated capacity limit and the value to be adjusted recorded by the quota slave unit.

[0032] The quota management unit responds to quota adjustment requests initiated by quota slave units. These requests are initiated by the quota slave units when their usage of storage resources in the corresponding MDS approaches or reaches their recorded capacity limit, or when there is a need to reclaim quota. This is the core interaction instruction between the quota slave units and the quota management unit. The quota adjustment request is not a simple numerical request, but rather includes key quota data recorded locally by the quota slave unit. The core data includes the currently recorded allocated capacity limit and the value to be adjusted. The positive or negative value of the value to be adjusted corresponds to different types of quota adjustments. When the value to be adjusted is positive, it indicates that the quota slave unit has increased storage resource usage demand and has initiated a quota allocation request to the quota management unit; the amount of quota requested is the positive value to be adjusted. When the value to be adjusted is negative, it indicates that the quota slave unit has idle quota and has initiated a quota reclamation request to the quota management unit; the amount of quota to be reclaimed is the absolute value of the negative value to be adjusted. Of course, in one implementation, a separate flag can be used to indicate whether the adjustment is for requesting an increase in quota or for withdrawing quota, while keeping the value to be adjusted a positive number.

[0033] When a quota slave unit initiates the request, the two types of data mentioned above are encapsulated and sent to the quota management unit through a dedicated communication link to ensure the accuracy and real-time performance of data transmission. After receiving the quota adjustment request, the quota management unit starts the data parsing process, extracts the current capacity limit and the value to be adjusted from the quota slave unit in the request, and retrieves the capacity limit data recorded by itself for the quota slave unit, so as to prepare data for subsequent quota data verification and calibration operations.

[0034] Step S12: In response to the quota data inaccuracy event, a calibration value is generated using preset rules based on the current allocated capacity limit and the value to be adjusted recorded by the quota slave unit and the capacity limit allocated to the quota slave unit recorded by the quota management unit.

[0035] After parsing the data for the quota adjustment request, a consistency check of the quota data is performed to determine whether a quota data inaccuracy event has been triggered. The quota data inaccuracy event defined in this method has a clear and unique judgment criterion: when the current allocated capacity limit recorded by the quota slave unit is less than the allocated capacity limit recorded by the quota management unit for that quota slave unit, a quota data inaccuracy event is determined to be triggered.

[0036] In the quota management system of a distributed storage system, the quota data interaction between the quota management unit and the quota slave units mainly consists of two core processes: quota application and allocation, and quota reclamation. In the quota application and allocation process, the predetermined execution logic is that the quota management unit first writes the allocated quota data to its local database for persistence, and then sends a quota allocation message to the quota slave unit. Upon receiving the message, the quota slave unit then writes the corresponding quota data to its local database to complete the update. If an MDS (Mean Distributed Storage) event occurs during this process... Faults such as system crashes, network interruptions, and process crashes can cause quota slave units to fail to receive allocation messages or update their local data after the quota management unit completes data writing. In this case, the capacity limit recorded by the quota management unit has increased, while the capacity limit recorded by the quota slave unit remains the original value, resulting in data discrepancies. In the quota reclamation process, the established execution logic is that the quota slave unit first writes the reclaimed quota data to its local database for persistence, and then sends a quota reclamation message to the quota management unit. After receiving the message, the quota management unit writes the corresponding quota data to its local database to complete the update. If the above-mentioned faults occur during this process, the quota slave unit will complete data updates, but the quota management unit will not receive the reclamation message or complete its local data update. In this case, the capacity limit recorded by the quota slave unit has decreased, while the capacity limit recorded by the quota management unit remains the original value, also resulting in data discrepancies.

[0037] The inconsistencies in quota data caused by the above-mentioned fault scenarios will ultimately manifest as the current capacity limit recorded by the quota unit being less than the corresponding capacity limit recorded by the quota management unit.

[0038] After determining that a quota data inaccuracy event has been triggered, the calibration value generation process is initiated. Based on the current allocated capacity limit and the value to be adjusted recorded by the quota slave unit, and the capacity limit allocated to the quota slave unit recorded by the quota management unit, the calibration value is generated using preset rules.

[0039] Step S13: Based on the calibration value, calibrate the current allocated capacity limit recorded by the quota slave unit and the allocated capacity limit recorded by the quota management unit for the quota slave unit associated with the quota data misalignment event.

[0040] In one implementation, when a client needs to write data to a directory, the quota slave unit associated with the metadata server where that directory resides first checks its locally cached remaining quota corresponding to that quota target (such as a user or directory). If the local quota is sufficient, the quota slave unit directly deducts the capacity required for this write operation, updates the locally used quota, and then allows the IO operation to proceed.

[0041] However, when the remaining local quota of the quota slave unit is insufficient to meet the current request, or when more quota needs to be requested from the global management unit according to the preset policy, the quota slave unit initiates a quota adjustment request to the quota management unit.

[0042] This request contains at least two key pieces of information: first, the currently recorded allocated capacity limit of the quota slave unit, denoted as quota_local_hard; and second, the amount of quota to be requested, i.e., the adjustment value delta. Upon receiving this request, the quota management unit checks its recorded allocated capacity limit for the quota slave unit, denoted as quota_local_0 (taking unit 0 as an example), and verifies the remaining global quota. If the global quota is sufficient, the quota management unit updates its record, incrementing quota_local_0 by delta, and authorizes this newly added delta quota to the quota slave unit. After receiving the authorization, the quota slave unit updates its local record of the capacity limit, incrementing it by delta, and can then use this new quota to complete I / O operations. The quota reclamation process is similar but in the reverse direction. When a file is deleted, the released quota is deducted from the quota slave unit's local record and may be returned to the quota management unit through a reporting mechanism, thereby reducing the value of quota_local_0.

[0043] The core of this invention lies in the embedded addition of data consistency verification and calibration logic to the conventional quota adjustment request interaction path described above. Specifically, upon receiving a quota adjustment request, the quota management unit does not immediately and unconditionally approve the request. Instead, it first compares the quota_local_hard carried in the request with its own recorded quota_local_0. This comparison is the key criterion for triggering calibration. In an ideal state of consistency, quota_local_0 should equal quota_local_hard, because the total amount allocated by the quota management unit should be fully reflected in the local record of the quota slave unit. However, after the specific fault described above occurs in the system, these two values ​​will deviate. A key and detectable manifestation of the "quota data inaccuracy event" defined in this invention is that the current allocated capacity limit (i.e., quota_local_hard) recorded by the quota slave unit is less than the allocated capacity limit (i.e., quota_local_0) recorded by the quota management unit for that quota slave unit. This state, where quota_local_0 > quota_local_hard, clearly indicates that the quota management unit believes it has allocated more quota to the slave unit than actually recorded by the quota slave unit. This difference (quota_local_0 - quota_local_hard) can be called "hovering quota" or "inconsistency difference." It has not been effectively recorded and used by either party, but it occupies the global quota, leading to the distortion of quota data.

[0044] Once the quota management unit detects that quota_local_hard < quota_local_0, it determines that a quota data misalignment event has occurred and immediately triggers the embedded calibration process. At this time, the quota management unit will call the preset calibration rules to process the current adjustment request delta. The core goal of the preset rules is to use this adjustment request as an opportunity to eliminate the detected inconsistent difference while meeting the current quota application (or recovery) requirements, so as to make the data of both the master and slave sides consistent. The basic logic of the rule is: since there is a positive difference diff = quota_local_0 - quota_local_hard, the delta quota of this request can be preferentially satisfied or offset from this suspended diff quota. The specific rules for generating the calibration value are as follows: The quota management unit will calculate diff. If this request is for applying for a quota (delta is positive), then compare delta with diff. If delta <= diff, it means that all the quota applied for this time can be satisfied by cashing in that part of the suspended quota without using new quota from the global quota pool. Therefore, the calibration value adjust is equal to delta. Logically, the quota management unit will mark this delta as having been officially allocated to the quota slave unit from the unconfirmed suspended state. However, note that since quota_local_0 already includes this part of the quota (this is exactly the source of the inconsistency), the quota management unit does not need to modify the value of quota_local_0. It only needs to indicate in the response to the quota slave unit to increase its quota_local_hard by delta. After this operation, the updated quota_local_hard_new of the quota slave unit = quota_local_hard_old + delta, while quota_local_0 remains unchanged, and the difference diff between the two will be reduced to diff - delta. If delta > diff, it means that the quota applied for this time is greater than the suspended quota. At this time, the rule will preferentially use up all the suspended quota diff to satisfy part of the application, and the remaining quota delta - diff needs to be truly applied from the global quota pool. Therefore, the calibration operation involves two parts: First, for the diff part, the same as the previous logic, the quota management unit instructs the quota slave unit to increase its quota_local_hard by diff to eliminate all inconsistencies. Second, for the remaining delta - diff part, the quota management unit needs to officially allocate it from the global quota, so it will update quota_local_0, increase it by delta - diff, and authorize the quota slave unit to increase its quota_local_hard by delta - diff in the response.The overall effect is that the quota from the unit's quota_local_hard increases by delta, while the quota management unit's quota_local_0 increases by delta - diff, ultimately making quota_local_hard_new equal to quota_local_0_new, restoring data consistency, and ensuring that this application is fully satisfied.

[0045] If the request is to reclaim quota (a negative delta indicates quota release), the rules still apply, but the logic is slightly extended. When the quota management unit detects an inconsistency (quota_local_0 > quota_local_hard) and receives a request to release quota, it realizes that the quota that the quota slave unit is trying to release may stem from a previously unsynchronized allocation. The calibration rule will first use the released quota abs(delta) to offset the inconsistency difference diff. Specifically, the quota management unit will reduce its recorded quota_local_0 by a amount that can be min(abs(delta), diff). Simultaneously, in the response, it may instruct the quota slave unit to adjust its local records accordingly (e.g., confirming the release operation, but the locally used quota has been reduced; the details of this interaction may be fine-tuned depending on the specific implementation, but the core principle is to use the reclamation operation to correct master-slave differences). In this way, the reclamation operation also becomes an opportunity to correct inconsistencies. Whether it's application or reclamation, the calibration operation is atomically bound to the processing of this business request. In other words, the calculation of the calibration value, the updating of data between the master and slave, and the decision to approve or reject the quota adjustment request are all completed within the same processing context and network round trip. This ensures the immediacy and atomicity of the calibration action, avoiding the long inconsistency window and concurrency issues caused by clearing the cache and then rebuilding it in existing technologies. After the calibration action is completed, the master and slave data reach an agreement on this quota item for the quota slave unit, and subsequent operations will be based on this consistent state until a new failure occurs and triggers calibration again.

[0046] For a more specific illustration, assume that the quota management unit records that a total quota of 100GB has been allocated to quota slave unit A (quota_local_0 = 100GB). However, due to a historical failure, quota slave unit A only records 80GB locally (quota_local_hard = 80GB), resulting in an inconsistent difference of 20GB (diff = 20GB). Scenario 1: At this time, quota slave unit A initiates a request to write data and applies for 10GB (delta = 10GB). The quota management unit detects that quota_local_hard (80GB) < quota_local_0 (100GB) and triggers calibration. Since delta (10GB) <= diff (20GB), the first rule is adopted. The quota management unit does not change quota_local_0 (it remains 100GB), but authorizes quota slave unit A to increase its quota_local_hard by 10GB in the response. After quota slave unit A updates, quota_local_hard_new = 90GB. At this time, there is still a 10GB difference between quota_local_0 (100GB) and quota_local_hard_new (90GB), but it has decreased by 10GB compared to before. This 10GB application is satisfied, and no new global quota is consumed. Scenario 2: If quota slave unit A applies for 30GB (delta = 30GB). Since delta (30GB) > diff (20GB), the second rule is adopted. First, the quota management unit instructs quota slave unit A to increase its quota_local_hard by diff, which is 20GB, to eliminate all inconsistencies. Then, for the remaining 10GB (30 - 20), the quota management unit officially allocates it from the global quota pool. Therefore, quota_local_0 is increased by 10GB to become 110GB, and quota slave unit A is authorized to increase its quota_local_hard by another 10GB. Finally, quota_local_hard_new of quota slave unit A = 80 + 20 + 10 = 110GB, and quota_local_0_new of the quota management unit = 100 + 10 = 110GB. The data of both sides is consistent, and the 30GB application is fully satisfied. Scenario 3: If quota slave unit A deletes a file and releases 15GB of space (delta = -15GB, indicating recovery). After the quota management unit detects the inconsistency, it uses this recovery operation for calibration. It can reduce quota_local_0 by min(abs(-15GB), 20GB) = 15GB to become 85GB. At the same time, in the response, it processes the update of the local quota of quota slave unit A (for example, confirming the release of the quota).This is equivalent to partially offsetting the 20GB of false records with the released 15GB of quota. Regardless of the scenario, calibration occurs with a normal business request, has minimal impact on the latency of the client's I / O path, and only adds a simple numerical comparison and conditional operation, achieving high-performance embedded calibration.

[0047] Furthermore, this implementation also considers the completeness of calibration. Besides calibration being triggered by quota adjustment requests initiated by quota slave units, the quota management unit can also proactively initiate calibration probes under specific conditions. For example, when the quota management unit detects that a quota slave unit's allocated quota is about to be exhausted, or when performing a periodic health check, it can proactively send a query request to that quota slave unit to obtain its current quota_local_hard value. By comparing this value with its own recorded quota_local_0, inconsistencies can also be detected. If detected, the quota management unit can proactively initiate a "calibration adjustment," with its delta set to 0, or a virtual adjustment value designed based on the difference (diff), thereby triggering the aforementioned calibration process to synchronize data, even if there are no actual business requests at this time. This provides another mechanism to ensure eventual data consistency. The entire system ensures high consistency of quota master-slave data through ubiquitous, interaction-based verification and on-demand calibration, greatly shortening the inconsistency window while avoiding the huge system overhead caused by full polling calibration. It achieves high-precision, high-real-time, and low-performance quota management, significantly improving the resilience and business continuity assurance capabilities of the distributed storage system in the face of various software and hardware failures.

[0048] In one implementation, generating a calibration value using preset rules includes: if the quota adjustment request is a quota increase request, and based on the case where the current allocated capacity limit recorded by the quota slave unit plus the value to be adjusted is less than or equal to the allocated capacity limit recorded by the quota management unit for the quota slave unit, using the allocated capacity limit recorded by the quota management unit as the calibration value.

[0049] As a technical solution, the step of generating a calibration value using preset rules includes: if the quota adjustment request is a quota increase request, and the current allocated capacity limit recorded by the quota slave unit plus the value to be adjusted is greater than the current allocated capacity limit recorded by the quota management unit for that quota slave unit, the current allocated capacity limit recorded by the quota slave unit plus the value to be adjusted is used as the calibration value.

[0050] In one implementation, the step of calibrating the quota data misalignment event associated with the quota slave unit's currently allocated capacity limit and the quota management unit's recorded allocated capacity limit for the quota slave unit, including: updating the quota management unit's recorded allocated capacity limit for the quota slave unit with the calibration value, and sending the calibration value to the quota slave unit so that the quota slave unit updates the quota slave unit's recorded currently allocated capacity limit with the calibration value.

[0051] In one implementation, generating a calibration value using preset rules includes: if the quota adjustment request is a quota reclamation request, subtracting the value to be adjusted from the currently allocated capacity limit recorded by the quota slave unit as the calibration value; calibrating the currently allocated capacity limit recorded by the quota slave unit and the allocated capacity limit recorded by the quota management unit for the quota data misalignment event according to the calibration value, including: updating the allocated capacity limit recorded by the quota management unit for the quota slave unit using the calibration value.

[0052] In one implementation, such as Figure 2 The `quota_leader` contains a `quota_leader_allocate` field, which stores the specific quota value allocated by the `quota_leader` to each `quota_local`. The `quota_local` also stores the quota amount received from the `quota_leader`, which should normally be the same as the value in `quota_leader_allocate`.

[0053] When `quota_local` requests a quota, it sends the requested quota value of 1G and its current `quota_local_hard=0` to `quota_leader`. `quota_leader` compares the `quota_local_hard=0` sent by `quota_local` with its stored value `quota_leader_allocate.quota_local_0`. Normally, these two values ​​should be equal. If they are equal, no further action is needed. If they are not equal, it indicates an anomaly and requires calibration. For detailed procedures, please refer to the calibration process below. Afterward, `quota_leader` allocates a 1G quota to `quota_local` and updates `quota_leader_total_allocate` and `quota_leader_allocate.quota_local_0`. Upon receiving the reply message from `quota_leader` allocating a 1G quota, `quota_local` increments its `quota_local_hard` value by 1G.

[0054] If a fault occurs during the quota application and allocation process, causing a discrepancy between quota_local_hard and quota_leader_allocate.quota_local_0, then an abnormal calibration process will be performed.

[0055] During the quota allocation process, such as Figure 3 When `quota_local` requests a quota, it sends the requested quota value of 0.5G and its current `quota_local_hard` value of 1G to `quota_leader`. `quota_leader` compares the `quota_local_hard` value of 1G sent by `quota_local` with its own stored value `quota_leader_allocate.quota_local_0` of 2G. Since the difference is 1G > the requested 0.5G, the `quota_leader` increment is calculated to be 0, and 1G is allocated to `quota_local`. The values ​​of `quota_leader_total_allocate` and `quota_leader_allocate.quota_local_0` do not need to be changed. Upon receiving the reply message from `quota_leader` allocating 1G of quota, `quota_local_hard` is incremented by 1G, becoming 2G.

[0056] like Figure 4 When `quota_local` requests a quota, it sends the requested quota value of 1G and its current `quota_local_hard` value of 1G to `quota_leader`. `quota_leader` compares the `quota_local_hard` value of 1G sent by `quota_local` with its own stored value `quota_leader_allocate.quota_local_0` = 1.2G. The difference is 0.2G < the requested 1G, so the increment for `quota_leader` is calculated to be 0.8G. Therefore, it allocates 1G to `quota_local`, and finally updates `quota_leader_total_allocate` and `quota_leader_allocate.quota_local_0` to 2G. Upon receiving the reply message from `quota_leader` allocating 1G of quota, `quota_local_hard` is incremented by 1G, making it 2G.

[0057] like Figure 5When `quota_local` reclaims quota, `quota_local_hard` is 2G, and the calculated amount of quota to be reclaimed is 1G. The reclaimed quota value of 1G and `quota_local_hard` of 2G are sent to `quota_leader`. `quota_leader` compares the `quota_local_hard` of 2G sent by `quota_local` with its own stored value `quota_leader_allocate.quota_local_0` of 2.2G. The difference is 0.2G, and the current `quota_leader` increment is calculated as -(1G + 0.2G). The values ​​of `quota_leader_total_allocate` and `quota_leader_allocate.quota_local_0` are then changed to 1G.

[0058] like Figure 6 When the quota_leader is about to run out, it sends a message to quota_local requesting calibration. quota_local sends quota_local_hard=1G to quota_leader. quota_leader compares quota_local_hard=1G sent by quota_local with its own stored quota_leader_allocate.quota_local_0=1.2G. The difference is 0.2G, so the quota_leader increment is calculated to be -0.2G, and the values ​​of quota_leader_total_allocate and quota_leader_allocate.quota_local_0 are changed to 1G.

[0059] In one implementation, such as Figure 7This specification also provides a quota calibration device applied to a quota management unit of a distributed storage system. The distributed storage system further includes several quota slave units configured and associated with each MDS. The quota management unit also records the maximum capacity allocated to the quota slave units associated with each MDS for each associated storage quota. The quota slave units record the maximum capacity allocated to the associated storage quota on the associated MDS and the amount already used. The device includes: a first module, configured to respond to a quota adjustment request initiated by a quota slave unit, the quota adjustment request including the currently allocated maximum capacity recorded by the quota slave unit. The first module is used to generate a calibration value based on a preset rule in response to a quota data misalignment event, which includes the quota data misalignment event where the current allocated capacity limit recorded by the quota slave unit, the value to be adjusted, and the quota management unit's recorded capacity limit for the quota slave unit. The second module is used to generate a calibration value based on a preset rule in response to a quota data misalignment event, which includes the quota slave unit's recorded current allocated capacity limit being less than the quota management unit's recorded capacity limit for the quota slave unit. The third module is used to calibrate the quota data misalignment event-related ...

[0060] In one implementation, generating a calibration value using preset rules includes: if the quota adjustment request is a quota increase request, and based on the case where the current allocated capacity limit recorded by the quota slave unit plus the value to be adjusted is less than or equal to the allocated capacity limit recorded by the quota management unit for the quota slave unit, using the allocated capacity limit recorded by the quota management unit as the calibration value.

[0061] In one implementation, generating a calibration value using preset rules includes: if the quota adjustment request is a quota increase request, and based on the case where the current allocated capacity limit recorded by the quota slave unit plus the value to be adjusted is greater than the current allocated capacity limit recorded by the quota management unit for the quota slave unit, using the current allocated capacity limit recorded by the quota slave unit plus the value to be adjusted as the calibration value.

[0062] In one implementation, the step of calibrating the quota data misalignment event associated with the quota slave unit's currently allocated capacity limit and the quota management unit's recorded allocated capacity limit for the quota slave unit, including: updating the quota management unit's recorded allocated capacity limit for the quota slave unit with the calibration value, and sending the calibration value to the quota slave unit so that the quota slave unit updates the quota slave unit's recorded currently allocated capacity limit with the calibration value.

[0063] In one implementation, generating a calibration value using preset rules includes: if the quota adjustment request is a quota reclamation request, subtracting the value to be adjusted from the currently allocated capacity limit recorded by the quota slave unit as the calibration value; calibrating the currently allocated capacity limit recorded by the quota slave unit and the allocated capacity limit recorded by the quota management unit for the quota data misalignment event according to the calibration value, including: updating the allocated capacity limit recorded by the quota management unit for the quota slave unit using the calibration value.

[0064] The implementation methods of the apparatus are the same as or similar to the corresponding implementation methods, and will not be described again here.

[0065] In one embodiment, this specification provides an electronic device including a processor and a readable storage medium storing machine-executable instructions executable by the processor. The processor executes the machine-executable instructions to implement the aforementioned quota calibration method. From a hardware perspective, a hardware architecture diagram can be found... Figure 8 As shown.

[0066] In one embodiment, this specification provides a readable storage medium storing machine-executable instructions that, when invoked and executed by a processor, cause the processor to implement the aforementioned quota calibration method.

[0067] Here, a readable storage medium can be any electronic, magnetic, optical, or other physical storage device that can contain or store information, such as executable instructions, data, etc. For example, a readable storage medium can be: RAM (Random Access Memory), volatile memory, non-volatile memory, flash memory, storage drives (such as hard disk drives), solid-state drives, any type of storage disk (such as optical discs, DVDs, etc.), or similar storage media, or combinations thereof.

[0068] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer, which can take the form of a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email sending and receiving device, game console, tablet computer, wearable device, or any combination of these devices.

[0069] For ease of description, the above devices are described in terms of function, divided into various units. Of course, in implementing this specification, the functions of each unit can be implemented in one or more software and / or hardware components.

[0070] Those skilled in the art will understand that embodiments of this specification can be provided as methods, systems, or computer program products. Therefore, this specification can take the form of a completely hardware implementation, a completely software implementation, or an implementation combining software and hardware aspects. Furthermore, embodiments of this specification can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0071] This specification is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments thereof. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0072] Furthermore, these computer program instructions can also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in the process. Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0073] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0074] Those skilled in the art will understand that embodiments of this specification can be provided as methods, systems, or computer program products. Therefore, this specification can take the form of a completely hardware implementation, a completely software implementation, or an implementation combining software and hardware aspects. Furthermore, this specification can take the form of a computer program product implemented on one or more computer-usable storage media (which may include, but are not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0075] The above description is merely an embodiment of this specification and is not intended to limit this specification. Various modifications and variations can be made to this specification by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this specification should be included within the scope of the claims of this specification.

Claims

1. A quota calibration method, characterized in that, A quota management unit is applied to a distributed storage system, wherein the distributed storage system further includes several quota slave units configured and associated with each MDS, the quota management unit also records the capacity limit of the associated storage quota allocated to the quota slave units associated with each MDS, and the quota slave units record the capacity limit associated with the storage quota allocated on the associated MDS and the amount already used, the method comprising: In response to a quota adjustment request initiated by a quota slave unit, the quota adjustment request includes the currently allocated capacity limit and the value to be adjusted recorded by the quota slave unit; In response to a quota data misalignment event, a calibration value is generated using preset rules based on the currently allocated capacity limit and the value to be adjusted recorded by the quota slave unit and the allocated capacity limit recorded by the quota management unit for that quota slave unit. The quota data misalignment event includes situations where the currently allocated capacity limit recorded by the quota slave unit is less than the allocated capacity limit recorded by the quota management unit for that quota slave unit. The calibration value is used to calibrate the quota data misalignment event associated with the quota slave unit record of the currently allocated capacity limit and the quota management unit record of the allocated capacity limit of the quota slave unit.

2. The method according to claim 1, characterized in that, The process of generating calibration values ​​using preset rules includes: If the quota adjustment request is a quota increase request, the quota management unit uses the quota management unit's recorded quota allocation limit as the calibration value, based on the case where the current allocated capacity limit recorded by the quota slave unit plus the value to be adjusted is less than or equal to the quota management unit's recorded quota allocation limit for that quota slave unit.

3. The method according to claim 1, characterized in that, The process of generating calibration values ​​using preset rules includes: If the quota adjustment request is a quota increase request, the current allocated capacity limit recorded by the quota slave unit plus the value to be adjusted is used as the calibration value, based on the case where the current allocated capacity limit recorded by the quota slave unit plus the value to be adjusted is greater than the current allocated capacity limit recorded by the quota management unit for that quota slave unit.

4. The method according to claim 2 or 3, characterized in that, The step of calibrating the quota data inaccuracy event associated with the calibration value, including the currently allocated capacity limit recorded by the quota slave unit and the allocated capacity limit recorded by the quota management unit for that quota slave unit, includes: The calibration value is used to update the maximum capacity allocated to the quota slave unit as recorded by the quota management unit, and the calibration value is sent to the quota slave unit so that the quota slave unit updates the current maximum capacity allocated to the quota slave unit as recorded by the calibration value.

5. The method according to claim 1, characterized in that, The process of generating calibration values ​​using preset rules includes: If the quota adjustment request is a quota reclamation request, the calibration value is obtained by subtracting the value to be adjusted from the current allocated capacity limit recorded in the cell. The calibration values ​​are used to determine the current allocated capacity limit of the quota slave unit and the allocated capacity limit of the quota slave unit as recorded by the quota management unit, which are associated with the quota data misalignment event. Update the quota management unit record of the maximum capacity allocated to the unit for that quota using the calibration value.

6. A quota calibration device, characterized in that, A quota management unit for a distributed storage system, the distributed storage system further comprising a plurality of quota slave units configured and associated with each MDS, the quota management unit further recording the capacity limit of the associated storage quota allocated to the quota slave units associated with each MDS, the quota slave units recording the capacity limit allocated to the associated storage quota on the associated MDS and the amount used, the device comprising: The first module is configured to respond to a quota adjustment request initiated by a quota slave unit, the quota adjustment request including the currently allocated capacity limit and the value to be adjusted recorded by the quota slave unit; The second module is used to respond to a quota data misalignment event by generating a calibration value using preset rules based on the currently allocated capacity limit and the value to be adjusted recorded by the quota slave unit and the allocated capacity limit recorded by the quota management unit for that quota slave unit. The quota data misalignment event includes situations where the currently allocated capacity limit recorded by the quota slave unit is less than the allocated capacity limit recorded by the quota management unit for that quota slave unit. The third module is used to calibrate the current allocated capacity limit of the quota slave unit and the allocated capacity limit of the quota slave unit recorded by the quota management unit, which are associated with the quota data misalignment event, based on the calibration value.

7. The apparatus according to claim 6, characterized in that, The process of generating calibration values ​​using preset rules includes: If the quota adjustment request is a quota increase request, the quota management unit uses the quota management unit's recorded quota allocation limit as the calibration value, based on the case where the current allocated capacity limit recorded by the quota slave unit plus the value to be adjusted is less than or equal to the quota management unit's recorded quota allocation limit for that quota slave unit.

8. The apparatus according to claim 6, characterized in that, The process of generating calibration values ​​using preset rules includes: If the quota adjustment request is a quota increase request, the current allocated capacity limit recorded by the quota slave unit plus the value to be adjusted is used as the calibration value, based on the case where the current allocated capacity limit recorded by the quota slave unit plus the value to be adjusted is greater than the current allocated capacity limit recorded by the quota management unit for that quota slave unit.

9. The apparatus according to claim 7 or 8, characterized in that, The step of calibrating the quota data inaccuracy event associated with the calibration value, including the currently allocated capacity limit recorded by the quota slave unit and the allocated capacity limit recorded by the quota management unit for that quota slave unit, includes: The calibration value is used to update the maximum capacity allocated to the quota slave unit as recorded by the quota management unit, and the calibration value is sent to the quota slave unit so that the quota slave unit updates the current maximum capacity allocated to the quota slave unit as recorded by the calibration value.

10. The apparatus according to claim 6, characterized in that, The process of generating calibration values ​​using preset rules includes: If the quota adjustment request is a quota reclamation request, the calibration value is obtained by subtracting the value to be adjusted from the current allocated capacity limit recorded in the cell. The calibration values ​​are used to determine the current allocated capacity limit of the quota slave unit and the allocated capacity limit of the quota slave unit as recorded by the quota management unit, which are associated with the quota data misalignment event. Update the quota management unit record of the maximum capacity allocated to the unit for that quota using the calibration value.

11. An electronic device, characterized in that, include: A processor and a readable storage medium storing machine-executable instructions that can be executed by the processor to implement the method of any one of claims 1-5.

12. A readable storage medium, characterized in that, The readable storage medium stores machine-executable instructions that, when invoked and executed by a processor, cause the processor to implement the method described in any one of claims 1-5.

Images