Data volume calibration method, device and equipment and readable storage medium

Abstract

The invention provides a data volume calibration method, device and equipment and a readable storage medium, and the method comprises the steps: selecting an aggregation object in response to the enabling of a junk data volume calibration function, and screening to-be-verified KV data in the aggregation object according to a verification bitmap; according to the to-be-verified KV data which are obtained through screening and are not marked as the to-be-deleted data, the corresponding metadata are compared, and according to a comparison result, the KV data which are junk data and are not marked as the to-be-deleted data are determined; and according to the KV data which are junk data and are not marked as to-be-deleted data and / or the KV data marked as to-be-deleted data in the aggregation object, calibrating the junk data volume in the aggregation object. The method has the technical effects that the total garbage amount of the system is efficiently and accurately calibrated, and the problem of misalignment of garbage amount statistics caused by fault accumulation is effectively solved.

Description

TECHNICAL FIELD

[0001] The present specification relates to the technical field of communication, and in particular, to a data volume calibration method and device, equipment and a readable storage medium. BACKGROUND

[0002] Redirect on write (ROW) technology, as an effective data management strategy, is widely used to improve storage performance. This technology aggregates the user's random and small-scale write operations and redirects them to new logical addresses, thereby converting random writes into sequential writes and combining small input / output (I / O) operations into large I / O operations, significantly improving the write efficiency and disk life of the storage system. Under the ROW mechanism, the association between user data and its corresponding metadata is recorded in the tail of the ROW object. Specifically, these metadata are usually stored in the form of key-value pairs (KV data), where the key contains information such as user identification, data offset, and length, and the value points to the corresponding ROW object name and its internal position and length. When the ROW object aggregates full data, the association relationship metadata in the tail will be finally committed and persisted to the disk.

[0003] The storage system based on the ROW technology faces the problem of the accuracy of system garbage collection (GC) statistics. Garbage collection is a key background process in a storage system for recycling the space occupied by invalid data, and the effectiveness of its recovery strategy is highly dependent on the accurate statistics of the total amount of invalid data (i.e., the amount of garbage) in the system. In actual operation, a distributed storage system may fail due to various reasons, such as unexpected power failure, hardware error, or software exception, etc. Such failure events may cause the failure of critical operation steps, such as the failure of the operation of submitting the metadata association relationship in the tail of the ROW to the metadata database (DB), or the failure of the persistent storage of the successfully calculated garbage amount. Once such a failure occurs, it will cause the loss of part of the garbage amount statistics data. Since the system runs continuously, such failures may occur multiple times, and over time, there will be a larger and larger deviation between the recorded garbage amount in the system and the actual existing garbage amount, i.e., the phenomenon of garbage amount misalignment occurs.

[0004] The inaccuracy of garbage amount statistics can directly cause a chain of negative effects. Garbage collection mechanisms usually dynamically adjust their collection strategies based on the statistics of garbage amount, such as determining when to trigger collection, the intensity or priority of collection, and the like. When the dependent garbage amount data is inaccurate, the formulated collection strategy will also be inaccurate. If the system underestimates the actual garbage amount, it may perform the collection operation too conservatively, causing invalid data to be unable to be cleaned up in time. On a disk with limited storage space, this may eventually cause the risk of the disk space being gradually occupied, thereby affecting the normal write operation of the storage system, and even causing the service to be unavailable. Conversely, if the system overestimates the garbage amount, it may cause unnecessary and excessive collection operations, consuming additional system input / output (I / O) and computing resources, thereby negatively affecting the performance of the foreground business of the system.

[0005] For the problem of systematic inaccuracy of garbage amount caused by multiple cumulative faults, there is a lack of effective detection and calibration mechanism. The conventional system recovery process may focus on restoring data consistency or service availability, but often ignores the accuracy repair of the garbage amount statistic index. Once the garbage amount is inaccurate, the system continues to run on the basis of incorrect information, and the problem of inaccurate garbage collection strategy cannot be fundamentally solved, and the potential risk of full disk always exists. Therefore, there is an urgent need for a technical solution that can accurately calibrate the garbage amount after it is inaccurate and simultaneously repair the collection strategy, so as to guarantee the long-term stability and efficient operation of the distributed storage system. SUMMARY

[0006] Therefore, the present specification provides a data amount calibration method, device, equipment and readable storage medium to improve the above-mentioned problem of lack of garbage data amount calibration.

[0007] The specific technical solutions are as follows: The present specification provides a data amount calibration method applied to a storage device, the method comprising: in response to a garbage data amount calibration function being enabled, selecting an aggregation object, screening KV data to be checked in the aggregation object according to a check bitmap, the check bitmap recording the marking information of each KV data marked as data to be deleted; comparing the corresponding metadata according to each KV data to be checked that is not marked as data to be deleted screened out, and confirming the KV data that is garbage data and not marked as data to be deleted according to the comparison result; calibrating the garbage data amount in the aggregation object according to the KV data that is garbage data and not marked as data to be deleted and / or the KV data marked as data to be deleted in the aggregation object; and cyclically executing the above steps to traverse and calibrate the garbage data amount of each aggregation object included in the storage device.

[0008] As a technical solution, the method comprises: according to the currently recorded garbage data amount, preferentially calibrating the aggregation object with less garbage data amount.

[0009] As a technical solution, the method comprises: accumulating the garbage data amount of each calibrated aggregation object to obtain a calibrated total garbage data amount, and calling and executing a corresponding preset garbage recovery strategy according to the total garbage data amount.

[0010] As a technical solution, the method comprises: in response to data deletion signaling, updating the check bitmap and marking the corresponding KV data as to-be-deleted data.

[0011] As a technical solution, the garbage data amount calibration function is enabled in response to the garbage data amount calibration instruction issued by the user, or the garbage data amount calibration function enabled at a preset time, or the event that the current total data amount exceeds a preset threshold.

[0012] The present specification also provides a data amount calibration device applied to a storage device, which comprises: a first module configured to select an aggregation object in response to the garbage data amount calibration function being enabled, and to filter to-be-checked KV data in the aggregation object according to a check bitmap, wherein the check bitmap records marking information of each KV data marked as to-be-deleted data; a second module configured to compare corresponding metadata according to each to-be-checked KV data not marked as to-be-deleted data obtained by filtering, and to confirm KV data that is garbage data and not marked as to-be-deleted data according to a comparison result; and a third module configured to calibrate garbage data amount in the aggregation object according to KV data that is garbage data and not marked as to-be-deleted data and / or KV data marked as to-be-deleted data in the aggregation object; and the first module, the second module and the third module are called in a loop to traverse and calibrate garbage data amount of each aggregation object included in the storage device.

[0013] As a technical solution, the first module preferentially selects an aggregation object with less garbage data amount according to a currently recorded garbage data amount.

[0014] As a technical solution, the third module is further configured to accumulate the garbage data amount of each calibrated aggregation object to obtain a calibrated total garbage data amount, and to call and execute a corresponding preset garbage recovery strategy according to the total garbage data amount.

[0015] As a technical solution, the first module is further configured to update the check bitmap and mark the corresponding KV data as to-be-deleted data in response to data deletion signaling.

[0016] As a technical solution, the garbage data volume calibration function enabled in response to comprises: responding to the garbage data volume calibration instruction issued by the user; or, responding to the plan of the garbage data volume calibration function enabled at a preset time; or, responding to the event that the current total data volume exceeds the preset threshold.

[0017] The present specification also provides an electronic device comprising a processor and a readable storage medium, the readable storage medium storing machine executable instructions capable of being executed by the processor, and the processor executes the machine executable instructions to implement the aforementioned data volume calibration method.

[0018] The present specification also provides a readable storage medium storing machine executable instructions, and the machine executable instructions, when invoked and executed by a processor, cause the processor to implement the aforementioned data volume calibration method.

[0019] The above technical solutions provided by the present specification bring at least the following beneficial effects: The introduction of the check bitmap quickly screens the data to be checked, and accurately compares the metadata, effectively solving the problem of inaccurate garbage volume statistics caused by fault accumulation. This method can efficiently and accurately calibrate the total system garbage volume, dynamically optimize the recycling strategy accordingly, significantly reduce the risk of full disk caused by inaccurate recycling strategy, avoid unnecessary IO overhead, and ensure the stability and space recycling efficiency of the storage system. BRIEF DESCRIPTION OF DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present specification or the prior art, the following will briefly introduce the drawings needed in the description of the embodiments of the present specification or the prior art. Obviously, the drawings in the following description are only some embodiments described in the present specification, and other drawings can also be obtained by those skilled in the art according to these drawings of the embodiments of the present specification.

[0021] Figure 1 is a flowchart of the data volume calibration method in an embodiment of the present specification; Figure 2 is a structural diagram of the data volume calibration device in an embodiment of the present specification; Figure 3 is a hardware structural diagram of the electronic device in an embodiment of the present specification.

[0022] Reference signs: first module 21, second module 22, third module 23. DETAILED DESCRIPTION

[0023] The terminology used in the description of the implementations herein is for the purpose of describing particular implementations only and is not intended to be limiting. As used in the description of the implementations and the claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It also will be understood that the term “and / or” as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items.

[0024] It will be understood that, although the terms first, second, third, etc. can be used herein to describe various information, the information should not be limited by these terms. These terms are only used to distinguish one piece of information from another. For example, a first information can also be termed a second information, and, similarly, a second information can also be termed a first information, without departing from the scope of the present description. Furthermore, the word “if’ can be interpreted as meaning “when” or “upon” or “in response to determining” depending on the context.

[0025] In view of the above, the present description provides a data volume calibration method, device, equipment and readable storage medium to at least improve one of the above technical problems.

[0026] The specific technical solutions are described as follows.

[0027] In one implementation, the present description provides a data volume calibration method applied to a storage device, the method comprising: in response to a garbage data volume calibration function being enabled, selecting an aggregation object, screening to-be-checked KV data in the aggregation object according to a check bitmap, the check bitmap recording marking information of each KV data marked as to-be-deleted data; comparing corresponding metadata according to each to-be-checked KV data not marked as to-be-deleted data obtained through the screening, and confirming KV data that is garbage data and not marked as to-be-deleted data according to a comparison result; and calibrating garbage data volume in the aggregation object according to KV data that is garbage data and not marked as to-be-deleted data and / or KV data marked as to-be-deleted data in the aggregation object; and repeatedly performing the above steps to traverse and calibrate garbage data volume of each aggregation object included in the storage device.

[0028] Specifically, as Figure 1 includes the following steps, the following steps can be exchanged in the order of priority according to the needs of the actual application scenario: Step S11, in response to a garbage data volume calibration function being enabled, selecting an aggregation object, and screening to-be-checked KV data in the aggregation object according to a check bitmap.

[0029] In real-world distributed storage system environments, especially those employing Redirect on Write (ROW) technology, unavoidable events such as unexpected power outages, node failures, and software anomalies can lead to metadata commit failures or interruptions in garbage persistence. These events do not always cause system crashes; sometimes they may only cause localized data inconsistencies. However, the cumulative effect of multiple minor failures can lead to an increasingly larger discrepancy between the garbage amount statistically recorded within the system and the actual amount of invalid data in the physical storage space.

[0030] A monitoring mechanism is set up to track abnormal fluctuations in garbage collection statistics or situations where the statistics do not match the rate of storage space consumption. When the monitoring logic detects signs that the garbage collection statistics may be inaccurate, such as after a long period of operation, the available space recorded by the system continues to decrease, but the statistical garbage collection volume does not increase to the expected level accordingly, or the triggering frequency and effect of garbage collection (GC) operations are significantly inconsistent with the data writing pattern, an alarm is triggered or a suggestion is made to start the garbage data volume calibration function.

[0031] System administrators can manually enable the calibration function based on this alert through the management interface or command-line tools. The system can also be designed with automated strategies, such as automatically and tentatively initiating the calibration process in the background periodically (e.g., during maintenance windows when system load is low) if an abnormal shutdown is detected during system startup self-test, to preventively detect and correct deviations.

[0032] "Responding to the garbage data volume calibration function being enabled" can be a passive action triggered by external management commands, or it can be an active maintenance task initiated by the system's internal intelligent scheduling mechanism.

[0033] Once the calibration function is enabled, the first step is to select the aggregation objects to be processed. In the ROW architecture, when user data is written, it does not directly overwrite old data, but is aggregated into a newly created, logically contiguous data unit, which is called an "aggregate object." The system maintains a garbage volume table, which records the basic information of all aggregation objects and their estimated garbage volume percentage. These aggregation objects are then categorized into different tiers based on their historically recorded garbage volume as a percentage of their total capacity.

[0034] For example, several thresholds can be set, such as 0-10%, 10-30%, 30-50%, 50-70%, 70-100%, etc. The lower the threshold, the smaller the proportion of data considered invalid in the aggregate object, and theoretically the lower the urgency of its reclamation, but the amount of invalid data may be large; the higher the threshold, the larger the proportion of garbage data contained in the aggregate object, making it a potentially high-quality object for garbage collection, but the number of invalid data may be small.

[0035] The calibration process employs a reverse traversal strategy, moving from low to high tiers. This design prioritizes scanning aggregates with low garbage content, as these objects typically have high data validity and require less metadata verification during calibration, thus reducing query pressure on the system metadata database (DB) and preventing drastic impacts on system resources. The background scanner starts from the lowest tier and sequentially retrieves the list of aggregates within that tier. The selection of aggregates can be done by sequentially traversing the list or by randomly selecting them under a certain strategy to ensure load balancing.

[0036] After selecting an aggregate object to be processed, filter the KV data to be verified in the aggregate object according to the verification bitmap.

[0037] The check bitmap is a continuously maintained data structure, essentially an array of bits, where each bit corresponds to a specific key-value (KV) data unit. When a user initiates a deletion operation or data is overwritten by a new version (due to the ROW mechanism, new data is generated, and old data becomes logically invalid), in addition to marking the data as invalid in the metadata, it is also recorded in the garbage collection-related check bitmap. Periodically or according to a specific strategy, the identification information (such as its unique key or hash value) of KV data accumulated over a period of time and marked as pending deletion is mapped to specific bits in the check bitmap, and the bit is set to "1" (indicating it has been marked as pending deletion) or "0" (indicating it has not been marked).

[0038] When calibrating an aggregate object, the calibration engine loads the check bitmap segment associated with that aggregate object and then iterates through all key-value (KV) data blocks contained within it. For each KV data, the engine queries its marking status by calculating the corresponding position of its identifier information in the check bitmap. The filtering logic is as follows: KV data that has been marked as "to be deleted" (i.e., the corresponding bit is "1") in the check bitmap can be directly identified as garbage data by the system, without the need for immediate subsequent metadata comparison. This is because bitmap querying is an extremely fast memory operation with minimal overhead. Through this rapid bitmap screening, a portion of known garbage data in the aggregate object can be immediately filtered out, greatly reducing the amount of data that needs to proceed to the next step of metadata querying. The number of KVs not marked as "to be deleted" in this step are the objects that need further verification.

[0039] Step S12: Based on the KV data to be verified that have not been marked as data to be deleted, compare the corresponding metadata, and confirm the KV data that are garbage data and have not been marked as data to be deleted based on the comparison results.

[0040] For these key-value (KV) data points that have been initially screened through bitmaps and have not been marked for deletion, the calibration process requires precise identity verification. In the ROW system, the authenticity and validity of the data are ultimately determined by the metadata database (DB). The metadata DB records the latest logical view of all valid data in the current system.

[0041] Extract the metadata information (e.g., Key, which may include username, offset, length, etc.) of the key-value data to be verified, and initiate a query request to the metadata database. The purpose of this query is to confirm whether the key-value data is still valid in the metadata database.

[0042] The comparison results usually fall into two categories: In the first scenario, a valid record corresponding to the key-value (KV) data does not exist in the metadata database, or the existing record points to a new ROW object in a different location (meaning the data has been overwritten by new data and the current version is invalid). This comparison result clearly indicates that although the KV data is not recorded in the verification bitmap (possibly due to a previous failure causing the marker to be lost), it is actually invalid data, i.e., garbage data. Therefore, this verification KV data is ultimately confirmed as garbage data and is not marked as KV data to be deleted.

[0043] In the second scenario, if the records in the metadata database confirm that the key-value data is still valid and up-to-date, then it is considered valid data and not junk data.

[0044] Metadata comparison can identify data that has failed to be marked by the normal garbage collection process due to various faults but is actually invalid. Although this step involves accessing the metadata database and has some IO cost, the total amount of KV data that needs to be queried is significantly reduced because a large amount of "known" garbage data has already been filtered out by the verification bitmap in the previous step. This keeps the IO overhead of the entire calibration process within a controllable range.

[0045] Step S13: Based on the KV data in the aggregation object that are garbage data and not marked as data to be deleted and / or KV data marked as data to be deleted, calibrate the amount of garbage data in the aggregation object.

[0046] After verifying all key-value data within the aggregate object, the actual amount of garbage data in the aggregate object can be accurately calculated. For the aggregate object being processed, the garbage data inside may consist of two parts (or only one part may exist, i.e., only one type of garbage data, or neither part may exist, i.e., no garbage data). The first part is the garbage data directly identified during the verification bitmap screening stage, and the second part is the garbage data newly discovered after metadata comparison.

[0047] The total physical storage space occupied by these two parts of KV data is calculated. This total is the actual amount of garbage data for the aggregation object at the current moment. The garbage amount record for the aggregation object in the garbage amount table is updated based on the calculated actual value.

[0048] Step S14: Repeat the above steps to iterate through and calibrate the amount of garbage data in each aggregate object included in the storage device.

[0049] After calibrating one aggregate object, the engine continues to select the next aggregate object from the list of the current gear, repeating the bitmap filtering, metadata comparison, garbage calculation, and record update process described above. This continues until all aggregate objects in the current gear have been processed. Then, the engine switches to the next higher gear and continues traversing and calibrating until all aggregate objects in all gears have been traversed.

[0050] Based on the accurate total amount of garbage data after calibration, a garbage collection strategy can be calibrated. The garbage collection strategy (e.g., the threshold for triggering collection, the intensity of collection, or the priority) depends on the accuracy of the total amount of garbage data. The calibrated total amount of garbage data is compared with several preset strategy thresholds to determine the most suitable current collection strategy A. This strategy A is then compared with the system's currently running collection strategy: if the new strategy A is less aggressive than the current strategy (e.g., the actual total amount of garbage is found to be less after calibration than previously thought), a large-scale garbage collection migration operation is not necessary immediately. The current strategy can simply be updated with the new strategy A, and the calibrated amount of garbage data recorded, avoiding an unnecessary and energy-intensive data migration. Conversely, if the new strategy A is more aggressive than or equal to the current strategy (e.g., the total amount of garbage is found to be far greater than recorded after calibration, posing a risk of disk overload), the garbage collection process should be initiated immediately. This involves migrating valid data, freeing up space, calibrating the amount of garbage data, and updating the collection strategy after collection is complete.

[0051] In this way, this implementation not only accurately calibrates the key indicator of garbage volume, but also puts the calibration results into practice, dynamically and accurately adjusting the system's recycling behavior. This fundamentally reduces the risk of disk overload caused by inaccurate garbage volume and recycling strategies, ensuring the long-term stability and efficient operation of the distributed storage system. The entire solution cleverly achieves a good balance between the thoroughness of calibration, the efficiency of the process, and the impact on front-end operations by introducing mechanisms such as tier traversal and verification bitmaps.

[0052] In one implementation, the method includes: prioritizing the calibration of aggregate objects with less garbage data based on the currently recorded amount of garbage data.

[0053] In one embodiment, the method includes: accumulating the amount of garbage data of each aggregated object after calibration to obtain the total amount of garbage data after calibration, and calling and executing the corresponding preset garbage collection strategy according to the total amount of garbage data.

[0054] In one implementation, the method includes: updating a check bitmap in response to a data deletion signaling and marking the corresponding key-value data as data to be deleted.

[0055] In one implementation, the response to the garbage data volume calibration function being enabled includes: responding to a garbage data volume calibration command issued by a user; or responding to a plan to enable the garbage data volume calibration function at a preset time; or responding to an event in which the current total data volume exceeds a preset threshold.

[0056] In one implementation, the garbage collection table is traversed by bin size, and the metadata of each bin is broken down into key-value pairs. After a quick bitmap analysis, a decision is made whether to query the metadata database again to obtain the amount of garbage in each bin. Once all bins have been traversed, the total amount of garbage can be calculated. Based on this total amount, a garbage collection strategy is derived. If the calculated strategy is less than the current strategy, garbage collection is not performed, and only the garbage amount is calibrated. If the calculated strategy is greater than or equal to the current strategy, the garbage is moved for calibration. After the garbage amount is calibrated, the current garbage collection strategy is updated. This function can run slowly in the background by traversing bins in reverse order from low to high, ensuring that the impact on foreground services during background scanning is controllable. Bin sizes are divided according to the proportion of garbage amount; the lower the size, the less garbage is represented. The GC collection strategy depends on the total amount of garbage in the system. Once the garbage amount is calibrated, the current collection strategy can be updated promptly to avoid an incorrect collection strategy that is too small, leading to disk overfilling.

[0057] In one implementation, such as Figure 2 This specification also provides a data volume calibration device applied to a storage device. The device includes: a first module, configured to select an aggregation object in response to the garbage data volume calibration function being enabled, and filter the key-value (KV) data to be calibrated in the aggregation object according to a verification bitmap, wherein the verification bitmap records the marking information of each KV data marked as data to be deleted; a second module, configured to compare the corresponding metadata with each KV data to be calibrated that is not marked as data to be deleted, and confirm that it is garbage data and not marked as data to be deleted based on the comparison result; a third module, configured to calibrate the amount of garbage data in the aggregation object based on the KV data that is garbage data and not marked as data to be deleted and / or the KV data marked as data to be deleted; and cyclically calling the first, second, and third modules to traverse and calibrate the amount of garbage data in each aggregation object included in the storage device.

[0058] In one implementation, the first module prioritizes selecting the aggregation object with less garbage data based on the amount of garbage data currently recorded.

[0059] In one implementation, the third module is further used to accumulate the amount of garbage data of each aggregated object after calibration to obtain the total amount of garbage data after calibration, and to call and execute the corresponding preset garbage collection strategy according to the total amount of garbage data.

[0060] In one implementation, the first module is further configured to update the verification bitmap and mark the corresponding KV data as data to be deleted in response to the data deletion signaling.

[0061] In one implementation, the response to the garbage data volume calibration function being enabled includes: responding to a garbage data volume calibration command issued by a user; or responding to a plan to enable the garbage data volume calibration function at a preset time; or responding to an event in which the current total data volume exceeds a preset threshold.

[0062] The implementation methods of the apparatus are the same as or similar to the corresponding implementation methods, and will not be described again here.

[0063] In one embodiment, this specification provides an electronic device including a processor and a readable storage medium storing machine-executable instructions executable by the processor. The processor executes the machine-executable instructions to implement the aforementioned data volume calibration method. From a hardware perspective, a hardware architecture diagram can be found... Figure 3 As shown.

[0064] In one embodiment, this specification provides a readable storage medium storing machine-executable instructions that, when invoked and executed by a processor, cause the processor to implement the aforementioned data volume calibration method.

[0065] Here, a readable storage medium can be any electronic, magnetic, optical, or other physical storage device that can contain or store information, such as executable instructions, data, etc. For example, a readable storage medium can be: RAM (Random Access Memory), volatile memory, non-volatile memory, flash memory, storage drives (such as hard disk drives), solid-state drives, any type of storage disk (such as optical discs, DVDs, etc.), or similar storage media, or combinations thereof.

[0066] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer, which can take the form of a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email sending and receiving device, game console, tablet computer, wearable device, or any combination of these devices.

[0067] For ease of description, the above devices are described in terms of function, divided into various units. Of course, in implementing this specification, the functions of each unit can be implemented in one or more software and / or hardware components.

[0068] Those skilled in the art will understand that embodiments of this specification can be provided as methods, systems, or computer program products. Therefore, this specification can take the form of a completely hardware implementation, a completely software implementation, or an implementation combining software and hardware aspects. Furthermore, embodiments of this specification can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0069] This specification is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments thereof. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0070] Furthermore, these computer program instructions can also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in the process. Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0071] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0072] Those skilled in the art will understand that embodiments of this specification can be provided as methods, systems, or computer program products. Therefore, this specification can take the form of a completely hardware implementation, a completely software implementation, or an implementation combining software and hardware aspects. Furthermore, this specification can take the form of a computer program product implemented on one or more computer-usable storage media (which may include, but are not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0073] The above description is merely an embodiment of this specification and is not intended to limit this specification. Various modifications and variations can be made to this specification by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this specification should be included within the scope of the claims of this specification.

Claims

1. A data volume calibration method, characterized by, The method is applied to a storage device, and the method comprises: In response to the garbage data volume calibration function being enabled, selecting an aggregated object, and screening to-be-checked KV data in the aggregated object according to a check bitmap, the check bitmap recording marking information of each KV data marked as to-be-deleted data; According to the to-be-checked KV data screened, comparing corresponding metadata, and according to a comparison result, confirming KV data that is garbage data and has not been marked as to-be-deleted data; According to the KV data that is garbage data and has not been marked as to-be-deleted data and / or the KV data marked as to-be-deleted data in the aggregated object, calibrating the garbage data volume in the aggregated object; Cyclically performing the above steps to calibrate the garbage data volume of each aggregated object included in the storage device.

2. The method of claim 1, wherein, The method comprises: According to the currently recorded garbage data volume, preferentially calibrating an aggregated object with a smaller garbage data volume.

3. The method of claim 1, wherein, The method comprises: Cumulatively calibrating the garbage data volume of each aggregated object after calibration to obtain a total garbage data volume after calibration, and according to the total garbage data volume, calling and executing a corresponding preset garbage collection strategy.

4. The method of claim 1, wherein, The method comprises: In response to data deletion signaling, updating the check bitmap, and marking corresponding KV data as to-be-deleted data.

5. The method of claim 1, wherein, The response to the garbage data volume calibration function being enabled comprises: In response to a garbage data volume calibration instruction issued by a user; Or, in response to a plan of the garbage data volume calibration function enabled at a preset time; Or, in response to an event that the current total data volume exceeds a preset threshold.

6. A data volume calibration apparatus characterized by comprising: The device is applied to a storage device, and the device comprises: A first module configured to, in response to the garbage data volume calibration function being enabled, select an aggregated object, and screen to-be-checked KV data in the aggregated object according to a check bitmap, the check bitmap recording marking information of each KV data marked as to-be-deleted data; A second module configured to, according to the to-be-checked KV data screened, compare corresponding metadata, and according to a comparison result, confirm KV data that is garbage data and has not been marked as to-be-deleted data; A third module configured to, according to the KV data that is garbage data and has not been marked as to-be-deleted data and / or the KV data marked as to-be-deleted data in the aggregated object, calibrate the garbage data volume in the aggregated object; The first module, the second module, and the third module are cyclically called to calibrate the garbage data volume of each aggregated object included in the storage device.

7. The apparatus of claim 6, wherein, The first module is configured to, according to the currently recorded garbage data volume, preferentially select an aggregated object with a smaller garbage data volume.

8. The apparatus of claim 6, wherein, The third module is further configured to cumulatively calibrate the garbage data volume of each aggregated object after calibration to obtain a total garbage data volume after calibration, and according to the total garbage data volume, call and execute a corresponding preset garbage collection strategy.

9. The apparatus of claim 6, wherein, The first module is further configured to, in response to data deletion signaling, update the check bitmap, and mark corresponding KV data as to-be-deleted data.

10. The apparatus of claim 6, wherein, The response to the garbage data volume calibration function being enabled comprises: In response to a garbage data volume calibration instruction issued by a user; Or, in response to a plan of the garbage data volume calibration function enabled at a preset time; Or, in response to an event that the current total data volume exceeds a preset threshold.

11. An electronic device, comprising: The device comprises: A processor and a readable storage medium storing machine executable instructions executable by the processor to implement the method of any one of claims 1-5.

12. A readable storage medium, characterized by, The readable storage medium stores machine executable instructions which, when invoked and executed by a processor, cause the processor to implement the method of any one of claims 1-5.

Images